In part 1 of “ETL 3.0: Getting Down To Business“, I described the core principles of an ETL architecture – flexibility, extensibility and autonomy – and the business benefits that flow from such an architecture. In particular, the business benefits because flexible, extensible, autonomous architectural principles empower IT to respond rapidly to changes in the business environment. An empowered IT serves the business by making better Product Selections and making better Staff Selections.
Beyond that, there are more ways to benefit the business through an architecture premised on flexibility, extensibility and autonomy.
Invention is the natural outcome of curiosity, need and opportunity. Invention begins when someone is faced with a problem (the need) that is systemic or chronic — a problem that goes on and on and demands attention. A problem that costs the business money. Someone wonders “why can’t we change something so that this problem goes away?” (the curiosity). And that person takes the time (the opportunity) to seek out a solution to the problem. That solution, when it is found, is the invention.
But an invention can shrivel on the drawing board unless there is a way to economically and effectively apply the change. That means there must be sufficient flexibility in the system to permit the invention to be tried, and – if successful – to be adopted.
An ETL application which is built around a single, all-encompassing and tightly-integrated product leaves little room for examining alternative solutions. Often, the time, effort and cost of testing the invention are too much to bear. Unless the invention is likely to produce significant value with minimal disruption, it can remain only an idea that never finds its way to implementation.
An ETL 3.0 environment welcomes invention. The hallmarks of the architecture – flexibility, extensibility and autonomy – encourages IT to introduce inventive solutions by minimizing the disruptive impact of change. This is an architectural view that accepts change, whether as remedies or as enhancements, as a natural way to grow and improve a system.
Bottom Line: The ETL 3.0 model empowers IT to better serve the business, by inviting and welcoming invention and minimizing the disruption of exploration, discovery and growth.
A variation of the invention is the one-off solution – a single-purpose solution, somewhat out of the ordinary and not fitting into the pre-defined standard that has been established for the system. Every IT organization has worked to create an ETL framework that allows for somewhat rapid development of an ETL job. But what happens when a solution doesn’t fit into the framework? Or when the framework is just too rigid for this solution?
We have all found ourselves creating a quick-and-dirty solution. And we have all promised to throw that solution out once we’ve addressed the immediate problem. And we have all found ourselves hiding it away, to be used again later.
There really isn’t a need to discard a solution that is quick-and-dirty. More than likely, you will need it again, either as a reference or as a basis for another solution.
What is actually needed is a framework that allows you to build a quick-and-clean solution, without sacrificing all of the governance and business rules that apply to normal solutions. A solution that is a one-off is still manipulating data within your domain, and needs to be compliant with the same rules.
An ETL 3.0 environment accepts the one-off solution as a normal (though infrequent) method for addressing problems. In the ETL 3.0 environment, a one-off solution is formed by a unique re-combination of existing parts, rather than by “hacking together” work that is truly quick, but also truly dirty.
Bottom Line: The ETL 3.0 model empowers IT to better serve the business, by offering ways to create a quick-and-clean solution, leveraging both the parts and the framework that already exists in normal production.