In the last blog entry, “Heraclitus, Change and ETL”, I described three scenarios that, as IT professionals, we never want to encounter twice. All of these scenarios were rooted in an inability to adapt to change.
In the first scenario, the business loses the opportunity to reduce its cost because the legacy ETL applications are wedded to your existing technology. Years ago, the decision was made to tightly couple the applications to the technology to gain the best possible performance. It was a good decision then — performance has been stellar, even at one time extraordinary. Today, however, that decision is a constraint, and costs the company a significant cost-savings opportunity.
What do you say to the business? You could defend the tight-coupling decision, but that doesn’t move you forward to the next opportunity. Unless you take action, you may find yourself making the same defense in another 12 months. That won’t be a meeting you will enjoy.
In the second scenario, the business is not offered the choice, only an impending and unavoidable disruption. When you chose this software tool several years ago, you paid due diligence to the vendor’s financial stability and outlook. But the unthinkable has happened — the economy sank, the bubble burst, credit became more difficult to get and the vendor is now floundering. There is no problem with the tool — it works as well as ever. But it no longer has a future and your investment in that tool is now at risk. This is the problem of the whole-solution product. You have your whole application, indeed the life-blood of your data warehouse, tied to this soon-to-be-obsolete product.
There is no easy way out of this predicament, but that’s not the question. The question is, as you bite the bullet and move your application out from under this deprecated product, what can you do to avoid landing right back into the same trap? You could search for the vendor that you’re certain will never fail — but in these times there are no sure bets. The product you need most may be the product of a new and financially unproven vendor. Should you bet your job on the new vendor? Or bet your job on the proven track record? How about if you don’t bet your job at all?
The third scenario goes to the heart of the cost-saving pressures that we face every minute. Staffing an ETL project presents its own set of challenges; you need expertise in a variety of tools and platforms — just one skill won’t do. Every additional source or system type added to your ETL mix has a multiplier effect on the staffing needs. As the total warehouse becomes more complex, the number of outsourcing options diminish geometrically, until — as in one case — the number of staffing options is reduced to zero.
What choices do you have? Of course, one option is to take the single-platform route — move all of your data warehouse platforms to a single family, and get by with one set of skills and tools. But this seems oddly reminiscent of other constraining problems.
What if you could use individuals who each add one expert skill to the team? And what if the very unique skills could be hired as temporary help, to fill the gap? And what if each expert could build their part of the solution knowing only how that part fit with other parts — but not knowing how the other parts worked? This has worked for years in other industries — why not in your multi-platform, multi-tool ETL shop?
I described three scenarios that are at the heart of what frustrates the business about IT. ETL 3.0 will help you avoid these scenarios in the future, but it’s important, first, to understand that these situations are not an inevitable part of delivering high-performing or quick-to-market IT solutions.
To meet the pace of change in technology and business that is part of the current marketplace, IT must remove the threat of vendor breakdowns, and exploit the advantages of lower-cost solutions while still meeting — and exceeding — the demand for new capabilities. As IT Professionals, our choices cannot simply be about leveraging the proprietary [and cool] features of a particular technology or having one technical part operating as fast as possible or focusing on the initial speed to implement or cost to deploy. Our focus must be on how we remove business barriers, how we empower the business to respond to competitive threats and how we contribute to business value.
This focus on the business is at the heart of ETL 3.0. I’ll describe that in the next dispatch.