Heraclitus, Change and ETL

ποταμοῖσι τοῖσιν αὐτοῖσιν ἐμϐαίνουσιν, ἕτερα καὶ ἕτερα ὕδατα ἐπιρρεῖ.1

Heraclitus, the ancient Greek philosopher, said “You cannot step twice into the same river”. We have been living this truth in IT for decades now. Change is with us, undeniable and inexorable. Sometimes we welcome the change; sometimes we resist the change; sometimes we fear the change. But change wins out. What we assume to be obviously true today will be obviously false one day.

IT Needs to Change

Change as a constant, driving force is well-recognized in the IT world. As we look back over the last 20 years, we see the arrival and departure of new technologies, new vendors, and new tools. We see new development processes as Agile and Scrum press hard against Waterfall and SDLC. We see new alliances, as one large company acquires — and sometimes dismantles — another, as familiar tools and languages become obsolete and new ones take their moment in the spotlight. We see Open Source gain a foothold against the major tool sets, and the demise of “cradle-to-grave” CASE tools against the speed and agility of IDEs. And we see the emergence of the internet, email, smart phones and WiFi creating a demand for IT services that is always present, always connected, always on. Change is both constant and immediate.

Business Needs to Change

The change in IT is reflective of the change in the business that IT serves. Business is under increasing pressure to satisfy a marketplace that demands better, cheaper, faster goods and services. Cheaper and faster comes from mass production, where competing companies offer the same product and try to distinguish themselves on price and marketing cost — on “mass customization” and a “360-degree view of the customer”. To meet this need, the business needs to know more about each customer. To know more, the business needs more data. And — because of the changes in technology — there is more data available all the time about everything everybody does. Business can keep up with this explosion in data because advances in technology make it both possible and cost-effective.

Which brings us to the need for an ETL architecture that allows the data warehouse to move at the speed of the business. We have seen that technology changes frequently, and usually for the better. When technology changes, can IT embrace that change, fold it into its arsenal and exploit the improvement it offers? Or is IT shackled to a technology that is tried and true but no longer measures up to the demands of modern business?

What Would You Do?

As you look at your ETL architecture, how would you respond to these change scenarios?

  • A new database server promises to provide the same capabilities as your legacy servers, at a dramatic reduction in cost. But it uses standard SQL and has its own proprietary utilities, much different from your legacy systems (which use a proprietary SQL). You decide it will cost too much to change your legacy ETL applications to use a new set of tools and commands, so you skip this cost-reduction opportunity. How do you explain that to the business?

  • The vendor of your core ETL software product is at risk of going bankrupt. You’re afraid that you will lose any support or future enhancements to the product. You need to find an alternative product, but you have 15,000 ETL jobs and 20,000 business rules embedded in the product. It will take 12 months and several million dollars to convert to a new tool, just to do what you’re able to do today. How do you avoid getting into this situation again?

  • You can reduce staffing costs by outsourcing the development work. You are using a leading ETL software product to extract data from 4 types of source systems and to load 3 different types of database servers. You’re looking for ETL developers who know all 8 products, but it’s apparent that there is no one with that combination of skills. You end up having to split the work among 3 different outsource firms to get developers with the skills to implement your ETL jobs, and managing the distribution of work is a nightmare. You’re afraid you’ll have to give up the outsourcing savings and bring everything back in-house. Do you have any other options?

None of these are comfortable situations to be in – and a well-crafted architecture can go a long ways toward avoiding these situations. This series continues with a look at the architectural principles behind ETL 3.0 — principles that drive such a well-crafted architecture.


1Translation: “Ever-newer waters flow on those who step into the same rivers”

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: