Datawarehouse, no more!

Introduction

I am sure you have heard the slogans circling around data and the related technologies: “Data is new oil”, “Data is the most important corporate asset”, “Data is new gold” so on and so forth. Following the data trends, companies are trying to modernize their data infrastructures, founding new data organizations, creating board awareness and making it a part of their digital journey. Moreover, regulators are pushing data management and data governance obligations on enterprises harder every day.

All those are true, and right.

However, I have doubts when I think whether we invest for the future in a proper way or not.  Do we know what we are doing? Are we capable enough to build the future?

I will explain my question marks in a structured way considering a number of dimensions and while discussing the problems, I will propose possible solutions in abstract forms as well.

Mother of all evils

Datawarehouses. Have you heard the stories about them? Many projects have failed while people were trying to build them. Many managers were fired. Many DBAs were cursed and many programmers forgot their professional notion while feeding data into them. A dark shadow, reaching today and struggling to expand tomorrow, coming from 1980s and 1990s.

A typical datawarehouse conventionally offers a number of fundamental values to the technological infrastructure of the corporations such as accumulating data historically, integrating high quality data in a single platform, bridging the application silos, providing cohesive data organised by business subjects, enabling data analytics and business intelligence, robust reporting etc.

Very attractive value propositions indeed. Therefore, all the universe invested in datawarehouses. Data architects, DBAs, data engineers and others started to work hard for finding the best way of copying data from the source systems, transforming them in an efficient manner, storing them in the correct target data models: Some built data marts, some went for great foundational layers in third normal form. Some picked the special purpose database appliance systems for overcoming underperforming queries and data integration software, some procured systems with fat cache feeding infiniband wires, some picked database systems specifically designed for data warehouses. Some failed, some ended up with the systems that are able to run.

Today, vendors are still playing the same game and companies are still building datawarehouses. Let’s analyse the picture of a typical datawarehouse habitat.

Figure 1: A typical datawarehouse scheme. Source: TDWI, 

Very familiar: a set of data sources comprises operational  systems such as core banking, CRM, ERP, accounting etc.; ETL modules for extracting data from the data structures of the source systems, transforming them and loading into the datawarehouse. Another set of ETL modules or some sort of views for feeding BI, reporting, analytics based or business reporting oriented data marts.

Problem #1: You always end up with data redundancy. It is in the nature of datawarehouses, they store copies of the real data. And no copy is as good as the genuine. For instance, if a court order requests the accounting data of your company, you deliver it from your accounting system or you validate the data you get from your datawahouse by using the data coming from your accounting system. Fresh, hot, operational and reliable as your blood. No cold fish needed!

Problem #2: ETL is guaranteed bad software engineering. Years ago, I highlightedthe characteristics of high quality software system in my blog. I will not jump into the details again but just want to remind two very core characteristics (a) high cohesion (b) low coupling. 

Your modules must be encapsulating functions and data that are strongly interrelated for being a highly cohesive software system. If you designed the module for sales management, all the routines, data structures, functions, assertions etc. in your module must be there for handling sales, nothing more, nothing less. 

In contrary, datawarehouse systems cannot stand for high cohesion because they are designed for covering all enterprise data coming from highly cohesive source systems. That’s one of the reasons a very strong metadata management layer is essential for datawarehouses. Otherwise, you naturally get lost. 

Low coupling refers to minimising the inter-module dependency of the software systems. For providing such an attribute, you need to hide the implementation details of a module from the other modules and you have to set the interfaces for must-have communications with other modules. 

By doing so, it is assured not to affect outer modules in a negative way like invalidating them, breaking their functionality etc. while you are changing the internal structure of your module. Every module is responsible for the interfaces they opened to other modules, keep the interface compatible to the original communication act then, enhance your internals independently. Be message oriented, no unknown or unwanted dependencies, no non-needed details of outer systems to manage. 

Low coupling is good, tight coupling is bad. 

Again, ETL is innately brings the attribute of tight coupling. All ETL tools and techniques force the data integration engineers to know the very internals of source systems: all tables, columns, meanings of the flags sored in the columns, all keys, lookup structures, everything. ETL is like a dagger you inserted into heart of the source systems. It hurts. It is not for good unless you are in the business of assassination. After your ETL systems are plugged into the depths of the ERP systems, CRM systems, Sales systems, Core Banking systems, your data flow towards the datawarehouse is always vulnerable to be invalidated every time a data structure change occurs in the source systems. You must be continuously trying to synchronize yourself with the new data structures and the new syntax and semantics attached to them.

That is mission impossible. That is bad software engineering. That is prying into others’ affairs. It is impossible to always be aware of every implementation details of outer systems for the sake of copying data. A total bad practice.

Perhaps, ETL developers don’t call themselves as software engineers because of this unhealthy heritage of the ETL approach. Most of the ETL professionals I’ve been interviewing are confident with the ETL tools they are familiar with, they know using the tool and they are not interested in the underlying infrastructures or the general idea. Some call themselves data integration specialists in a funny way because ETL is able to integrate “nothing”, just an ugly between-systems dependency technique.

Problem #3: The world is spending millions for enabling cloud foundry, open API, microservices, continuous delivery, continuous integration, hyper connectivity etc. meanwhile what’s datawarehuse world doing? Copying terabytes of data between systems in batch fashion for providing already old data, being affected by every change done in any source system, getting bigger and bigger and so on. 

What is it good for? 

How will IT managers assure coordinating super fast micro systems which are responsible for core operations running in private, public or hybrid clouds, communicating event based, deployed a number of times intraday by devised automations with “reactive, slow, cold and fat datawarehouses”? 

Imagine you devised the best impact  analysis system for alerting datawarehouse people on any sort of source system changes, what will happen in a continuous delivery world? You will get hundreds of alerts and will be thrashing while trying to cope with them in your slow and fragile datawarehouse hinterland.

Architecture for the future

All negative aspects were mainly declared so far. A dark picture, seems like there is no way to escape. Any solutions? Yes.

Firstly, stop investing for traditional datawarehouses. Vendors are motivating you in a false way: the more database instances you have, the more license fees they earn. This is just the legacy sales business. Paying for datawarehouses is like sending your money to 1980. Full stop.

Second, uninstall ETL tools. Re-educate your ETL programmers for extracting the high profile software engineer in them if any left.

Third, focus on the software module design. Classify your modules as data providers and data consumers. Make your modules provide and/or consume data in a responsible way through exposed, well defined services. It is possible to follow industry specific guidelines for identifying key data services (e.g. BIAN for banking).

Fourth, invest for enterprise data bus which is to provide robust access to any sort of identified data service of the modules you developed. There can be many options of data access: ad-hoc, continuous, record based, bulk fashion etc. Harness the necessary technologies for proper data use cases; in-memory layers, self-service data analysis platforms, embedded data validation layers, volatile and non-volatile data presentations etc. Rule of thumb is “produce data knowingly, consume data knowingly”. Awareness is the key. Sample illustration is below.

Figure 2: Enterprise data bus in the middle of highly cohesive software modules

Enterprise data bus and the data services published responsibly is just an idea, not the perfect solution but, unlike datawarehouse and ETL, bearing no intrinsic anomalies. I think, we should find better ways of data and information representation, non-hierarchical data structures, non-linear data processing and handling anyhow. This sort of thinking will yield better talents, better academy and industry. Not the dead sales business in 21^st century.

Know the basics, derive for better.