The Evolution of Data Platform Vendors into Data Exchange Vendors!
IT began the data journey with wave of investments for Companies to build Enterprise Data Warehouses. IT led this charge and the ROI business cases were so dramatic that the investment costs were not the focus. IT invested in many products (ETL Products, Data Base, Reporting Tools…) with focus on building scalable and platforms Best of breed tools. This lasted for a while until the business grew impatient with IT and the speed at which they could deliver.
The Business then started to invest in their own teams to build hub Data Marts off of the EDW to address their frustration with agility of IT. Many used cheap SQL Server technology meant for departmental use and early successes were plentiful. In fact many of the business data teams portrayed IT as slow and expensive and not capable of leadership in this area. Much of the success by the business data teams was due to 3 differences from IT.
- Funding Model differences - Business used Opex for funding which could be allocated dynamically and had no ridged tracking requirements on spending of Capital Dollars used by IT. IT had to plan spend 18 months in advance and was based on rigid requirements/build process rather than agile like processes adopted by business.
- Development Process Standards - Business based data teams were not held to the IT standards and processes (Change management, Release Management, Architecture Management, Production Support transition…). They subscribed to a DevOps like model where the same team supported what they built.
- Business Acceptance – Business was more accepting of Business Based data solutions because they were all on the same team and it was coming out of Business P&L. Whereas, IT built solutions where from another group with a different P&L so the business could demand perfection with no impact to their own P&L.
This model continued for many years until data volumes began to outstrip the capabilities of data mart technology and the technology capabilities within the business.
IT then saw the opportunity to step back into leadership role and build case for driving investments in new Data Lake and Cloud technology. IT saw this as a way to move back into leadership role in Data but this time driving costs down from the original EDW investments was a main focus. IT went from best of breed tools to driving for open source code and new data platforms and platforms (AWS/Azure/GCP…) with focus on cost as main attribute. In addition, the model of buy before build was inverted in many cases and with use of open source tools the priority was custom development using frameworks and classes. This has led the rise of new IT based Data Engineering Teams and for the most part this has been a productive and successful drive by IT. IT can now scale ingesting unlimited volumes and types of data that the old EDWs couldn’t handle and at a fraction of the cost of the EDW platforms.
Then, the Business (in many companies) caught onto data science opportunities (AI/ML) and again splintered off from IT and began to apply data science principles on data replicated within the new Cloud data platforms. This became easy with the implementation of business charge back accounts in most cloud platforms so that the cost again was backed into business P&L and not in the IT P&L.. In one Customer, IT spend after moving from EDW (On-Prem) to cloud Data Platform went down but total data platform spend across all business areas went up 300%. This can be attributed to the powerful replication techniques available within cloud data platforms and without strong governance this is what happens.
Well, what would you think if Data Vendors rise up and evolve from a data technology platform into an actual Data Exchange? What if you only had to load your own transaction data into the platform and all the external data used today was already posted on same platform and was ready to query (albeit you may need some mapping tables). This would dramatically reduce the amount of redundant ETL work currently done by each company and would accelerate the availability of data dramatically.
The real question is, what will be the role sort between IT and Business given the rise of an external entity?
Time for us to write a book together as we both lived this and continue to lead the data evolution!
Great article Dan, for sure its going in that direction, example Microsoft, Adobe and SAP has created 'Open data initiative' to host customer profile data in data exchange. We need to be cautious and rather decentralize the authority of managing those data exchanges so the platform provider do not charge premium. Also the model should be more federated rather than moving the data to the same cloud of the market place, there should be a mechanism to integrate it on the fly for consumption ! More details on the open data initiative https://www.adobe.com/experience-platform/open-data-initiative.html#:~:text=Adobe%2C%20Microsoft%2C%20and%20SAP%20have,customer%20experience%20management%20can%20be.
Great view on the Data Exchange. Is Data Exchange == Data Fabric ? Daniel Price
Great insight, Dan! I would like to also suggest a 4th IT issue: competing business demands forced IT to rationalize and prioritize deliverables. Even with the inevitable shift to data mart solutions within the business, IT still remains as a centralized data source. Therefore, it totally makes sense to also provide centralized source for data governance to ensure data quality and consistency across those business solutions.