Making the Most of Big Data: Data-Driven or Data-Centric?

Most companies have the goal of being data driven, and most are making some level of investment to get there. In their annual survey of Fortune 1000 executives on the topics of data and innovation, NewVenture Partners report that nearly 3/4 of executives surveyed are getting measurable results from their investments in Big Data and AI initiatives, up from less than 50% last year. Progress was made on making better business decisions, better customer service, and cost reduction. No wonder that over 98% of those executives say their companies aspire to be data-driven. Almost 37% even said they had actually achieved it. But is this the right goal, or is something more needed?

For at least two decades, data-driven decision making has been used drive up predictability and efficiency. Data-driven decision making required a build-out of IT infrastructure to be able to collect and manage data. This included enterprise resource planning systems, customer relationship management systems, and supply chain management systems. The gains from those systems seem obvious but were hard to quantify initially.

A 2011 study by Brynjolfsson, Hilt, and Kim on business practices and IT investments of large publicly traded companies found that data-driven decision making was associated with a 5-6% increase in output and productivity beyond what could be attributed to other investments and IT usage. Even if in the past few years we have progressed further, do we not hope for a whole lot more? What prevents getting there?

An interesting perspective is to look back at other history changing technologies. For example, in The Second Machine Age, Brynjolfsson and McAfee look back on how long it took electric motors to significantly improve productivity over steam engines on the factory floor. For the first couple of decades, machines with electric motors were installed in the same floor layout. Noise was reduced, but productivity was nearly the same. It wasn’t until thirty years later when factories were organized by workflow rather than power distribution that there was a significant gain in productivity.

Similarly, to get the most out of the huge amounts of data being collected, companies need to optimize around the data, not just add data to the existing processes. Becoming such a data-centric company requires changes in organization and processes, not just technology and skills.

I am far from the first to advocate data-centric organization over a data-driven one, but many have focused more on the data model. Certainly if you don’t have a data model and architecture, then combining data from difference sources can be problematic. But being data driven needs to go much deeper than that to include organizing the business and processes around data. Companies that can move quickly to optimize around the data, have a very good chance of seeing significant productivity gains from big data in a time frame much shorter than the thirty years it took for electric motors.