Small Data ....Big Value!

It was in the summer of 1999, when I first heard about the “problem” of Big Data. By the turn of the century, the world seemed abuzz with the countless opportunities this “problem” had got transformed into! It didn’t take long for the realization to dawn upon-“Everyone wants to do something about “Big Data” but nobody knows what to do or how to go about it”!

Everyone around appeared overwhelmed- “We are producing more data in one day than what has been generated over the entire one hundred years”…..”We live in a data economy-corporations need to learn how to convert data into competitive advantage or perish”…..”Every customer contact is a moment of truth, if we can grasp that in real time, we will truly succeed in our objective of customer-centricity” and the best of all…..”Data is the new oil”. 

With the advent of social media, there is seemingly so much more information around, most of which is unstructured and unruly, getting generated every second. Even if organizations invest zillions and get a handle of the huge clusters of data and unrelated information, the ability to comprehend, let alone analyze and derive meaningful insights, poses a huge challenge.

A simple yet powerful concept that I got fascinated with early in my consulting career was hypotheses based problem solving, also referred to as evidence based consulting. Any large, complex and ambiguous problem, which most business problems tend to be, can be broken down into workable modules. Instead of getting overawed by the size of the problem, hypotheses testing can help us quickly start navigating towards a solution. Some may say that it is an “Intuitive” approach, at its heart it is a fairly “smart” scientific method used to prove ideas using evidence.

I believe this approach is equally applicable when we are trying to solve a Big Data problem. In my view, hypotheses based approach can help organizations get-over the problem of-how to get a handle of the data and what to do with it once you have got the data. Before getting into a full blown war with big data, it is important to ask a few questions:

• Can I define the business problem that is at hand? (Problem statement)
• Can I break it down into modular elements? (Problem disintegration)
• Can I ask a series of “Why” questions to reach to the most granular level for each of the modules? (Causal analysis)
• What are the likely answers to the most elemental problem? (Hypotheses generation)
• What data would I need to prove or disprove the hypotheses? (Smart Data)
• How do I get the data I need? (Smart Data collection)
• What analysis I need to do to prove/disprove the hypotheses? (Smart Data analysis)
• What actionable insights can I extract from the analysis? (Insight generation)
• What is the action plan to drive value from insights? (Big Value realization)

With this understanding, we can train ourselves to search for what we need and attain a data nirvana- “It is important to have data, but it is more important to know what to do with that data”.

Smart data can help us achieve Big Value. To quote a wise-man who once said, “In our search for oysters, we need not boil the entire ocean, all we need is the knowledge-where to dredge and how!”