Got Data?

One of the most (ab)used concepts in the business world today is the notion of "data-driven" decisions. This is the view that actions need to be predicated on careful analysis of available information, whether it be from a technical experiment or market survey. It conjures up images of managers sitting in a big room, going through presentations put together by analysts and engineers who have been patiently crunching numbers. People interpret the numbers, ask a bunch of questions, make a decision - and move on to the next issue at hand.

As we move into the age of Big Data, the amount of information coming into businesses will grow exponentially. Certainly beyond the capacity of any single human being to fully understand - or act upon. Since our minds are capable of only deciphering relationships between a limited number of parameters, we will increasingly have to rely machines to do the analysis - and decision making - for us. All of this means that we have to be absolutely sure that we are getting GOOD data - and not something that is irrelevant or, worse, misleading.

In this day and age, however, getting two people to agree on what constitutes a set of good data is pretty challenging. That's because of our inherent bias towards "selecting" only those data that fit with our worldview. This is essentially the "Black Swan" effect - till you see one, you have no reason to accept it as real.

And what about situations where we have very little data available - such as early stage research? An interesting podcast from Knowledge@Wharton reveals something I suspect we knew all along - that data counts, but there is the "gut feel" that is just as important.

http://knowledge.wharton.upenn.edu/article/gut-feel-and-investing/

Someone who has "been there, done that" is likely to view data very differently from a fresh MBA/PhD who has his head full of theoretical concepts. As I read somewhere "Good judgment comes from experience. And experience comes from bad judgment". Data are necessary, but clearly not sufficient.