It is not about Data Scientists

Big Data has risen a new challenge in the organization, and despite what I hear and read everywhere I don't think it is just related to finding new competencies.

First of all those competencies are not new at all. Data Mining on what used to be called VLDB (Very Large DataBase) is a problem as old as databases are, and I have publications on this topics from late '90s, covering and describing exactly those algorithms used today to find correlations and patterns in BigData. And even if you do not have anyone with solid experience on this specific topic, if you have hired an IT professional, the mathematical and statistical background she/he has developed during his education will help her/him to close the gap quickly.

The problem is different, bigger and much more difficult to address.I found LittleMismatched my favourite business example to explain it to my customers.

Imagine three friends joking around one of the biggest mystery in the universe: those orphan socks lying in your tray, socks you cannot use any more because their twins were never found again. "Why even bother wearing matching socks in the first place?" they though, and when the laughter came to a stop, it occurred to them that they just might have a legitimate business idea on their hands. In no time, they were talking seriously about the idea of selling mismatched socks and they created a 5 Millions dollars business, in a very mature market with a strong competition coming from all those low cost-production countries.

So what made the difference? Had they better data than those big companies producing socks? Were they a bunch of Data Scientists developing the ultimate mining algorithm? No. They simply had creativity.

And this is where most of the companies approaching BigData are failing. They are very focused on the IT portion of the project and the mining part becomes something they often see as the new frontier. But to transform those information into value for the company creativity is needed and so far, when I've asked my customer to mention me a job they consider synonymous of creativity, no one has ever answered: "the Data Scientist! ".

Creating value from big data is an exercise that requires creativity and deep understanding of the business, and those qualities are spread across the organization among business users, and not concentrated in a single data genius. The challenge created by Big Data is a new and sometimes huge separation between business and information, now inaccessible for volume and technology complexity.

Companies should focus on giving business users access to big data information, even in a simplified form, instead of looking for the maximum complexity and sophistication no-one can really benefit from.

Only then BigData will start generating value in the organization.