The biggest big data challenge: data, tools or the ‘human’ element?

When talking big data, two elements often come up first, the data and the technologies used to analyse it. While there are challenges around these two, the third and usually elephant in the room is the culture change that will be required as a part of the move to adopt big data. Often this element makes the first two look comparatively easy.

The issue is that, in adopting big data, you are asking the organisation to do something different, to become data driven, to put more value on insights that come out of data than gut feel.  Many enterprises are not particularly ready for that.

If you think about how many businesses work, they come from the top down. The people who are the most experienced have the most gut feel about something, and they are the ones who make the decisions and run the operation. With big data in play, you are potentially asking them to make decision on data that might be telling them things that they don’t instinctively feel are right.  A typical scenario is then that they will often then question whether, in fact, it is the the data that is wrong.

Likewise, with big data you are, in many cases, turning the business on its head with regard to where it sees the role of analytics. For many organisations, analytics, historically, is something you do at the end of a process. In a data driven operation, you build analytics capabilities in right at the front end, into new product development, or right at the start of a new project, so that an essential part of the initial discussion becomes about how you are going to generate data to measure the success of it.

But can the data be wrong?

While there are times the data can be wrong, we also need to be cognisant of the fact that as humans, we more often look for the stories in the data, and we will see something that will look like the cause, when in fact it is actually a correlation. This means that you really need to have people that really understand data and the differences between cause and correlation.

It is also very human to find data sources that represent what we believe about things. This makes it essential to have a third party view of things, or to design a hypothesis that really tests the facts to determine whether, or not, they are true. So the old adage, there are lies, damn lies and statistics, does in fact require that you are really careful about data. It is essential that you take a more scientific approach to the extent that you more need to disprove your beliefs as opposed to being able to prove what you think is correct. It is key to have some proper rigour around what you do with data, rather than it being a free for all; here’s the data do with it what you will.

A further fact is that we tend to focus data on the things you think it can do well, so it is important that users be realistic in applying big data to things to which it is best suited. For example, in terms of using big data for totally blue sky innovation and creativity – on the whole, I would say it is not best suited for that. If all creative decisions are run by big data outputs, there is a risk that everything could become very prosaic, and out-of-the-same-box. However, in terms of helping identify which trends have the potential to become very successful, that is a different matter.

Take TV for example... There seems to be trends in viewing habits, where a trend comes along in the media, lasts for some while and then we jump to something totally different. Ten years ago, we didn’t have the plethora of reality TV shows we do now. I personally don’t know if you can predict the switch, but when it comes, big data certainly has the potential to analyse all the various types of feedback and see what has legs, and identify, what has the most likelihood of being a success for as long as that trend lasts. What is required is a balance between the creative and the formulaic.

So as you can see, there are a number of definite challenges around asking organisations to run themselves in a very scientific, data driven manner, and not least is the challenge of being human!