The problem with "Big Data"

You can't help but get caught up in all the excitement surrounding "Big Data". After all, it's the current big thing - the revolution that will change business forever. What is "Big Data" and is it all it's cracked up to be?

What is Big Data?

Data so big and/or rich that conventional tools and techniques used to perform analysis are rendered useless

We now get data from an incredible number of different sources, and it's relentless; website analytics, social media followers/likes/favourites, browsing habits, customer relationship management (CRM), sales pipelines, purchase histories, sensors, email, phones, etc. etc.

"Big Data" is all about trying to find relationships and correlations in all this disparate data so we can better understand our company and our customers.

It's not about the quantity of the data, but the quality of the data that counts

A common misconception is "Big Data" is all about gathering and storing huge amounts of data. In reality "Big Data" is the revolution of specialised tools and techniques for shifting through large amounts of data and uncovering previously unseen insights.

What people don't tell you about "Big Data"

Big data is like looking into a wishing well, only to find none of the coins you have are big enough!

Much of the talk about "Big Data" revolves around the type of data you should collect and how to collect it e.g. Google Analytics and purchase histories.

Far less is written on what to actually do with the data once you have it. That's because with "Big Data", performing the analysis is the difficult part. It's not like you can load the data into an Excel spreadsheet and start crunching some numbers - for one thing, Excel 2013 has a row limit of 1,048,576...hardly big data size!

In addition, because the analysis of "Big Data" involves unstructured data, the statistical relationships that are uncovered cannot be easily verified and as such you need to use extreme care with any results and conclusions.

Processing power!

Many of the tools and techniques* used in "Big Data" have been around for a long time. One of the reasons many people don't know about them is due to a totally different problem - processing power.

Training systems to find hidden patterns in data takes a long time and demands a lot of resources

Back in the early 2000's I set up a number of experiments to teach a computer the underlying syntax of the English language. By today's standards, the data sets were tiny (some 3000 sentences), but these took nearly 2 weeks for the computer to learn!

Things have come a long way since then (better algorithms and a huge growth in processing power). Even so, only the biggest companies in the world have the computing resources available to really take advantage of "Big Data".

Where does that leave us?

The reality of "Big Data" is still beyond most companies, but not for long. In an amazing show of openness, Google, Microsoft and now Amazon, have all made available tools that can be used to analyse disparate data sets.

We are on the verge of a computing revolution that will drastically change the way in which companies learn and interact with their customers. Exciting times!

*If there's sufficient interest, I'm happy to write about the tools and techniques used - please leave your comments.

This article has been cross posted on koolth's blog. John is available to speak at conventions, conferences, groups and events, so please get in touch to enquire or make a booking.