Big Data 101

“Information is not knowledge” –Albert Einstein

Google now shows me 54 million results when searched for "big data"... Yet it seems the importance is getting lost in the hype for such a life changer technology. I was at a talk where many CTOs of big enterprises were present. When asked who invested in big data technologies in 2015 almost no hand went up. I think we all need to take another look at what this technology means for us. 

The idea that we use and analyze all available data to make better informed decisions, and to learn more about just about anything, has been around since we have had the word data. Even the very definition of data, as it’s used in daily life, is, facts and statistics collected together for reference or analysis. In this sense, one could think that data existed even before the word itself.

Certainly, until computers came along what data we could collect would only be limited to what we could remember or write down. Consequently, what we could process and analyze of this written data was also very limited, in hindsight.

Then, the computers came along. Today, computing is everywhere in different shapes and forms, even on the go. On top of it, in the very near future, with Internet of Things (IoT), it’s becoming hard to imagine where and what they are not going to be.

All these computers in our personal and business lives, in different forms, from hand held devices to being integrated into factory machines, didn’t just enable us to process collected data, like we initially designed them to do. Computers, serving as a communication tool with the addition of Internet technology, also allowed us to generate and collect much more data. Looking at the numbers, zettabytes a year worth more data. IBM says we are now generating several quintillion bytes of data every day. Moreoever, we haven’t seen anything yet, with IoT, this growth of data is exponential, to say the least.

We now have a word for this massive amounts of data we are exponentially collecting and looking at. It’s called big data. Reading for a little bit after a simple search on any search engine about the subject will reveal why they call Big Data, “the new oil”. It’s use and applications are not yet all seen but it’s already obvious Big Data will be related to every aspect of what we do. This new massive collection and analyzing of data is here to shape our future.

Looking at big data, helping us make important decisions and reshape our day to day personal and business lives is no small matter. And, definitely, when it comes to data science and making use of all the available data, we have to think even bigger then big data as its currently generally thought of. While big data right now refers to extremely large data sets we have so much more useful data that might not look like big data at first, in smaller chunks. It should be obvious that it’s not always the size of the data set that qualifies what is worth analyzing. There is no reason to limit big data science to only extremely large data sets. We can apply the same science to many different data sets, big and small. In the end what matters is simply this, “is the data, when analyzed, making an individual, an organization, a machine, or a process, more informed and smarter?”. If the answer is yes, then that data is important enough to study, even if it’s not a massive data set. Big data or smaller sets, any data worth analyzing to make things and people smarter is worth paying a lot of attention.

In business, big or small, smart data can help us be smarter, much smarter. Some businesses use customer relationship management software to collect data and react better to sales related inquiries. Some businesses look at behaviors of employees to create a better working environment.

Examples are already piling up. However, even now with some real world use cases at hand, we are only making use of the data we have at a very limited scale. Look around you! Everything is data! And it can all be used to make a better world! This might seem like an exaggeration to you but I promise you it's not. We are all instinctually aware this to be true. We are even collecting data in our smart devices about ourselves, our health, our fitness. We know this is how we can get better. In fact, this is true for just about anything, especially for business.

We value being smart as individuals, showing intelligence and good judgment. This is exponentially important for an organization formed of many individuals, and machines. It's time for many more businesses to get on the path to a smart data driven business.