Big Data - it's complex...

We live in an information economy. The challenge that we face these days involves our ability to make sense of the overwhelming volume of information that we are required to process. Our current analytical methodologies are limited in their ability to convert this volume of data into reliable information, with an even greater stretch required to develop this information into a coherent knowledge base. As the world becomes increasingly complex and interconnected, some of our biggest challenges have begun to seem intractable. Complexity science provides new ways to look at information. By viewing multiple sources of information as a complex system, we are offered an alternate lens with which to uncover and understand a whole new set of possibilities that are both exciting and perplexing. 

How does complexity help? Complexity comes into play when many parts interact in many different ways so that the whole takes on a life of its own: it adapts and evolves in response to changing conditions. Paradoxically, what makes a “complex system” so interesting and confusing at the same time is that its collective characteristics cannot easily be predicted by analysing the many components that come together to make the system, i.e., the whole is greater than from the sum of its parts. For example, let’s take any city and consider it as a complex system. Analysis of a city’s inhabitants, traffic flows, electricity consumption, wi-fi download rates, building occupancy rates, airport noise, local weather etc, in isolation are unable to reliably provide useful information about a city’s characteristics. However, it is the combination of these interrelated sources of data and the collective patterns that they form that provide us with meaningful information about the city. These multiple sources of information are so interconnected that the collective 'mash-up' (big data) takes on a life of its own.  This attribute of a complex system is called emergent behaviour. 

The digital revolution is driving much of the increasing complexity and pace of life. Even with new computational tools and techniques to digest the ever increasing sources of big data, we will be unable to completely understand complex systems. The reason for this is that the behavior of complex systems cannot be predicted.  But what we can do however, is retrospectively understand the likelihood of certain behaviours given a combination of conditions. Therefore the challenge for us becomes fundamentally different. We are not trying to forcibly predict outcomes by holding on to a set of starting assumptions to understand an outcome through analysis, but rather align ourselves to the flow of the complex system as a living entity, and understand what it is trying to say to us.  Are you listening?