Exciting and Terrifying Future of Big Data

What an exciting and terrifying time we live in!

With the rapid advance in data science, big data and Internet of Things, we now are seeing a confluence of technologies that make Isacc Asimov's Psychohistory in The Foundation achievable within in a matter of years. Months perhaps.

It was about 25 years ago, I read Asimov's The Foundation. Hari Seldon, a Pyschohistorian in Asimov's book, develops the science to predict the future with advanced mathematics and mob pyschology. In the book's timeline, Psychohistory wasn't yet capable of predicting individual behavior. But the state of advancement was in the cusp of making that leap.

 Today, 2014, with the advent of the Internet of Things, we are accelerating the digitization of ordinary life. Events, patterns of behavior, health, entertainment choices, some in automatic decision modes, are captured in digital form by internet gadgets connected to the cloud. Human history has also been in digital form for at least 25 years now. Digital news was first, then chat rooms, IRC, ICQ, MySpace, Yahoo Groups, Facebook, Google+, Twitter. The social media explosion captures almost every nook and cranny of our daily lives.

 Over the last 5 years, virtualization and cloud technology has brought massive storage and computing capabilities that any Joe Schmoe with a credit card can power up at any moment. This was followed by an explosion of Big Data platforms, Hadoop, Casandra, Vertica, R, Python, etc. all heralding the desire and infant capability of computing with massive and complex data. We have been building experiences, theories and techniques to get a grip and conceptually grasp what was once just an ocean-sized blob of data.

 Along this line, consider the Big Data field as the engineering aspect of its counterpart, Data Science. Big Data creates the infrastructure to build data handling and manipulation capacity, Data Science explores the methods, theories to validate correctness, techniques to handle data efficiently, and to weave Statistics, Graph Theory, Game Theory and other Mathematics. Layer a whole dollop of Sociology, Psychology, Marketing smarts, even Linguistics on top of that to transform Big Data into Big Insight.

But the real impact to us, is that the state of this technology can now stitch together our personal data scattered from across all cyberspace and predict our idiosyncrasies. It can be empowering. I like when my Lumosity brain profile tells me where my strengths are, and maybe soon, can also tell me I'd be spectacular at a job in Target that just opened 3 minutes ago.

But it can also be creepy. That Target anecdote has become legend in Data Science circles. But closer to home, how about creating a dossier of your neighbor or their kids from their online activity? Check out PacketNinjas and Maltevo. Once you've clicked 250 likes get ready for your appointment with the Big Data Shrink who'll know you much better than your spouse.

There's too many data sets available and cross-linking them is limited only by imagination. Privacy protection has yet to catch up. Today, we have predominantly reactive privacy protections. More punitive than preventive. What are you going to do when the damage is already done? No one reads Privacy Policies. And if they did, they'd encounter vague promises filled with loopholes. We need a privacy watchdog group, much like the Better Business Bureau to tell us ordinary mortals when these opaque analytic monoliths (Facebook, Google, Microsoft, Amazon, Twitter, Yelp we're looking at you), are misbehaving. 

 To be continued..