Big data is heralded as being one of the key drivers in upcoming technological revolutions. Used by massive companies such as Amazon, Google and IBM, big data is having a transformative effect on many domains.  

Riding the exponential growth of information technologies, it is now economically feasible to turn the petabytes of previous stagnant data into something that is worth paying millions for. Viktor Mayer-Schönberger and Kenneth Cukier’s 2013 bestseller, “Big Data: A Revolution That Will Transform How We Live, Work, and Think,” attempts to parse this emerging field for both the transformative, and the potentially harmful effects of big data.

Through the relentless pace of Moore’s law, the cost of storing information has plummeted. Conversely, the burgeoning market for sensors in everyday objects has seen a rapid increase in the ability to collect that information.  Mayer-Schönberger and Cukier argue that the inexpensive storage and analysis of this data is allowing for the abolition of tried and true statistical practices. They outline how messy big data sets often outperform much smaller selective sets in real world applications.  The practice of N = all, where an entire subset is analyzed verse a small representative selection, is now not only possible, but more effective. 

Viktor Mayer-Schönberger and Kenneth Cukier
Big Data: A Revolution That Will Transform How We Live, Work, and Think
Amazon Price: $27.00 $4.99 Buy Now
(price as of Apr 18, 2015)

The authors outline interesting examples, such as how a New York City municipality was able to predict the explosion of manholes from seemingly irrelevant data, resulting in a drastic reduction of potentially harmful explosions.  They show how Google flu trends can predict the spread of the common flu using only search queries, and how Amazon is using analytics from e-readers to discover the interests of their users.

Throughout the book they stress the concept of “good enough, is good enough,” the progression of statistics away from causality towards correlation. The notion that the exact cause of a phenomenon can be impossible to discern, but an accurate prediction can still be very useful.

As we have seen, data is growing incredibly fast. This inevitably will lead to many benefits, but as the authors are quick to point out, also has the potential to have grave consequences. 

Privacy has becoming increasingly difficult to protect. The ways in which large companies can collect and use our personal data is growing rapidly. The authors outline how insurance companies use predictive analytics to assist in their risk profiles, and how governments are using data crunching algorithms to predict who is most likely to commit a crime. They show how there is a myriad of ways our data can be sold to other companies without user informed consent, raising a number of potential ethical issues.

Mr. Mayer-Schönberger and Mr. Cukier have written a comprehensive outline of both the history, and the future of big data. They present the subject in manner that is easily accessible to the layperson, and with enough intrigue to keep the pages turning.  “Big Data: A Revolution That Will Transform How We Live, Work, and Think,” will prepare you for the fast approaching reality of a world driven by big data.