“Big data” has been a buzzword for the last decade in Silicon Valley. Investors and tech companies, from little-known startups to corporate giants, have poured billions of dollars into software and computer systems that promise to pore through mountains of information and glean useful insights into business trends or consumer behavior. Big data has opened the door to do things that weren’t possible before, enabling the collection of vast stockpiles of information while advances in computing hardware and online networking have made it possible to run more sophisticated analytical programs and crunch bigger sets of data more quickly.
Technology has filled people’s lives with crowd-sourced, data-driven metrics and left many convinced of their validity. We look to Yelp rankings to find a decent restaurant, TripAdvisor to gauge the quality of accommodations, and Netflix to see which shows to watch. Amazon, Google, Facebook — all are ubiquitous presences in everyday life with data at their core. Many of us, without question, accept the information that these sources provide. And therein lies a failure.
The vast majority of pundits and pollsters predicted a Democratic presidency. Then on the evening of November 8, their predictions were turned upside down. But Tuesday was not a failure of data; it was a failure of forecasting and analysis by humans. The data was as good as it could be, but the critical thinking behind the analysis was missing.
The fact remains that no algorithm, no matter how advanced, can replace the intuition and the capacity for judgement that is uniquely reserved for the human brain. Data science is a nascent technology with trade-offs. It can provide incredible insights, but can also be a blunt instrument missing context and nuance. While technology can crunch data faster and with greater accuracy than people, that should not imply that it is better or can provide a more accurate version of the truth. In the end, it wasn’t Big Data that failed in the election, it was the humans who never questioned it.