We spoke today with a gentleman working on a business plan for a very clever, and potentially lucrative, business. Research is required to tune the concept, and especially to predict which types of customers will be most interested. So we suggested that instead of focus groups or quantitative studies, he instead stage a small web campaign, insert a snippet of Quantcast code into the banners and landing pages, and use it to match inbound visitors to the vast data sets of user behavior online — which would pinpoint the exact demographics of the audience who likes his offer.
You know. Open the door and observe who walks through it.
Data is dangerous because it lulls us into false security — we often want to predict what will happen based on erroneous theories, and then fail to see the reality transpiring before our eyes. Wired notes this week the meltdown on Wall Street was tied to a single math formula that allowed investors a shortcut to assess hugely complex risks (um, big mistake). Our Twitter colleague Max Zeledon points out Nassim Taleb’s thesis that major events are really unpredictable; humans in hindsight try to make sense of the disorder in the universe by linking data points into logic flows, when the reality is Black Swans — things that shouldn’t exist — often just pop up. You can bend your mind thinking about the paths of fate; see Schrödinger’s cat and then ask which of yourselves is going in to work tomorrow morning.
Sometimes data can predict events, if screened carefully; Google does this beautifully with its little known Flu Trends site, collecting user Google searches for flu remedies to predict outbreaks in the United States two weeks before the Centers for Disease Control. Tel Aviv University professors are sorting Gnutella music searches by the location of consumers to predict when small bands will spike into bell curves of popularity. And in our favorite example, a simple chart comparing home prices vs. rent over the past 28 years indicates clearly that the U.S. housing market was due for a massive headache in 2009.
So are we humans vain to try to see the future, based on the data at hand? Or does randomness really make forecasting impossible? We ordered Taleb’s Black Swan tonight to learn more. Until then, we’ll keep trying.