This article was taken from the May 2014 issue of Wired magazine. Be the first to read Wired's articles in print before they're posted online, and get your hands on loads of additional content by <span class="s1">subscribing online.
Google predicts the spread of the flu using internet search requests it has received. Bing Travel forecasts the price of a plane ticket by analysing years of flight-reservation data. Credit score provider FICO is able to foresee whether you are likely to take your pills on time (using demographic data), and US retailer Target can spot a pregnancy (before the rest of your family does) thanks to tiny shifts in shopping behaviour. All this is based on big data, our growing ability to gain insights from comprehensive data sets. This was impossible in a small data world in which we relied on subsets of data.
The core economic point about big data is that data's true value lies in reusing it for novel purposes. Old search requests reveal the spread of the flu, old ticket prices shed light on pricing mechanisms. And previous purchases expose subtle customer-preference changes.
Reusing data used to be expensive -- because collecting, storing and analysing it was. Technology has changed that. As companies realise that most of data's value is still hidden, but can be extracted, they will hold on to it as much as they can. Those well positioned to capture and collect huge swathes of data as part of their business believe they hold a winning ticket. After all, Microsoft bought Farecast, the firm that predicted airline ticket prices, for about $100 million (£61m). But Farecast did not collect the data; it had licensed it from one of the leading travel reservation systems. Shortly thereafter, Google bought that reservation system -- and with it the data -- for about a billion dollars. Big-data ideas are lucrative, but having the data is even more so.
This may lead to a world in which even innovative big-data startups fail because they can't collect the information. And with holders of data thinking about its hidden value, startups may not get a licence to access holders' treasure troves, even if they are willing to pay. The result could be a dystopian big-data ecosystem of a few large data holders controlling the raw material that could fuel innovation.
Whenever there is a severe danger that markets fail, we need the state to come in. The obvious response is to regulate data access through standard and fair licensing terms. That's already happening in a number of cases in the US. But there is another, less draconian and more ingenious way to help big data startups to be innovative. It is to provide them with the very resource they have the biggest difficulty in obtaining: data.
Government collects lots of data -- so much in fact that it does not even know what it has. A significant portion is sensitive personal data and thus would have to be filtered before it could be made available. But huge data troves have little or no personal dimension and thus can be shared easily; think of temperature and weather data. Open-data advocates have long argued that government should make its data troves accessible -- and in the UK and elsewhere a growing number of public agencies are doing that. But the argument has mostly been that this improves transparency and public oversight, and increases public debate.
My argument is different: I suggest that government makes its data troves accessible as a "data subsidy" to facilitate big data innovation, help big data startups and thus enable the evolution of a big data ecosystem. From this perspective, open data is a powerful new tool for a nation's innovation policy. Plus, data subsidies are much cheaper than old-fashioned monetary ones -- a bonus in times of tight public budgets. This could lead to a world in which data, or at least a large chunk of it, turns into a new infrastructure that facilitates growth and provides novel insights to improve people's lives. We should give it a try.
Viktor Mayer-Schönberger is professor of internet-governance at the Oxford Internet Institute and co-author of Big Data: A Revolution That Will Transform How We Live, Work and Think
This article was originally published by WIRED UK
