The Consumer Price Index is one of the country's most closely watched economic statistics, a key measure of inflation and buying across the U.S. The trouble is that it's compiled by the U.S. government, which is still stuck in the technological dark ages. This month, the index didn't even arrive on time, thanks to the government shutdown.
David Soloff, co-founder of a San Francisco startup called Premise, believes the country needs something better. He believes we shouldn't have to rely on the creaky wheels of a government bureaucracy for our vital economic data.
"It's a half-a-billion dollars of budget allocated toward this in the U.S., and they're closed," Soloff said when I met him earlier this month during the depths of the shutdown, before questioning the effectiveness of the system even when it's up and running. "The U.S...has got a pretty highly evolved stats-gathering infrastructure [compared to other countries], but it's still kind of post-World War II old-school."
In Soloff's view, the government's highly centralized approach to analyzing the health of the economy isn't just technologically antiquated. It's doesn't take into account how much the rest of the world has been changed by technology. At Premise, the big idea is to measure economic trends around the world on a real-time, granular level, combining the best of machine learning with a small army of on-the-ground human data collectors that can gather new information about our economy as quickly as possible. This model doesn't wait a month to build a new model. After all, a price spike on one continent or food shortages on another will reverberate around the world much quicker than that.
This is a company of the Big Data Age. Soloff is a systems-obsessed data geek who about a decade ago gave up life as a Wall Street quantitative analyst -- a.k.a., a "quant" -- before diving into software startups. His co-founder, machine-learning specialist Joe Reisinger, spent six years as a researcher at Google. And other data-savvy companies such as LinkedIn and Cloudera figure in the resumés of other Premise employees. Investors include Google Ventures, Andreessen Horowitz, and Harrison Metal.
"I like the way systems work," Soloff says. "I like to see how movement on one end of it influences a reaction on the other end."
To see how such movements radiate across the global economy, Premise takes an approach he describes as "machine-human hybrid computing." The company's computers trawl more than 30,000 websites worldwide to gather data on millions of products, from pricing and availability to quality and customer ratings. But, at the same time, about 700 part-time workers following daily "shopping lists" snap smartphone photos of mostly perishable goods in physical stores and markets. These represent a huge chunk of global commerce that never gets translated to the internet.
"As you get out into the world and you move away from the sort of developed world internet, what you start to see is the amount of money people have to spend on the staples of life is obviously a lot smaller," Soloff says. "And any movements in those perishable goods and services have a massive impact on their day to day security and well-being."
For example, India in recent years has been wracked by price spikes in onions, a staple, which don't just strain households but stoke political turmoil. The sooner such trends can be identified or even anticipated, the more prepared government and business decision-makers can adapt.
The data crunched by Premise doesn't stop at the economic trends themselves, Soloff says. The platform itself is designed for maximum pattern recognition, such that the system will extract most of the data it needs from photos taken by workers in the field. Often, the only input from humans is the photo itself. Premise's software will pull not just prices from labels but identify, for example, the type, color, and size of vegetable in the photograph so that the associated data can be matched to that same variety of vegetable in another city.
At the same time, human intelligence is leveraged on the e-commerce side of the analysis. Using a mechanical turk approach, web workers help Premise identify e-commerce sites in specific markets. "This stuff is unindexed. You can't go to Google and, like, figure out everyone selling consumer products in Indonesia," Soloff says. "If I want to know where an Indian housewife is going to buy household cleaning agents online, I may know one or two sites. Finding the next 15 or 20 sites is darn near impossible unless I've got local market knowledge."
For all its dependence on a large group of workers, however, Soloff balks at the term "crowdsourcing." To feed its algorithms with the most meaningful data, Premise's system is constantly seeking to optimize its approach to sampling. Instead of looking to extract a pure signal from massive amounts of undifferentiated noise, the company tries to fine-tune by identifying the most efficient ways to deploy its collectors to get the most meaningful data for the least amount of effort. "This notion of 'yeah, I'm going to consume the Twitter fire hose and magically uncover patterns' I think is a fiction," Soloff says.
While its current product offerings are focused on inflation and food security data, Soloff could see the platform expand to answer questions that government bureaucracies don't touch, such as "How busy is a city?"
"It's our pretty firm belief that human economic activity has also been totally altered by new technologies. But the indicators that are being put forth are relics of another era," Soloff says. "So how and when and in what way does that infrastructure get upgraded? That's where we come in."
