It seems like data is changing everything we do.
Data analysis has transformed biological research over the past decade. It has reinvented the business world by way of "big data" software platforms along the lines of Hadoop, an open source tool originally built by Yahoo and Facebook. It's even changing historical studies, thanks to a movement called Cliodynamics.
Cliodynamics is a field of study created by Peter Turchin in the early 2000s. The idea is to use data as a means of predicting the future, but also as a way of testing theories about what happened in the past. You build a model that seeks to explain history, and then you test this model using real historical data.
The movement's latest aim is to analyze the origins of complex societies. In a paper published today in Proceedings of the National Academy of Sciences, Turchin and a trans-disciplinary team from the University of Connecticut, University of Exeter, and the National Institute for Mathematical and Biological Synthesis attempt to overturn the long standing belief that large-scale states are the product of agriculture.
Early humans were hunter-gatherers. They had relatively simple social structures, which consisted of perhaps a few dozen people, all of whom knew each other, and they didn't engage in complex cooperative tasks. But eventually, complex societies evolved -- complete with governments, armies, agriculture, education, and other large scale, cooperative projects. With their paper, Turchin and his collaborators analyzed the spread of the social norms that allowed societies to expand across millions of people.
"You cannot have a large state without bureaucrats, but bureaucrats are expensive. You have to pay them," he says. "So the big question is how do complex societies evolve when they are so expensive?"
The standard theory, which Turchin calls the "bottom up" theory, is that humans invented agriculture around 10,000 years ago, providing resource surpluses that freed people up for other ventures. But what Turchin and his team have found is that the bottom-up theory is wrong, or at least incomplete. "Competitions between societies, which historically took the form of warfare, drive the evolution of complex societies," he says.
To test the two competing theories, Turchin and company designed two mathematical models for predicting the spread of complex societies. One based only on agriculture, ecology and geography. The other included those three factors, plus warfare. Then, they used data from historical atlases to determine whether these models matched up with the way the different states and empires actually evolved.
The model that included warfare predicted about 65 percent of the historical variance, while the agricultural model explained only about 16 percent, suggesting that warfare was more important in the spread of social norms that lead to complex societies.
Turchin admits that the model is far from perfect -- it includes no population data, for example -- but for the most part, it was able to predict the spread of large-scale states between 1,500 BC to 1,500 AD. He also notes that whether or not simple societies were warlike is hugely controversial, but says that by the time their models start, warfare was widespread. "Proximate causes for warfare are numerous: competition for resources (mainly territory), revenge and strategic consideration (attack your enemy before they are ready to attack you)," he says.
He adds, however, that agriculture is also a part of how complex societies evolve, and he and his team are already working on their next research project, which will involve crop yield data. Economic and ideological competition, he says, can help mold societies too. For example, the competition between the United States and the Soviet Union.
"Back in the 1920s in the United States, we had pretty naked capitalism. Workers were expected to get paid whatever they could get," he says. "But then the Soviet Union came on the scene, and suggested that workers should get paid more."
He says that during the original "Red Scare" in the 20's, corporations -- fearing a large scale shift to communism in the U.S. -- voluntarily began paying higher wages and implementing more social programs, such as pensions. But a few decades later, economic competition forced Russia to allow more free trade and democracy.
But the biggest driver? War.
