Social science is great at making wacky, wonderful claims about the way the world—and the human mind—works. College students walk more slowly after being exposed to words relating to elderly people. Elections are determined by the outcome of college football games. Obesity is contagious, you can have business success by standing in an expansive "power pose," baseball players with a K in their name are more likely to strike out, and hurricanes with girl names are more dangerous than hurricanes with boy names.
What do the above claims all have in common? They were published in respected scientific journals, they were publicized in the news media, their publication was contingent on finding a "statistically significant" comparison from small samples. And I don't think there's good evidence for any of them.
It turns out that it’s easy for researchers to find significant effects by manipulating data—perhaps not with conscious intent. Outside researchers have tried to replicate some of these studies and failed to come up with the same results, as demonstrated in psychology’s Reproducibility Project. Many pixels have been spilled in the last few years explaining why scientists and citizens shouldn’t believe a lot of published and publicized research.
People have started to come to terms with the fact that a research team, or even an entire subfield of science, can get trapped in a loop of confirmation of noisy findings. That means you can't take published, peer-reviewed studies for granted. Statistical significance does not mean what most people think it does. And that gives us the luxury to go beyond defining the problem—and start considering solutions.
Institutional reforms such as post-publication review can help, along with the more prominent publication of replication studies, which typically get ignored in favor of splashier results. But beyond this, researchers need the freedom to fail.
In business, the freedom to fail gives people the freedom to succeed. Limited liability and bankruptcy protection provide a safe place for risk taking, a social safety net allows individual resources to be invested rather than hoarded, and a flexible education system with second and third chances gives students the opportunity to experiment. In contrast, a high-pressure environment with zero tolerance for failure encourages followership and cautious choices.
That’s what the scientific community looks like now. If you go into experimental science, your advisers and tenure committees expect publication after publication, success building upon success. But those expectations are a recipe for only the most trivial of successes. What is presented as bold, outside-the-box thinking often turns out on closer inspection to be nothing but empty theorizing accompanied by data analysis that can find "statistical significance" out of noise.
Often enough, the statistical errors that lead to those trivial successes go unrecognized—unexamined, even, in the interest of keeping publications and careers running. And when they are examined, in the form of a replication attempt, those errors still can go unadmitted. To me, one of the most frustrating aspects of the recent debate over replication is the unwillingness of researchers to admit, when their studies did not replicate, that maybe they'd been in error, that perhaps they'd improperly drawn strong conclusions from noisy data.
Statistics is hard, and there's no sin in making a mistake. But not admitting a mistake, even after your error has been shown both through statistical analysis and non-replication ... well, if you have that attitude, you're doomed to walking in circles forever.
The true heroes of science are not the technicians who are able to run an experiment on 50 people and squeeze out statistical significance, a book contract, and a TED talk. Rather, they are the people who refuse to take Yes for an answer, who test out their theories on hard problems and are willing to admit failure. You can confidently ride that horse only if you've first learned what it means to fall.
