How much of science is truly accurate, and how much should we distrust? John Ioannidis, professor of disease prevention at the Stanford School of Medicine, spends his time trying to answer these questions - by combing through research papers to root out false, misleading and exaggerated results. The 51-year-old estimates that the majority of research results are inaccurate - which raises concerns for other scientists and doctors who rely on these results in their work, and also for the public perception of science. So what's going wrong, and how do we fix it? As a director at the Meta-Research Innovation Center at Stanford, Ioannidis looks into risk factors such as experimental error, bias and conflicts of interest to understand their role in causing false results. He's become an eminent voice on how to improve research so science can be more dependable. He talks to WIRED about how technological advancement has made it easier to make mistakes and the consequences of placing too much pressure on researchers to perform.
WIRED: What's involved in your work as a meta-researcher?
John Ioannidis: Meta-research is research about research. It asks how we perform research, how we can improve our practices and make it more efficient and more reliable so that it can be applied successfully. These are questions that arise in practically every scientific field.
You look at false results in medical research. What's their potential impact?
When it comes to issues of scientific curiosity, if you get it wrong, people will not get hurt. But scientists will still be working on concepts they shouldn't be wasting their time with. When it comes to medical applications, you can confuse people about what they need to do with their health and how to treat disease.
Some might view this as evidence of all scientists being intentionally misleading in their research. How do we overcome this misconception?
We're not talking about fraud or scientists misleading the public. We are human, so we're likely to have our biases. I think this needs to be recognised. At the same time, science is a difficult endeavour - it's difficult to make genuine, impressive discoveries, and to have them validated, and to have them change the world. What we see is that some of the factors responsible for science being successful are also responsible for some of the problems.
What do you mean by this?
One dimension of success is that science has grown very fast in terms of volume of work being done, data that can be analysed, tools and software, methods and statistical potential. The more data we get, and the more potential to generate results, the greater the chances of getting false results. In the past, someone linked smoking with cancer; that's easy to see because a smoker has a 20-fold higher chance of having lung cancer. But now, the things we're after have odds - ratios of 1:1.05, or 1:1.03; soft signals buried within a sea of noise. The challenges now are more intense and problems more difficult. You see the paradox that science is improving, but its true positive rates might be going down.
How could bias come into this?
Science is struggling to fight against bias. The more complex our analysis becomes, the more complex our protocols are and the more convoluted the space that we need to search, the more the potential for bias goes up. It's not that people are cheating; sometimes we're cheating ourselves - only because we succumb to some sort of cognitive bias when we're given the opportunity.
In a recent investigation you looked at meta-analyses - which combine the results of multiple studies and examine them - and found these overarching results were often flawed.
I kept noticing that many meta-analyses were substandard. There are different ways this happens. In clinical trials we see lots of meta-analysis that looks at the same question again and again and is often sponsored by the manufacturer of the product being assessed. This becomes more of a marketing tool; it's like producing advertisements. It creates misleading results that will probably lead to more misleading results.
You've said there's unprecedented pressure on scientists to be constantly performing. How much does this contribute to the problem?
There are so many people in science now and funding is pretty limited, which means there's a lot of competition to show they've come up with something that has tremendous potential, and eventually to oversell whatever they've come across. Journals are not helping to ameliorate that pressure - actually, they make it worse. I think people expect too much, both within and outside science. We know that science will save the world, but not every single experiment.
Nevertheless, are you optimistic about science's future?
Yes. There is huge interest in improving our research practices. As scientists, we really want to get closer to the truth, so our motive is very well aligned with getting there.
This article was originally published by WIRED UK
