Individual scientists make mistakes, cut corners and occasionally cheat. Science is trustworthy not because scientists are saints but because its institutions are built to catch errors. This level looks at those institutions, how well they work, the crisis that revealed their weaknesses, and the reforms now under way.
Peer review
Before a study appears in most scientific journals, the editor sends it to other experts, its peers, who check whether the methods are sound and the conclusions follow from the data. They may reject it, demand changes or recommend publication. Systematic outside review is younger than many people assume; it became standard at many leading journals only after the Second World War.
Not everyone welcomed it. In 1936 Albert Einstein submitted a paper to the American journal Physical Review. The editor sent it to a reviewer, who found a serious error. Einstein, used to German journals that published without outside review, angrily withdrew the paper. Yet the reviewer was right, and the corrected version, published elsewhere, reached the opposite conclusion. Even geniuses benefit from critics.
Peer review has limits. Reviewers usually cannot see the raw data or repeat the experiment. They can miss errors and fraud, and they can be biased toward famous names or exciting results. Peer review is best understood as a first filter, not a guarantee.
Replication
The stronger test is replication: another team repeating the study to see whether it gets the same result. A finding that holds up across independent replications is far more trustworthy than a single report. When replications fail, the original result is cast into doubt, whatever journal published it.
The replication crisis
In 2015 a project called the Open Science Collaboration, involving some 270 researchers, tried to repeat 100 studies published in leading psychology journals. While 97 percent of the original studies had reported significant results, only about 36 percent of the replications did. Similar problems turned up in cancer biology, economics and other fields. Several causes were identified:
- Publication bias: journals preferred exciting positive results, so failures went unpublished in desk drawers.
- Flexible analysis: researchers could try many ways of analyzing data and report the one that “worked,” often without realizing they were fooling themselves.
- Small samples: studies too small to detect real effects reliably produced many false alarms.
- Incentives: careers rewarded many novel publications more than careful, slower work.
Error and fraud
In March 1989 the chemists Martin Fleischmann and Stanley Pons announced at a press conference in Utah that they had achieved nuclear fusion at room temperature, a potential source of almost limitless energy. Laboratories worldwide rushed to repeat the experiment. Most failed, and within months “cold fusion” was widely rejected. This was honest error, caught quickly by replication.
Fraud is rarer but more damaging. In 1998 the British doctor Andrew Wakefield and colleagues published a paper in The Lancet suggesting a link between the measles, mumps and rubella (MMR) vaccine and autism. Large studies found no such link, investigations found that the work was dishonest, and in 2010 The Lancet fully retracted the paper and Wakefield was struck off the UK medical register. By then, fear of the vaccine had spread, vaccination rates had fallen and measles had returned in places where it had been rare. Correction works, but it can be slower than the damage.
Self-correction at its best
In 2011 physicists on the OPERA experiment measured neutrinos apparently arriving about 60 billionths of a second faster than light should allow. Instead of claiming a revolution, they published the data and asked other physicists to help find the mistake. In 2012 the cause was traced to equipment problems, including a loose fiber-optic cable connection. The episode is a model of how science should handle surprising results.
The reforms
The replication crisis prompted real change. Preregistration asks researchers to publish their hypotheses and analysis plans before collecting data, so they cannot quietly change them afterward. Some journals now offer registered reports, agreeing to publish a study based on its design, whatever the results. More journals require researchers to share data and code. Large replication projects are now treated as valuable science in their own right.
Worked example: the chocolate diet
In 2015 the journalist John Bohannon ran a deliberate experiment on the system. With colleagues, he conducted a real but tiny diet study with a handful of volunteers, measured eighteen different outcomes, and reported the one that happened to look good: eating chocolate seemed to speed weight loss. The paper was published in a journal that performed little real review, and news outlets around the world reported it before Bohannon revealed the hoax. What warning signs could a careful reader have spotted?
- Tiny sample: a few people per group can produce striking flukes.
- Many measures: measure enough things and one will look significant by chance.
- No replication: a single surprising study, not a body of evidence.
- Weak venue: a journal willing to publish almost anything for a fee.
The lesson is not that science cannot be trusted. It is that trust should track the evidence: a large body of replicated research deserves confidence; one small, surprising study deserves interest and patience.