Psychologists, too, can fall for exciting stories. In the 2010s the field went through a painful self-examination known as the replication crisis.

The replication crisis

A finding is said to replicate when other researchers repeat the study and get a similar result. In 2015 a large team called the Open Science Collaboration repeated 100 published psychology studies. Only about a third produced statistically significant results again, and the effects found were on average about half the size of the originals.

Researchers identified several causes: small samples, flexible analysis and publication bias. Small studies produce noisy results. Flexible analysis means trying many ways of analyzing data and reporting the one that “works.” Publication bias means journals favored striking positive results, so failures stayed in file drawers. Reforms followed, including registering study plans in advance, sharing data, and large replications run across many laboratories.

What did not hold up

• Ego depletion. A popular 1998 study by Roy Baumeister and colleagues suggested that self-control is like a muscle that tires with use. But in 2016 a large coordinated replication across more than 20 laboratories found an effect near zero. The strong claim is now widely doubted.
• Some priming findings. A 1996 study reported that people who unscrambled sentences containing words linked to old age walked more slowly afterward. A 2012 attempt with automatic timing could not reproduce it, and several similar “behavioral priming” effects have also failed. Kahneman, who had featured such studies in his 2011 book, later acknowledged that he had relied too heavily on small studies.

What held up

Many core findings did well. In the large Many Labs project of 2014, anchoring replicated strongly. A 2020 replication of the classic prospect theory problems across 19 countries found the main patterns, including framing effects, held up broadly. Loss aversion remains debated: some researchers argue it is smaller or more dependent on context than often claimed, while others defend it. Availability, confirmation bias, hindsight bias, overconfidence and the planning fallacy all remain well supported.

One story reversed in an unexpected direction. A 1985 study concluded that the “hot hand” in basketball was an illusion. In 2018 the economists Joshua Miller and Adam Sanjurjo showed that the original method contained a subtle statistical error, and reanalysis suggests a modest hot hand may exist. Sometimes the “bias” belonged partly to the researchers.

Are heuristics really errors?

The German psychologist Gerd Gigerenzer has long criticized the heuristics and biases program for focusing on failures. His central idea is ecological rationality: a heuristic is good or bad depending on the environment it is used in. A simple rule that fits its setting can beat a complex calculation.

• The recognition heuristic. Asked which of two cities is larger, choose the one you recognize. Gigerenzer and Daniel Goldstein found that German students did about as well as, or better than, American students on pairs of American cities, because recognition tracked size. Sometimes less knowledge helps.
• The gaze heuristic. A fielder catching a high ball does not compute its path. Players appear to keep the angle of their gaze to the ball roughly steady while running, which brings them to where it lands.
• Frequency formats. Gigerenzer and colleagues found that many errors on base-rate and conjunction problems shrink sharply when problems are stated in natural frequencies, such as “10 out of 1,000 people,” instead of single-event probabilities.

Kahneman and Tversky replied that biases still appear in many realistic settings. Today most researchers see the two views as complementary: shortcuts usually serve us well and sometimes fail predictably, and the useful question is when.

Worked example: reading a famous finding today

Suppose a friend shares a video claiming that holding a “power pose,” hands on hips, for two minutes before an interview will change your hormones and make you bolder. The claim came from a small 2010 study that became hugely popular. How should you weigh it?

1. How big was the original study? The 2010 study had a few dozen participants. Small samples can produce large, eye-catching effects by chance.
2. Has it been replicated independently? A 2015 study with about 200 participants found no effect on hormones or on risk taking, although people did report feeling somewhat more powerful. One of the original authors later said publicly that she no longer believed the effects were real.
3. Which part holds up? Separate the claims. Reports of feeling more powerful have found some support; effects on hormones and behavior have not.
4. Is the effect plausible in size? A two-minute pose producing big changes in body chemistry and life outcomes should raise an eyebrow. Large claims from small interventions need strong evidence.
5. Does it fit a story we want to believe? Simple self-improvement tips are appealing, which is exactly when confirmation bias is strongest.

Apply the same questions to the biases in this course. Anchoring passes: replicated by many labs. Ego depletion fails the second test. Loss aversion passes the main tests but is still argued about in its size.

Finally, remember Gigerenzer's point: when a study shows people “failing,” ask whether the task resembles real life, where the heuristic may work well.