Most real beliefs are not certain. You are fairly sure it will not rain tomorrow, quite sure your car will start, and unsure who will win the next election. A powerful idea in modern epistemology is that belief comes in degrees, often called credences, which can be expressed as probabilities from 0 (certainly false) to 1 (certainly true). The question becomes: how confident should you be, given your evidence?
Proportioning belief to evidence
David Hume wrote in An Enquiry Concerning Human Understanding (1748) that “a wise man proportions his belief to the evidence.” Strong evidence should produce strong confidence; weak or mixed evidence, tentative confidence. Treating every claim as either certain or worthless throws information away.
Calibration
A person is well calibrated if their confidence matches how often they are right. Of all the things a calibrated person says they are 70 percent sure of, about 70 percent turn out true. The political scientist Philip Tetlock's long studies of forecasting found that many experts were poorly calibrated, often doing little better than chance on long-range political predictions, while some ordinary volunteers trained to think in probabilities did much better. Calibration improves with practice and feedback. The most common failure is overconfidence: people who say they are 90 percent sure are often right much less often than that.
Base rates
One of the most important ideas in reasoning about evidence is the base rate: how common something is in the first place, before considering the specific evidence. Ignoring it is called base rate neglect.
A classic example: a disease affects 1 in 1,000 people. A test catches 99 percent of people who have it, and wrongly flags 5 percent of people who do not. You test positive. How likely is it that you have the disease? Many people guess about 95 percent. Work it through with 100,000 people:
- 100 have the disease, and about 99 of them test positive.
- 99,900 do not, and 5 percent of them, about 4,995, also test positive.
- So about 5,094 people test positive, and only 99 of them are sick: about 2 percent.
The test is useful (it raised the chance from 0.1 percent to about 2 percent), but because the disease is rare, most positive results are false alarms.
Bayesian thinking in plain words
The English minister and mathematician Thomas Bayes worked out the rule behind this calculation; his essay was published in 1763, after his death. In plain words, Bayes' theorem says:
- Start with your prior: how likely the claim was before the new evidence (often the base rate).
- Ask how much more likely the evidence would be if the claim were true than if it were false. This ratio measures the strength of the evidence.
- Combine the two into a posterior: your updated confidence.
Lessons follow. Evidence that would be just as likely either way should not change your confidence. Claims that start with very low priors need very strong evidence. And no single piece of evidence settles everything; beliefs should move step by step as evidence accumulates.
Common traps
- Confirmation bias: noticing and remembering evidence that supports what we already believe, and discounting the rest.
- Anecdotes over data: one vivid story can feel stronger than statistics about thousands of cases.
- Absence of evidence: failing to find something is evidence it is not there only if you would probably have found it if it were.
Worked example: the miracle supplement
An advertisement claims a new supplement improves memory. It features twenty glowing testimonials and a doctor's endorsement. How confident should you be that it works?
Start with the prior. Most supplements marketed for memory have not been shown to work in careful trials, so a reasonable starting confidence is low.
Ask how strong the evidence is. Would twenty glowing testimonials exist if the supplement did nothing? Almost certainly. Any product sold to thousands of people will produce some enthusiastic users, through chance, the placebo effect, or memory improving for other reasons, and advertisers choose the best stories. Since this evidence is about as likely whether or not the supplement works, it should barely move your confidence. The same goes for a paid endorsement.
Ask what strong evidence would look like. In a randomized controlled trial, people are assigned by chance to take the supplement or a look-alike placebo, without knowing which, and their memory is then measured. A benefit in such a trial is much more likely if the supplement works than if it does not. A positive result, especially one repeated by independent researchers, should move you a lot.
Update. Without such a trial, your confidence should stay close to the low prior. This is not closed-mindedness; it is proportioning belief to the evidence. If good trials appear, you should change your mind, and the same reasoning tells you by how much.