An experiment is a question put to nature. But nature answers only the question you actually asked, and a badly designed experiment asks the wrong one. This level covers the tools scientists use to make a test fair: variables, controls, randomization, placebos and blinding. Each was developed because someone, somewhere, was fooled without it.

Variables

In an experiment, the researcher changes one thing on purpose and watches what happens to another. The thing deliberately changed is the independent variable. The thing measured is the dependent variable. Everything else that might affect the result, such as temperature, light or the age of the subjects, should be kept the same. These are controlled variables. If two things change at once, you cannot tell which one caused the effect.

The control group

To know what an intervention did, you need to know what would have happened without it. That is the job of the control group. In 1668 the Italian physician Francesco Redi tested the popular belief that maggots arise spontaneously from rotting meat. He put meat in several jars. Some he left open; others he sealed or covered with fine gauze. Flies landed on the open jars, and maggots appeared there. In the covered jars, no maggots appeared in the meat. By comparing jars that differed in only one way, Redi showed that maggots came from eggs laid by flies, not from the meat itself.

An early clinical trial

In 1747, aboard the British ship HMS Salisbury, the naval surgeon James Lind took twelve sailors suffering from scurvy, a deadly disease of long voyages. He divided them into six pairs, kept their basic diet the same, and gave each pair a different remedy: cider, dilute acid, vinegar, seawater, a spicy paste, or two oranges and a lemon a day. The pair given citrus recovered dramatically; the others did not. Lind published his findings in 1753, though it took the Royal Navy until 1795 to supply lemon juice to its fleet. The example shows both the power of a fair comparison and how slowly institutions can act on one.

Randomization

If a doctor chooses which patients get a new treatment, she may unconsciously pick the healthier ones, and the treatment will look better than it is. The solution is to assign subjects by chance. Randomization makes the groups similar, on average, in every way, including ways nobody thought to measure. A landmark 1948 British trial of the antibiotic streptomycin for tuberculosis, designed with the statistician Austin Bradford Hill, is often cited as the first modern randomized controlled trial. Today the randomized controlled trial is the standard test for new medicines.

Placebos and blinding

People often feel better simply because they believe they are being treated. This is the placebo effect. To separate it from the effect of the drug, the control group receives a placebo, such as a sugar pill, that looks identical to the real treatment.

Researchers can fool themselves too. In the early 1900s a horse in Berlin called Clever Hans seemed able to do arithmetic, tapping out answers with his hoof. In 1907 the psychologist Oskar Pfungst showed that Hans could answer only when the questioner knew the answer and was visible. The horse was reading tiny, unintended changes in posture and expression. The fix is blinding: in a single-blind study, subjects don't know which group they are in; in a double-blind study, neither the subjects nor the people treating and assessing them know.

Enough cases, and doing it again

One pair of sailors is a small sample. Chance alone can produce a striking result in a small group, so larger samples give more trustworthy answers. Results should also be repeated, ideally by different teams. The statistics of how large a sample must be are covered in STAT 101; here the key idea is simply that a single small experiment is a clue, not a conclusion.

Worked example: testing a tomato fertilizer

A garden company claims its new fertilizer boosts tomato yields by 30 percent. How would you test it fairly?

• Variables: the independent variable is the fertilizer; the dependent variable is the weight of tomatoes each plant produces.
• Control: some plants get the fertilizer; others get the same amount of plain water.
• Controlled variables: same tomato variety, soil, pot size, watering and sunlight.
• Randomization: use a coin or random number list to decide which plants get fertilizer, so you don't give it to the healthiest-looking seedlings.
• Blinding: have a friend prepare the two liquids in identical bottles labeled A and B, and weigh the harvest without knowing which is which.
• Sample size: use forty plants, not four.

Each step closes off a way of being fooled. If the fertilized plants still produce clearly more, the claim has passed a fair test. If they don't, you have saved yourself the cost of the fertilizer, and learned that a confident claim is not the same as evidence.