We rarely measure everyone. Instead we study a sample to learn about a population, the whole group we care about. Polls, health surveys and many government statistics work this way. Done well, a sample of a thousand people can describe a nation of millions surprisingly accurately. Done badly, a sample of millions can be useless.

Random sampling

The key idea is the random sample, one in which every member of the population has a known, equal chance of being chosen. A cook tasting soup needs only a spoonful, provided the pot is well stirred. Random selection is the stirring: it makes the sample likely to resemble the population in every respect, including ones nobody thought to measure.

Margin of error

Even a perfect random sample will not match the population exactly, because chance decides who is picked. The margin of error describes this sampling uncertainty. For a well-run random sample of about 1,000 people, the margin of error on a percentage is about plus or minus 3 percentage points, at the usual 95 percent confidence level. If such a poll finds 52 percent support, the true figure is very likely between about 49 and 55 percent.

Three points are worth remembering:

• Precision depends mainly on the size of the sample, not of the population. A sample of 1,000 works about as well for a large country as for a large city.
• Precision improves slowly. To halve the margin of error, you need a sample about four times as large.
• The margin applies to each figure. The uncertainty about the gap between two candidates is larger, nearly double.

Sampling bias

The margin of error covers only chance. It does not cover biases from who was sampled, who answered and how the questions were worded. These can be far larger.

The classic case is the 1936 US presidential election. The magazine Literary Digest mailed about ten million ballots and received over two million replies, which pointed to a clear win for Alf Landon. Franklin Roosevelt won in a landslide, carrying 46 of 48 states. The magazine had drawn many names from sources such as telephone directories and car registrations, which in the Depression overrepresented better-off people, and those who chose to reply differed from those who did not. George Gallup, using a far smaller but more carefully chosen sample, correctly predicted Roosevelt's victory.

Online polls where anyone can click to vote suffer the same problem. People who choose to take part, often those with the strongest feelings, are not a random sample.

Nonresponse

Even a random sample is only as good as its responses. Nonresponse bias occurs when the people who answer differ systematically from those who do not. Response rates for telephone polls have fallen sharply in recent decades, often below 10 percent. Pollsters adjust by weighting, giving more weight to answers from groups that are underrepresented, but this works only if they adjust for the characteristics that matter.

Question wording

How a question is asked can change the answer. In a 1941 experiment, Americans were asked either whether the United States should “forbid” public speeches against democracy, or whether it should “not allow” them. The two mean the same, yet far more people agreed to “not allow” such speeches than to “forbid” them. Small changes in wording can shift survey answers. So can leading questions (“Don't you agree that...”), the order of questions, and the wish to give socially acceptable answers. Good surveys publish their exact questions so others can judge.

Worked example: reading a poll report

The city of Lakeford is deciding whether to build a new library. Two “polls” appear in the same week.

Poll 1 is run by a professional polling firm. It contacted a random sample of 1,000 adult residents. Results: 54 percent support the library, 46 percent oppose. The margin of error is plus or minus 3 percentage points.

Poll 2 is a website survey that 20,000 visitors chose to answer. Result: 80 percent oppose.

Which is more trustworthy? Poll 2 is twenty times larger, but its sample is self-selected: people who felt strongly enough to click, and who visit that site. Size cannot fix that bias. Poll 1 is far smaller but designed to represent the city.

How certain is Poll 1? Support of 54 percent, plus or minus 3, means true support is likely between 51 and 57 percent. The 8-point lead is likely real, but the uncertainty on the gap is roughly double the margin, about plus or minus 6 points, so the lead could be as small as about 2 points.

What did they ask? Suppose Poll 1 asked, “Do you support building a new public library?” A different poll might ask, “Do you support a property tax increase of $50 a year to build a new library?” The second question adds the cost and could produce a very different answer. Neither is wrong, but they measure different things.

Who answered? If Poll 1 reached older residents more easily than younger ones, the firm should weight the results to match the city's age profile and say so in its report.