Chapter 10: The Law of Small Numbers

Type: Chapter 10

Category: Psychology

Difficulty: Intermediate

Reading time: 6 min

Core idea

A survey of kidney cancer rates across all 3,141 US counties reveals a striking pattern: the counties with the lowest cancer incidence are mostly rural, sparsely populated, and located in Republican-leaning Midwestern and Southern states. Easy enough to explain — clean air, fresh food, no pollution. But the counties with the highest cancer rates are also mostly rural, sparsely populated, and located in the same Republican-leaning states. The rural lifestyle cannot explain both extremes.

The actual explanation has nothing to do with lifestyle. Rural counties have small populations. Small samples produce extreme outcomes — both very high and very low — by random chance. There is no phenomenon to explain. The pattern is a statistical artifact, not a causal fact. But System 1 cannot help searching for a cause, because System 1 does not understand statistics.

Kahneman and Tversky called this pattern the law of small numbers: people behave as if the law of large numbers — which guarantees that large samples reliably represent the population — applies to small samples too. It does not.

Why it matters

The bias toward causal thinking

System 1 is built to find causes. When presented with data, it automatically asks: What generated this pattern? This is usually the right question — most patterns in the physical and social world do have causes. But for purely statistical patterns generated by random sampling, the question has no good answer. There is no cause. Yet System 1 will find one anyway, and the causal story will feel completely satisfying.

The kidney cancer example demonstrates how completely System 1’s causal reflex overrides statistical awareness. Even trained researchers fall into it. When told about high-incidence rural counties, they generate vivid stories about poverty, diet, and lack of medical access. When told about low-incidence rural counties, they generate equally vivid stories about clean living and fresh food. Both stories feel true. Neither is the explanation.

The research design problem

The same bias infects scientific research design. Researchers who study real effects often design studies that are too small to reliably detect those effects — they believe in the law of small numbers. They expect a small experiment to reproduce population-level findings with high fidelity. When a small study yields a strong result, they report it as confirmation; when it yields a weak result, they often redesign or add participants. This produces systematic publication bias and inflated effect size estimates in the literature.

Kahneman and Tversky found that even statisticians — professionals whose job is to understand sampling — systematically over-trusted small samples in their intuitive judgments, even when they knew the correct statistical answer.

What “merely statistical” facts feel like

Statistical facts are facts about probability distributions, not about particular cases. That rural counties have higher variance in cancer rates is a fact about sampling, not about any specific county. System 1 cannot represent this kind of fact intuitively. It converts statistical variability into a causal story, every time.

The result: people over-interpret noise. Every run of extreme results in a small sample gets explained, when the correct response is to wait for more data.

Key takeaways

Key takeaways

• The law of small numbers fallacy: people treat small samples as if they reliably represent the population — expecting extreme results to carry causal meaning when they are sampling artifacts.
• The kidney cancer example: both highest and lowest cancer-rate counties are rural and small-population — not because of lifestyle, but because small samples produce high variance by chance.
• System 1 cannot represent 'merely statistical' facts — it converts random variation into causal narratives, generating explanations for patterns that require no explanation.
• Researchers design underpowered studies because they over-trust small samples, producing inflated effect sizes and publication bias in the scientific literature.
• The intuitive bias toward small-sample confidence is not corrected by statistical education — statisticians show the same bias in their informal judgments.
• Practical correction: when extreme results come from small groups or short time periods, ask 'how large is the sample?' before generating causal explanations.

Mental model

click to zoom

How small sample size produces extreme outcomes and how System 1 misinterprets the pattern as causal.

Read it as: Large samples track the true population rate reliably. Small samples produce extreme outcomes purely by chance — some counties will always look like outliers. When System 1 sees the extreme result, it generates a causal story. The story feels convincing. It is wrong. The correct response to extreme results from small samples is skepticism, not explanation.

Practical application

The antidote to the law of small numbers is asking the sample size question before generating explanations.

Before explaining an extreme result

Ask: How many observations does this result come from? Is this the top- or bottom-ranked group from a large set? If yes, expect regression to the mean — the next measurement will likely be closer to average. An explanation of the extreme result is probably explaining noise.

Specific failure modes to watch for:

• School rankings: The best-performing schools in most rankings are small schools. So are the worst. Small schools appear at both extremes — not because small schools are better or worse, but because small samples have high variance.
• Winning investment funds: A fund that outperformed for three years from a universe of 5,000 funds is not surprising. With enough funds, some will win by luck. Designing an explanation of the winning strategy is explaining a random outcome.
• Early clinical trial results: Phase I/II drug trials use small samples. Promising results in small trials frequently fail to replicate in Phase III. The initial result was a small-sample artifact, not a signal about the drug.

Example

A school district ranks all 35 elementary schools by reading test scores. The top 3 schools all have fewer than 150 students. The district superintendent proposes adopting the pedagogical methods used in those top schools across the district.

But 28 of the 35 schools have 400+ students, and the 7 smallest schools appear at both ends of the ranking — 3 at the top, 3 at the bottom, 1 near the middle. The pattern is sampling variance, not educational excellence. The superintendent is preparing to scale up a noise artifact. Before initiating a reform, the district should run the analysis again next year to see whether the same small schools dominate — if it is a real effect, they will; if it is sampling variance, different small schools will appear at the extremes.

Related lessons

Related

Chapter 9: Answering an Easier Question Heuristic substitution — the mechanism by which System 1 substitutes easy judgments (causal stories) for hard ones (statistical reasoning). Chapter 16: Causes Trump Statistics When a compelling causal narrative is available, statistical base rates are ignored — the deepest consequence of System 1's causal bias. Cognitive Bias The systematic errors in judgment that the law of small numbers fallacy exemplifies.