Chapter 20: The Illusion of Validity

Type: Chapter 20

Category: Psychology

Difficulty: Intermediate

Reading time: 6 min

Core idea

During his military service in the Israeli army, Kahneman helped evaluate officer candidates. A key assessment was a leaderless group challenge: a team had to move a heavy log over a tall wall without letting the log or any team member touch the wall. Observers watched closely and rated each candidate’s leadership potential. They were confident in their ratings.

When candidates’ performance was later tracked through officer training, the ratings were essentially uncorrelated with actual performance. The assessment was nearly worthless as a predictor. When Kahneman and his colleagues were informed of this, they accepted the evidence intellectually — and continued to make confident assessments the next day, feeling exactly as certain as before.

This is the illusion of validity: the subjective experience of confidence in a prediction is generated by the coherence of the information used to make it, not by the actual predictive validity of that information. Coherence feels like validity. It is not.

Why it matters

The coherence-confidence illusion

When all the evidence available about someone points in the same direction — he is confident, he speaks clearly, the team respects him — the impression is coherent, and coherent impressions generate high confidence. The problem: coherence is a property of the story, not of the prediction. A coherent story about a candidate can be constructed from factors with low predictive validity just as easily as from factors with high validity. The confidence generated is the same in both cases.

The Israeli officer assessment example is replicated across domains: financial analysts whose predictions of stock movements are uncorrelated with actual movements; clinical interviewers whose assessments of mental health treatment outcomes don’t outperform actuarial models; sports scouts whose subjective assessments fail to beat statistical models of past performance. In each case, the professionals feel appropriately confident. In each case, the confidence is generated by coherence rather than predictive accuracy.

Stock market experts and political forecasters

Kahneman cites two major domains where the illusion of validity is especially consequential:

Stock-picking: an analysis of individual brokerage accounts found that the stocks investors sold and replaced performed about as well as the stocks they bought — implying that their trading decisions added no value on average. But the investors felt they had a skill-based edge. The trading activity that generated commissions was built on an illusion of validity.

Political forecasting: Philip Tetlock’s study of expert political forecasters found that their predictions barely outperformed random chance, and that more famous and confident experts performed no better than less prominent ones. Yet the experts remained confident, and continued to make confident predictions, because the activity of reasoning about politics generates coherent narratives that feel like insight.

Why valid intuitions exist and illusions are hard to distinguish from them

Not all expert intuitions are illusions. Chapter 22 will address when intuitions can be trusted. The key distinction: valid intuitions develop through years of practice in environments that offer rapid, accurate feedback. Surgeons develop valid intuitions about complications because they see outcomes quickly; poker players develop valid intuitions about hand strength from thousands of games. Stock-pickers and political forecasters do not — the feedback is delayed, noisy, and easily rationalized.

The problem is that the experience of confidence is identical whether the intuition is valid or illusory. Both the chess grandmaster (who has valid pattern recognition) and the analyst (who has confident impressions) feel the same quality of “knowing.”

Key takeaways

Key takeaways

• Illusion of validity: confidence in predictions is generated by the coherence of available information, not by actual predictive validity — these are entirely different things.
• Israeli officer assessment: raters who were told their predictions had near-zero validity felt as confident in their next assessment as before — the evidence did not update the feeling.
• Stock-picking illusion: traders whose sold stocks perform as well as bought stocks — implying no alpha — maintain confidence that their selections reflect skill.
• Political forecasting: Tetlock found expert forecasters barely outperform random chance; high-profile experts perform no better than low-profile ones — but confidence is uniform.
• The critical distinction: valid expert intuition requires a domain with regular, accurate, fast feedback — which most professional forecasting domains do not provide.
• The illusion is maintained by narrative rationalization: when predictions fail, explanations are available that preserve the sense of skill.

Mental model

click to zoom

The illusion of validity — how coherence generates confidence independently of actual predictive accuracy.

Read it as: Coherent information generates confident predictions whether or not the information actually predicts well. In feedback-rich domains with clear, fast outcomes, coherence sometimes reflects real pattern recognition. In forecasting domains with delayed, noisy feedback, it usually does not. The feeling of confidence is identical in both cases — which is what makes the illusion so hard to detect from the inside.

Practical application

When to trust your expert intuitions

Ask: Does this domain provide fast, accurate, and unambiguous feedback? If yes (chess, surgery, firefighting, aviation), expert intuitions may be valid. If no (stock-picking, clinical prediction from interviews, political forecasting), they are likely illusions of validity.

Institutional responses:

• Replace intuitive prediction with actuarial models where possible: Chapters 21–22 document that simple formulas typically outperform clinical experts in low-feedback domains. Using formulas removes the illusion while retaining whatever valid signal exists in the data.
• Track prediction accuracy systematically: the illusion of validity is maintained because predictions are rarely tracked against outcomes rigorously. Building a scorecard — with actual vs. predicted outcomes — breaks the illusion over time.
• Be particularly suspicious of high confidence after a coherent brief: the more perfectly a narrative fits together, the more likely it is that the coherence is doing the cognitive work rather than actual predictive validity.

Example

A management consultant advises a company on talent development. She interviews twelve high-potential employees and writes brief profiles. She predicts that three will “rise to senior leadership within five years” and rates six others as “likely to hit mid-level plateaus.” She presents these with considerable confidence.

Eight years later, two of her predicted leaders have left the company within eighteen months. One of her “plateau” predictions reached the C-suite. The correlations between her predictions and outcomes are negligible. She would say the predictions were correct given what was knowable at the time — but the Israeli officer evidence suggests: if we ran this again with the same impressions and a different random outcome draw, the correlation would remain near zero. The confidence was generated by how well her impressions fit together, not by any actual predictive signal in the interview data.

Related lessons

Related

Chapter 19: The Illusion of Understanding Hindsight bias and the narrative fallacy — the retrospective complement to the illusion of validity's prospective overconfidence. Chapter 21: Intuitions vs. Formulas The evidence that simple actuarial formulas consistently outperform expert clinical prediction in low-feedback domains. Cognitive Bias The systematic errors in judgment — overconfidence and the illusion of validity as among the most consequential.