A club player defends their gambit with three beautiful games. Those games are real. But to know what they show, one question is missing: how many less successful games did they leave in the folder?

Survivorship bias appears when the filter making certain cases visible disappears from our reasoning. A success can be instructive without constituting a representative sample.

The aircraft that returned from missions

In 1943, Abraham Wald worked at Columbia’s Statistical Research Group on estimating aircraft vulnerability from damage to survivors. His memoranda were reprinted in 1980 by the Center for Naval Analyses.

His problem involves missing data: aircraft that are shot down do not return to provide their damage records. The hits observed on returning aircraft therefore do not directly describe the vulnerability of the entire fleet. The calculations require assumptions about damage and survival.

The lesson goes beyond “reinforce where there are no holes”. An area with few hits among survivors does not, by itself, prove that every hit there would be fatal. We must examine the selection mechanism and the model’s assumptions.

What becomes visible may have been selected

Advice from a player who succeeded

When a grandmaster describes their progress, their experience can teach you a great deal. It does not show how many players followed a comparable method without achieving the same result.

Ask what is being offered: an account, a technical explanation or an estimate of a programme’s effectiveness. An account need not be false to become misleading when turned into a guarantee.

The Polgár sisters’ success does not, by itself, constitute a controlled experiment separating every influence on playing strength. Sufficiently comparable situations would be needed. Imagining thousands of families that disappeared from the statistics does not replace those data either.

Recorded games do not represent all play

A database depends on its collection rules: competitions covered, time controls, playing strength, publication and period. Before reading an opening percentage, examine those criteria and the number of games.

A high score does not prove that a variation offers a theoretical advantage. The players choosing it, their opponents and the circumstances may differ. Conversely, a line’s absence from a book demonstrates neither its refutation nor hidden excellence.

Opening theory may also incorporate analysis of positions never played. Reducing it to spectacular published games alone would be another generalisation.

Before comparing two results

Your own gambit

Revisit your wins, draws and losses with this opening. Record the results, but also the opponent’s rating, your colour and the time control. If the sample is small, retain that uncertainty in your conclusion.

A gap between your memory and the recorded score can have several causes: selective memories, forgotten games, a recent run or simple imprecision. It is not an isolated, exact measurement of survivorship bias.

To go further, choose a few ordinary losses to analyse. They may reveal a difficulty that never appears in your three best attacks.

Sources

  • Wald, A. (1943; reprinted 1980), A Method of Estimating Plane Vulnerability Based on Damage of Survivors, Statistical Research Group, Columbia University; Center for Naval Analyses, CRC 432. Reprinted original memoranda.