The Complete Statistical Fallacy Checklist: 55 Mental Debug Maps
A quick-reference index to all 55 statistical fallacies, organized into 8 categories. Designing a sample? Check Category I. Making causal claims? Check Category IV. Running a model? Check Category VI. One line per fallacy — click through for the full story.
This checklist is a quick-reference index for the 55 Statistical Fallacies series. Usage is simple:
- Designing a sampling plan → scan I. Sampling Bias
- Designing a survey or measurement tool → scan II. Measurement Bias
- Interpreting ratios, averages, or aggregated numbers → scan III. Numerical Intuition Traps
- Trying to infer causation from correlation → scan IV. Causal Inference
- Making analytical decisions, feeling objective → scan V. Cognitive Bias
- Running regressions, models, or ML → scan VI. Statistical Methods
- Designing A/B tests or experiments → scan VII. Experiment Design
- Preparing or reading a report → scan VIII. Presentation & Reporting
I. Sampling Bias (6 fallacies)
Your sample does not represent your target population
| # | Fallacy | One-liner |
|---|---|---|
| 1 | Survivorship Bias | You only see the cases that survived; the failures are invisible |
| 2 | Sample Selection Bias | Your data collection process has a built-in filter that systematically skews the sample |
| 3 | Coverage Bias | Your sampling frame never reached certain groups at all |
| 4 | Self-Selection Bias | Only people with strong feelings respond; the silent majority is not in the sample |
| 5 | Convenience Sampling Bias | The easiest people to reach are not the same as your actual users |
| 6 | Time Window Bias | Sampling at a specific time captures only who was present at that moment |
II. Measurement Bias (6 fallacies)
What you measured is not what you thought you measured
| # | Fallacy | One-liner |
|---|---|---|
| 7 | Social Desirability Bias | Respondents answer with what society expects, not what they actually think |
| 8 | Observer Bias | The measurer’s expectations contaminate the measurement |
| 9 | Recall Bias | Relying on memory instead of records; human memory reconstructs rather than replays |
| 10 | Instrument & Measurement Error | The measurement tool itself introduces systematic error |
| 11 | Confirmation Bias in Collection | Only recording observations that confirm expectations; the database is selective from the start |
| 12 | Temporal & Seasonal Bias | Treating seasonal fluctuations as the result of your own efforts |
III. Numerical Intuition Traps (6 fallacies)
Mathematical results that defy intuition
| # | Fallacy | One-liner |
|---|---|---|
| 13 | Base Rate Fallacy | Dazzled by 99% accuracy, forgetting that the base rate is only 1% |
| 14 | Gambler’s Fallacy | Thinking that after 10 heads in a row, tails is “due” |
| 15 | Law of Small Numbers | 5 user interviews do not mean 80% of users want a feature |
| 16 | Simpson’s Paradox | Every subgroup improves, yet the overall result gets worse |
| 17 | Ecological Fallacy | Wealthy areas have longer lifespans — that does not mean moving there will make you live longer |
| 18 | Atomistic Fallacy | A teaching method that worked in one school does not mean it works for the whole district |
IV. Causal Inference (5 fallacies)
Correlation is not causation
| # | Fallacy | One-liner |
|---|---|---|
| 19 | Confounding Factor | A third variable simultaneously causes both X and Y, creating a spurious correlation |
| 20 | Reverse Causality | The causal direction is reversed — more police does not cause more crime |
| 21 | Collider Bias | Two conditions jointly filter your sample, manufacturing a correlation that does not exist |
| 22 | Spurious Correlation | Ice cream sales correlate with drowning rates, but ice cream does not cause drowning |
| 23 | Mediation Fallacy | Controlling for the mediator on the causal path severs the very relationship you were studying |
V. Cognitive Bias (7 fallacies)
The brain’s systematic shortcuts lead you astray
| # | Fallacy | One-liner |
|---|---|---|
| 24 | Confirmation Bias (Analysis Phase) | Only seeking information that supports your prior beliefs, actively avoiding counterevidence |
| 25 | Anchoring Bias | The first number you see constrains every subsequent judgment |
| 26 | Availability Heuristic | What comes to mind easily does not mean it is more likely to happen |
| 27 | Representativeness Heuristic | Judging probability by how well something “matches a stereotype,” ignoring base rates |
| 28 | McNamara Fallacy | Measuring only what can be quantified, ignoring the unmeasurable but important |
| 29 | Goodhart’s Law | Once a metric becomes a target, it is no longer a good metric |
| 30 | Path Dependence | Sunk costs keep you on the wrong path |
VI. Statistical Methods (11 fallacies)
The tool’s assumptions were violated, or you misread its output
| # | Fallacy | One-liner |
|---|---|---|
| 31 | Regression to the Mean | Treating extreme values naturally reverting to the mean as evidence your intervention worked |
| 32 | Multicollinearity | Predictor variables are highly correlated; coefficient estimates become unstable |
| 33 | Omitted Variable Bias | Leaving out a key variable; the model misattributes its influence to other variables |
| 34 | Overfitting | Memorized the noise in training data; performs terribly on new data |
| 35 | Data Leakage | Used information during training that would be unavailable in the prediction context |
| 36 | Look-ahead Bias | Used future data in backtesting that could not have been known at the time |
| 37 | Extrapolation Bias | Extended the model beyond its training range; conclusions are unreliable |
| 38 | P-value Misinterpretation | p = 0.03 does not mean “there’s a 97% chance the new version is better” |
| 39 | Effect Size Neglect | Statistically significant but tiny effect; no practical meaning |
| 40 | Underpowered Study | Sample size too small to detect a real effect that exists |
| 41 | Multiple Comparisons | Run enough tests and you will inevitably get a p < 0.05 by chance |
VII. Experiment Design (9 fallacies)
The experiment was rigged from the start
| # | Fallacy | One-liner |
|---|---|---|
| 42 | Hawthorne Effect | Subjects who know they are being observed change their behavior |
| 43 | Placebo Effect | Believing the intervention works produces real subjective improvement |
| 44 | Experimenter Expectancy Effect | The researcher’s subtle cues signal the “correct answer” to subjects |
| 45 | Intervention Bias | The treatment group and control group were treated differently in ways beyond the variable |
| 46 | Non-Response Bias | Non-responders are a fundamentally different population from responders |
| 47 | Questionnaire Bias | The wording and structure of questions steers the direction of answers |
| 48 | Information Bias | The data labels themselves are wrong; annotators’ biases corrupt the training foundation |
| 49 | Detection Bias | Looking harder finds more, but detection rate does not equal occurrence rate |
| 50 | Exclusion Bias | Removing “outliers” may discard the most important signal as noise |
VIII. Presentation & Reporting (5 fallacies)
The data itself is fine — what you choose to show is the problem
| # | Fallacy | One-liner |
|---|---|---|
| 51 | Truncated Y-Axis | Y-axis does not start at zero; tiny changes look like rockets launching |
| 52 | Dual-Axis Manipulation | Two different Y-axis scales make unrelated variables look perfectly synchronized |
| 53 | Cherry-Picking / Texas Sharpshooter Fallacy | Only showing successful results, hiding failed attempts |
| 54 | File Drawer Problem | Negative results are locked in a drawer and never published; public conclusions are systematically skewed positive |
| 55 | Publication Bias | The entire publication system favors accepting research with significant positive results |
