The Replication Crisis

When Science Cannot Reproduce Its Own Results

In 2015, the Open Science Collaboration attempted to replicate 100 psychology studies published in top journals. Only 36% of the replications produced statistically significant results matching the original findings. This was not a scandal unique to psychology — similar replication rates have been found in cancer biology (25%), economics (61%), and social sciences (62%).

The implications are staggering: a significant proportion of published scientific findings may be wrong. Policies, medical treatments, and public beliefs based on these findings may rest on shaky foundations. The 'replication crisis' does not mean science is broken, but it does mean that any single study should be treated with more caution than most people and journalists give it.

Why Studies Fail to Replicate

Publication bias. Journals prefer to publish positive, surprising results. Studies that find nothing interesting are rarely published, creating a 'file drawer problem' where negative results disappear. This distorts the published literature toward false positives.

P-hacking. Researchers (often unconsciously) try multiple analyses until they find a statistically significant result. With enough variables and analysis choices, random data can produce 'significant' findings.

Small sample sizes. Studies with few participants detect effects that are often noise. The effects that pass the statistical significance threshold in small studies tend to be the most extreme — and therefore the least likely to replicate.

Perverse incentives. Academic careers depend on publishing frequently and in prestigious journals. This creates pressure to produce novel, significant findings — exactly the kind of findings most likely to be false positives.

A Worked Example: How Random Data Produces 'Significant' Findings

The conventional significance threshold, p < 0.05, means there is a 5% chance of a false positive even when nothing real is going on. Now imagine a researcher tests 20 independent hypotheses on random, meaningless data.

With 20 tests, the probability of finding at least one 'significant' result purely by chance is 1 - (0.95)^20 = about 64%. A researcher who runs many analyses and reports only the one that 'worked' will routinely publish false positives — without any deliberate fraud. This is p-hacking, and it is a central driver of the replication crisis. The defenses are pre-registration (committing to a single analysis in advance) and statistical corrections for multiple comparisons.