Science has a measurement problem, and the numbers on it are uncomfortable. In 2015 the Open Science Collabor…

Science has a measurement problem, and the numbers on it are uncomfortable. In 2015 the Open Science Collaboration took 100 psychology studies from three top journals and ran them again, with the original materials and larger samples. 97% of the original papers reported a statistically significant result. 36% of the replications did. The effects that survived were about half the size originally reported. In 2018 Camerer and colleagues did the same for 21 social science experiments published in Nature and Science between 2010 and 2015. The two most prestigious journals in the world. 13 of 21 held up. 62%. Again, the surviving effects were roughly half the original size. Here is the part that should bother you most. Before running those replications, the team ran a prediction market where other researchers bet on which findings would hold. The market was right 86% of the time. The field already knew which results were shaky. That information existed. It just wasn't in the papers, and it wasn't in the citations those papers kept collecting. This isn't fraud. It's an incentive structure. Journals reward novelty, careers reward publication count, and nobody gets promoted for checking someone else's work. A finding surprising enough to publish is, almost by construction, more likely to be noise. Two things I keep coming back to: A single study is a hypothesis, not a fact. Effect sizes shrink as evidence accumulates, so treat the first number you see as a ceiling rather than an estimate. Replication isn't bureaucracy. It is most of the actual work, and it is the part we fund least. Sources: Open Science Collaboration, Science, 2015. Camerer et al., Nature Human Behaviour, 2018.