These and other similar headlines followed the results of a large-scale initiative called the Reproducibility Project, recently published in Science magazine, which appeared to show that a majority of findings from a sample of 100 psychology studies did not hold up when independent labs attempted to replicate them. (A similar initiative is underway in cancer biology and other fields: Challenges with replication are not unique to psychology.)

Headlines tend to run a little hot. So the media’s dramatic response to the Science paper was not entirely surprising given the way these stories typically go. As it stands, though, it is not at all clear what these replications mean. What the experiments actually yielded in most cases was a different statistical value or a smaller effect-size estimate compared with the original studies, rather than positive evidence against the existence of the underlying phenomenon.

This is an important distinction. Although it would be nice if it were otherwise, the data points we collect in psychology don’t just hold up signs saying, "there’s an effect here" or "there isn’t one." Instead, we have to make inferences based on statistical estimates, and we should expect those estimates to vary over time. In the typical scenario, an initial estimate turns out to be on the high end (that’s why it ends up getting published in the first place — it looks impressive), and then subsequent estimates are a bit more down to earth.