Back and Forth on the Value of Replication

From Stuart:

My friend Jordan Dworkin recently wrote an excellent piece titled, “How Much Should We Spend on Scientific Replication?” It is the first attempt to model the probability that funding replication studies will be more impactful than just funding new scientific studies.

I saw the piece ahead of time, of course, but after doing a little more rumination, I have a couple of thoughts that I should have offered earlier.

First, Dworkin says that we should prioritize replicating new studies, based on the typical pattern of how citations accumulate:

First is when the replication happens — the attention a paper receives, and the extent to which it drives follow-on research, are time-dependent. Papers tend to accrue ~2.5% of their total citations in the first year after publication, 7.5% in the second, and ~12% each in years 3-5; by the sixth year post-publication, the average paper has already received almost half of the direct attention it ever will. Because the capacity of a failed replication to reduce a study’s impact depends on when the replication occurs, we should not prioritize replicating studies that have already accrued a lot of citations — at that point, it may be too late to capture most of the replication’s value. Instead, we should aim to replicate papers that are likely to accrue many citations in the future.

I’m not sure I agree with this assumption that the value of replication is mainly in heading off citations early on in a paper’s lifetime.

The famous 2006 Alzheimer’s paper that turned out to be likely fraudulent was cited over 3,600 times according to Google Scholar. 414 of those citations occurred since 2021, starting 15 years after the paper was published. Here’s the historical pattern (and keep in mind that mid-2022 was when the potential fraud was uncovered):

I think it’s fair to say this was still an influential paper, 15+ years after publication, and the full extent of its influence wasn’t just in the direct citations but in all the follow-on papers and grants, many of which might not have cited the 2006 paper directly:

I think there’s something to the following stylized model of how a lot of science operates:

Paper A comes out in Year 1. It gets a fair bit of attention early on, because it does something new/surprising/useful. Papers B, C, and D then build upon it in the next year