Who’s to Blame for the Reproducibility Crisis?

Earlier this year, in this very magazine (1), I described the apparent lack of reproducibility in today’s scientific literature, especially in the fields of biology, biomedicine and/or analytical science. But who is to blame? And what can we do to turn the tide?

First – a technical point: using antibodies of a commercial nature can scupper reproducibility before an experiment even begins. It’s likely that researchers with limited control over how their antibodies are generated are simply unaware that such reagents are not reproducibly manufactured, packaged or shipped. Neither the manufacturers nor users of such reagents are performing QC or QA to demonstrate reproducibility in the final antibodies. Thus, we can apportion some blame to those manufacturing the reagents; however, anyone versed in analytical method validation (AMV) will know that QC procedures require demonstration of reproducibility (method robustness and ruggedness). It appears that this is rarely done for most biochemical, biological, immunological, biomedical and related studies. What about analytical studies?

In general, AMV is not formally taught at either the undergraduate or graduate levels. Instead, it has to be “self-taught” by reading the relevant texts or review articles. But are scientists paying due attention to AMV? In my experience, publications are frequently generated, submitted and reviewed by people who know terribly little about AMV. It’s a lot less surprising that so many published papers are not reproducible, when we realize that they were never shown to be reproducible when first submitted. Indeed, reviewers rarely require such demonstrations before accepting manuscripts for publication. And I believe that this may well be the crux of the problem.

This author has reviewed many analytically oriented manuscripts over several decades and has often read the reviews of his colleagues, working on the same manuscripts, for the same editors/journals. It is rare to see a request for any evidence of repeatability or reproducibility. Journals do not demand it in their guidance for authors, editors do not demand it in writing to authors or reviewers, and the reviewers do not demand it in their reviews. Is it really any wonder that there is – and perhaps always has been – irreproducibility in scientific publications?

Too few authors, editors, publishers and reviewers are sufficiently rigorous in their expectations of manuscripts – even in the very best of journals. Academics do not care to be demanding, and most reviewers do not practice AMV for their own research – and nor do they want to (so one can only assume that they do not believe it worthy of being pursued in science). Why then, would they demand it of the manuscripts they are reviewing?

The reproducibility crisis we now hear so much about could have been prevented by more responsible, conscientious and demanding reviewers and editors. However, they were not, are not and may never be sufficiently critical. And yet, irreproducibility will not end until we change how (and why!) we accept manuscripts.

I call on reviewers, editors and authors to agree that AMV must be a major part of all manuscripts received. I believe if we work together on setting the right standards, we could eliminate a great deal of science’s irreproducibility in a single stroke.