The great reproducibility problem

Thorough, low cost, application-centric validation strategies need to be developed and validation needs to be performed by the vendor as well as the end user. If we can’t reproduce findings – then it isn’t science anymore. This is why the reproducibility problem in biomedical science is such a key issue – and we are all in this together says Dr Manoja Eswara

If you have ever been frustrated by not being able to replicate or reproduce experiments published by other labs, or for that matter, from your own – you are not alone. Like it or not, we are all part of the looming reproducibility crisis in biomedical research.

Although irreproducibility is a well known – but often ignored – issue, there is a recent outpouring of concern among the scientific community, owing to the magnitude of the problem and its economic impact^1,2. Findings suggest that approximately 50% of scientific literature is not reproducible, which translates to wasteful spending of nearly 28 billion dollars annually in the US alone^2,3. Apart from wasted time and money, invalidated research diminishes scientific credibility in the public eye. These alarming facts have provoked serious introspection and discussion by funding agencies, publishing groups, universities and industry partners to identify the core causative factors and come up with guidelines and best practices to avert the situation.

The causes

An extensive survey in the US of scientists across academia and industry by the Global Biological Standards Institute (GBSI) led to the identification of the following key reasons for the reproducibility crisis: - Variability in biological reagents (such as antibodies, cell lines) - Improper study design (lack of proper controls) - Biased data analysis and interpretation of results - Inadequate documentation and  reporting of protocols

There are other reasons, of course, pressure to publish, lack of funding to replicate experiments thoroughly, selective reporting, excluding negative results and insufficient peer review all contribute. The good news is scientific misconduct, which is an outright offense, is a minor contributor to this problem.

How to fix it?

Implementing and strictly following unifying guidelines and standards in biological research can thwart all of these root-causes for irreproducibility. But, global participation of the scientific community is essential to take on this herculean task. Towards this goal, the NIH, together with journal groups led by Nature and Science, have conducted workshops with scientists and journal editors and drafted consensus guidelines for research training, best practices, journal policies and reporting standards to enhance transparency and reproducibility^4,5,6. The NIH has highlighted four key areas in grant applications in order to promote rigour and reproducibility: - The scientific premise of the  proposed research - Rigorous experimental design for  unbiased results - Consideration of biological variables - Authentication of biological reagents

The curious case of antibodies

Biological reagents are the principal workhorses in biomedical research and are the top reason for irreproducibility. Take the example of antibodies; they are the most widely used protein-binding agents found in a variety of applications. There are a huge variety of antibodies available commercially, from different vendors, for the same protein. However, there is a huge vendor-vendor and batch-batch variability in binding specificity, cross-reactivity, binding strength, applicability in different techniques, tissues and model systems.

The product information from the vendor is often inadequate and searching through the literature does not always give the required reagent information because of insufficient documentation and reporting. This accounts for enormous false positives and false negatives, making antibodies the lead contributor to irreproducibility^7,8. There is, therefore, gaining momentum to fix the antibody crisis. Independent, non-profit groups such as the GBSI, SciCrunch and the European Antibody Network⁹ are stepping up to raise the awareness and promote the use of best practices and standards for antibody usage. And there are definitely a few key areas they are highlighting.

Thorough, low cost, application-centric validation strategies need to be developed and validation needs to be performed by the vendor as well as the end user

Validation measures: Owing to the broad range of applications, in which antibodies are used, there is a huge emphasis on validating antibodies for precise applications^9,10. Thorough, low cost, application-centric validation strategies need to be developed and validation needs to be performed by the vendor as well as the end user. The Antibody Validation Initiative by the GBSI is functioning towards this goal by establishing expert working groups to craft validation guidelines for specific applications.

Proper reporting: The best way to avoid ambiguity in antibody usage is to report full information such as the product name, supplier, catalogue number and methods when citing the antibodies in publications. A superior reference method that is catching up among scientists is the use of RRIDs (Research Resource Identifiers)¹¹, which are unique identifiers to cite biological reagents including antibodies, cell lines, software, and organisms.

RRIDs are established by the Resource Identification Initiative to promote the identification, discovery, and reuse of reagents and can be accessed and explored at the Resource Identification Portal of SciCrunch.

Centralised resources: Contrary to the lack of reporting in literature, there is too much antibody info vaguely dispersed on the web, making it extremely difficult to find customised information. Portals such as Antibodyregistry, Antibodypedia etc are trying to centralise antibody source and validation information from vendors and end users.

Recently, a platform known as BenchSci has emerged, which mines all the validation data available within the peer-reviewed literature. Using filters for specific experimental contexts, it can help scientists quickly and easily identify the most relevant published figures and associated antibodies within seconds, ensuring that scientists can make an evidence-based decision on antibody selection. Moreover, scientists can create projects to save specific figures, submit antibody reviews, and most importantly, share negative results produced by antibodies that did not work as intended. A similar comprehensive framework of standards and resources are being developed for other problematic biological reagents such as cell lines¹² as well. Setting rigorous research standards would take time and money, but once established, would have a great impact on the economics and credibility of research. The key to fixing the reproducibility problem lies in the hands of researchers; we should proactively seek and make the best use of these resources.

All for one

Advancements in scientific research are progressions built on previous discoveries; reproducibility not only authenticates candid findings but also self-corrects for flaws in experimental design, reporting and interpretation. The reproducibility problem will take the collective effort of the entire research community to solve. It is every scientist’s responsibility to consciously minimise variability, embrace best practices and standards and uphold transparency to promote reproducibility in research.

Author: Dr Manoja Eswara is a research scientist in cancer molecular biology and epigenetics at the Lunenfeld-Tanenbaum Research Institute, Toronto, Canada.

References: 1. Baker, M. 1,500 scientists lift the lid on reproducibility. Nat. News 533, 452 (2016). 2. Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The Economics of Reproducibility in Preclinical Research. PLOS Biol. 13, e1002165 (2015). 3. Freedman, L. P. & Gibson, M. C. The impact of preclinical irreproducibility on drug development. Clin. Pharmacol. Ther. 97, 16–18 (2015). 4. McNutt, M. Journals unite for reproducibility. Science 346, 679–679 (2014). 5. Journals unite for reproducibility. Nat. News 515, 7 (2014). 6. Collins, F. S. & Tabak, L. A. Policy: NIH plans to enhance reproducibility. Nat. News 505, 612 (2014). 7. Baker, M. Reproducibility crisis: Blame it on the antibodies. Nat. News 521, 274 (2015). 8. Bradbury, A. & Plückthun, A. Reproducibility: Standardize antibodies used in research. Nat. News 518, 27 (2015). 9. Roncador, G. et al. The European antibody network’s practical guide to finding and validating suitable antibodies for research. mAbs 8, 27–36 (2015). 10. Uhlen, M. et al. A proposal for validation of antibodies. Nat. Methods 13, 823–827 (2016). 11. Bandrowski, A. et al. The Resource Identification Initiative: A Cultural Shift in Publishing. Neuroinformatics 14, 169–182 (2016). 12. Freedman, L. P. et al. Reproducibility: changing the policies and culture of cell line authentication. Nat. Methods 12, 493–497 (2015).