- Open Access
Better reporting for better research: a checklist for reproducibility
© Kenall et al. 2015
- Received: 30 June 2015
- Accepted: 30 June 2015
- Published: 23 July 2015
How easy is it to reproduce or replicate the findings of a published paper? In 2013 one researcher, Phil Bourne, asked just this. How easy would it be to reproduce the results of a computational biology paper?  The answer: 280 h. Such a number is surprising, given the theoretical reproducibility of computational research and given Bourne was attempting to reproduce work done in his own lab. Now at the National Institutes of Health (NIH) as Associate Director of Data Sciences, Bourne is concerned with the reproducibility of all NIH funded work, not just his own—and the problem is large. In addition to work in computational biology (which theoretically should be more easily reproducible than “wet lab” work), hallmark papers in cancer through to psychology have been flagged as largely unreproducible [2, 3]. Closer to home, GigaScience has carried out similar work to quantify reproducibility in their content. Despite being scrutinized and tested by seven referees, it still took about half a man-month worth of resources to reproduce the results reported in just one of the tables . “Reproducibility” is now increasingly on the radar of funders and is making its rounds in the wider media as well, with concerns of reproducibility making headlines at The Economist  and New York Times , amongst other outlets.
It is critical to note that irreproducible work doesn’t necessarily mean fraud occurred, nor even that the findings are incorrect; likewise, reproducible research can still be incorrect. While this key point is well-understood by most scientists, this is not always easy to explain to the general public. However, as most research is paid for through tax payers, public trust in research is essential. We—researchers, funders, and publishers—must do a better job at communicating this message to the public. We must better explain that science is an activity that continually builds on and verifies itself. But we also must develop policies that better support this process—policies, for example, that promote transparency and allow for improved verification of research.
Clearly important for clinical research, verification is equally important for preclinical research, something we all have an equal stake in. No one can innovate new drugs overnight, no matter how rich they are, no matter which doctor they see. Better, more robust preclinical research benefits us alla. Our ability to rely on published data for potential therapeutics is critical, and recently its reliability has been called into question .
One well-publicised example of this was brought to light in an oncology study of preclinical research findings in which researchers were able to confirm only 11% of the findings [8, 9]. Although the relevance of more robust research is clear in the area of oncology, it is also important for more exploratory research that might never make it to the preclinical setting. Funding and time are both increasingly limited, and the waste generated from follow-up work based on irreproducible research is high. A recent study by Freedman et al. estimated this at approximately $28 billion a year for preclinical research in the United States alone .
The NIH have recently taken bold steps to begin to tackle the need for better design, more appropriate analysis, and greater transparency in the conduct and reporting of research. In January 2014 the NIH announced they would fund more training for scientists in data management and restructure their grant review process to better value other research objects, such as data . But it is peer review and the editorial policies and practices of journals that have come under the greatest scrutiny, and in June 2014 a set of guidelines for reporting preclinical research were proposed by the NIH to meet the perceived need for more stringent standards . These guidelines ask journals to ensure, for example, that authors have included a minimum set of information on study design, that statistical checks have been carried out by reviewers, and that authors have been given enough information to enable animal strains, cell lines, reagents, and so on, to be uniquely identified reagents (for a full list of requirements, see the NIH Principles and Guidelines for Reporting Preclinical Research).
aFor further discussion of this around clinical trial transparency and reliability, see Ben Goldacre’s Bad Pharma.
bTo better support our authors in adhering to this checklist, we have also recently revised our section on data availability, detailing where authors can deposit their data and how to cite their data in their manuscript. We also have in-house staff available to work with authors to find a home for their data. http://www.biomedcentral.com/about/editorialpolicies#DataandMaterialRelease.
cThe Center for Open Science with stakeholders from research have recently devised an easy to use set of guidelines based on eight standards and three levels of adherence. With this checklist, all journals will adhere to level 2 requirements. At present, all BioMed Central journals adhere to level 1 requirements. http://www.sciencemag.org/content/348/6242/1422.figures-only.
AK wrote the first draft and SE, LG, LB, TS, and DS contributed additional edits to the text and comments. Core members of the Reproducibility Working Group responsible for the checklist and its implementation were LB, AK, TS, and DS. All group members contributed equally to the project. All authors read and approved the final manuscript.
We thankfully acknowledge the useful feedback on the checklist from Susanna Sansone at BioSharing (https://www.biosharing.org/), the entire BMC Biology and Genome Biology editorial teams, especially Penny Austin and Rafal Marszalek, and the Research Integrity team, especially Maria Kowalczuk and Elizabeth Moylan (http://www.biomedcentral.com/authors/biomededitors), at BioMed Central. This editorial was published jointly in BMC Neuroscience, Genome Biology, and GigaScience.
Compliance with ethical guidelines
Competing interests LB, LF, AK, TS, and DS are employees at BioMed Central. SE and LG are employees at the BGI.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Garijo D, Kinnings S, Xie L, Xie L, Zhang Y, Bourne PE, Gil Y (2013) Quantifying reproducibility in computational biology: the case of the tuberculosis drugome. PLoS One 8(11):e80278. doi:10.1371/journal.pone.0080278 PubMed CentralPubMedView ArticleGoogle Scholar
- Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2(8):e124. doi:10.1371/journal.pmed.0020124
- Baker M (2015) First results from psychology’s largest reproducibility test. Nature. http://www.nature.com/news/first-results-from-psychology-s-largest-reproducibility-test-1.17433
- González-Beltrán A, Li P, Zhao J, Avila-Garcia MS, Roos M, Thompson M et al (2015) From peer-reviewed to peer-reproduced in scholarly publishing: the complementary roles of data models and workflows in bioinformatics. PLoS One 10(7):e0127612. doi:10.1371/journal.pone.0127612 PubMedView ArticleGoogle Scholar
- “Trouble at the lab”. The Economist, Oct 19, 2013. http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble
- Johnson G (2014) New truths that only one can see. The New York Times, Jan 20, 2014. http://www.nytimes.com/2014/01/21/science/new-truths-that-only-one-can-see.html?_r=0
- Perrin S (2014) Preclinical research: make mouse studies work. Nature. http://www.nature.com/news/preclinical-research-make-mouse-studies-work-1.14913
- Ellis, Begley (2012) Drug development: raise standards for preclinical cancer research. Nature. http://www.nature.com/nature/journal/v483/n7391/full/483531a.html
- Kolata G (2011) How a new hope in cancer testing fell apart. The New York Times, July 7, 2011. http://www.nytimes.com/2011/07/08/health/research/08genes.html?_r=1
- Freedman LP, Cockburn IM, Simcoe TS (2015) The economics of reproducibility in preclinical research. PLoS Biol 13(6):e1002165. doi:10.1371/journal.pbio.1002165 PubMed CentralPubMedView ArticleGoogle Scholar
- Collins, Tabak (2014) Policy: NIH plans to enhance reproducibility. Nature. http://www.nature.com/news/policy-nih-plans-to-enhance-reproducibility-1.14586
- NIH principles and guidelines for reporting preclinical research. http://www.nih.gov/about/reporting-preclinical-research.htm
- Bustin S, Beaulieu J-F et al (2010) MIQE précis: practical implementation of minimum standard guidelines for fluorescence-based quantitative real-time PCR experiments. BMC Mol Biol 11:74. doi:10.1186/1471-2199-11-74 PubMed CentralPubMedView ArticleGoogle Scholar
- Hrynaszkiewicz I (2009) PRISMA statement published—and endorsed by Biomed Central’s Journals. http://blogs.biomedcentral.com/on-medicine/2009/07/27/prisma-statement-published-and-endorsed-by-biomed-centrals-journals/
- http://biomedcentral.com/bmcneurosci/authors/instructions/minimum_standards_reporting. The BioMed Central Checklist can also be found in our collection on BioSharing (https://www.biosharing.org/collection/BMC)
- Center for Open Science Transparency and openness promotion guidelines. https://osf.io/ud578/?_ga=1.173437419.933499240.1433864758
- (2014) Journals unite for reproducibility. Nature 515:7. doi:10.1038/515007a