Review Explores Reproducibility and Replicability in Science

CAMBRIDGE, MA — December 16, 2020 — In 2019, the National Academies of Science, Engineering, and Medicine (NASEM) published a consensus report for the US Congress—Reproducibility and Replicability in Science—which addressed a major methodological crisis in the sciences: The fact that many experiments and results are difficult or impossible to reproduce. The conversation about this report and this vital topic continues in a special, 12-article feature in issue 2:4 of the Harvard Data Science Review (HDSR), published Dec. 16.

Growing awareness of the replication crisis has rocked the fields of medicine and psychology, in particular, where famous experiments and influential findings have been cast into doubt. But these issues affect researchers in a wide range of disciplines—from economics to particle physics to climate science—and addressing them requires an interdisciplinary approach.

"The overall aim of reproducibility and replicability is to ensure that our research findings are reliable," states HDSR editor-in-chief Xiao-li Meng in his editorial. "Reliability does not imply absolute truth—which is an epistemologically debatable notion to start with—but it does require that our findings are reasonably robust to the relevant data or methods we employ."

"Designing sound replication studies requires a host of data science skills, from statistical designs to causal inference to signal-noise separation, that are simultaneously tailored by and aimed at substantive understanding," Meng continues.

Guest edited by Victoria Stodden (University of Illinois, Urbana-Champaign), the special theme collection presents research and commentary from an interdisciplinary group of scholars and professionals. Articles include:

• Interview with Reproducibility and Replicability in Science Committee Chair Harvey V. Fineberg, president of the Gordon and Betty Moore Foundation and HDSR guest editor, Victoria Stodden, committee member by HDSR editor-in-chief Xiao-Li Meng
• "Self-Correction by Design" by Marcia McNutt, president of NAS
• "Leveraging the National Academies 'Reproducibility and Replication in Science' Report to Advance Reproducibility in Publishing" by Manish Parasha, assistant director for strategic computing at the White House Office of Science and Technology Policy, and director of the Office of Advanced Cyberinfrastructure at the National Science Foundation
• "Toward Reproducible and Extensible Research: from Values to Action" by Aleksandrina Goeva (Broad Institute), Sara Stoudt (Smith College), Ana Trisovic (Harvard University)
• "Reproducibility and Replicability in Economics" by Lars Vilhuber (Cornell University)
• "Reproducibility and Replicability in Science, A Metrology Perspective" by Anne L. Plant (National Institute of Standards and Technology) and Robert J. Hanisch (National Institute of Standards and Technology)
• "Perspectives on Data Reproducibility and Replicability in Paleoclimate and Climate Science" by Rosemary T. Bush (Northwestern University), Andrea Dutton (University of Wisconsin, Madison), Michael N. Evans (University of Maryland, College Park), Rich Loft (National Center for Atmospheric Research), and Gavin A. Schmidt (National Aeronautics and Space Administration)
• "Science Communication in the Context of Reproducibility and Replicability: How Non-Scientists Navigate Scientific Uncertainty" by Emily Howell (University of Wisconsin-Madison)
• "Learning Lessons on Reproducibility and Replicability in Large Scale Genome-Wide Association Studies" by Xihong Lin (Harvard University)
• "Selective Inference: The Silent Killer of Replicability" by Yoav Benjamini (Tel Aviv University)
• "Trust but Verify: How to Leverage Policies, Workflows, and Infrastructure to Ensure Computational Reproducibility in Publication" by Craig Willis (University of Illinois at Urbana-Champaign) and Victoria Stodden
• "Reproducibility and Replication of Experimental Particle Physics Results" by Thomas R. Junk (Fermi National Accelerator Laboratory) and Louis Lyons (University of Oxford, Emeritus)

The editors hope to take advantage of the collaborative features available on the open-source publishing platform, PubPub, where HDSR is hosted. Readers around the world can freely read, annotate, and comment on the essays—continuing this important conversation.

- This press release was supplied by The MIT Press. It has been edited for style