Sharing interesting observations and measurements about the world around us is at the heart of the scientific process, so that others can learn and repeat the experiments. New scientific learning and theories are recognized when a consensus of scientists practicing in a specific area is reached about the experimental outcomes. This consensus can only be reached if the results are reproducible. In addition, these new learnings are applied to the resources and products that we use in our daily lives. Tests for quality and safety are developed to ensure appropriate and safe use. The data from the safety and quality tests need to be accurate and accepted. To be acceptable, the experiments producing the data must be validated and reproducible.
In our modern world with fast, easy, and sometimes overwhelming amounts of communication, there is more visibility than ever before about the need for reproducible data.
What is Reproducibility?
According to the National Academy of Sciences, reproducibility is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis.1 To be reproducible, results need to follow the same procedures and control what can be reasonably controlled. Reproducibility does not mean exactly identical results. Random elements will still impact the results, so an understanding of statistics and measurement uncertainty2 is still required. Reproducibility is also different from replication. Replication is an attempt to identically repeat an experiment. Figure 1 shows Zach Scott’s representation of replication and reproducibility.3
Of course, lack of reproducibility in experiments is nonsense. “Non-reproducible single occurrences are of no significance to science.”4 To be credible in the scientific community and to deliver technical value to customers, laboratory managers must deliver reproducible results. In the end, the quality of our data is the most important part of our roles as laboratory managers.
- Laboratory managers have several ways to ensure their labs are producing high quality, reproducible technical results:
- High expectations
- Method validation
- Understanding measurement uncertainty
- Effective training
- Effective documentation
- Good lab practices
- Shared results
It is important for all laboratory managers to clearly and specifically establish the expectation that the scientific outcomes delivered by the lab are of high quality, are reproducible, and meet all the expectations of good science. Being clear about why this is important and never accepting shoddy work is a part of laboratory leadership. Laboratory managers can encounter problems when they assume their scientists understand this, or when they are too busy to be aware of the details of work in the lab.
Method validation is a key step in driving reproducible outcomes. Method validation takes advantage of all the experience, talent, and knowledge of the scientists developing new methods and breaking new scientific ground to teach others how to follow the path. Typical validation examines these different areas of the method:5
- Accuracy—closeness of the result to the accepted value
- Precision—degree of scatter from a series of measurements
- Repeatability—same conditions within a short time
- Intermediate Precision—reproducibility, different conditions over a longer time
- Specificity—ability to assess the analyte in different matrices
- Detection limit—lowest amount of the analyte that can be observed
- Quantitation limit—lowest amount of the analyte that can be measured
- Linearity—results are proportional to amounts
- Range—the interval over which the method has linearity
Method validation takes into account both replication and reproducibility. Using precision measurements, the ability of the method to produce valid outcomes is determined for a series of experiments conducted by the same person, on the same instrument(s), under the same conditions, in a short period of time. The same measurements are done for experiments conducted by different people, on different instruments, under similar conditions over longer periods of time. If the experiments produce valid results under these conditions, then we have significant confidence that others using the method will generate reproducible outcomes.
Measurement uncertainty provides a clear understanding of the repeatability of results. The accuracy, precision, and standard deviation of results provides a clear indication of the quality of a technical outcome. All scientific labs need to understand if differences observed in experiments are significant. Does a set of results demonstrate that the outcomes are all the same, with some spread of uncertainty, or do they represent a real, although small, difference? Labs must be able to discern the difference and adequately communicate that difference to their clients or customers. Lab managers can ensure an improved understanding of measurement uncertainty by ensuring staff receive adequate statistical training, and documenting the key measurement uncertainty terms, formulae, and calculations in an internal standard operating procedure (SOP).
Training is an important function in all labs. Despite the high quality of college education, most new employees require significant training to contribute to the lab’s work. In addition, learning organizations are constantly sharing and training across staff to improve flexibility, innovation, and agility for the lab. Training is recognized by most accreditation agencies as a vital part of demonstrating competency in lab work. A review of the lab training records is an important piece of most lab audits. Laboratory managers can ensure effective training by focusing on these aspects:
- Choose an experienced teacher for the training
- Provide training to the teachers to improve their teaching skills
- Ensure the training includes actual, hands on, practical activities. Reading alone is not training
- Utilize both shadowing and on-the-job (OTJ) training
- Shadowing allows the student to observe as the teacher executes
- OJT training allows the teacher to observe as the student executes
- Effectively document important SOPs and train to them
Document, document, document. According to one of our A2LA auditors, if it wasn’t documented, it didn’t happen. Documentation is a vital step in the process. Not just to demonstrate to outsiders that effective training has occurred, but also to lay out the steps of the process, method, or experiment that needs to be completed. Most labs use SOPs to document how to properly complete the important things the lab does. These SOPs become the official how-to documents that all staff are expected to understand and follow.
SOPs may be very detailed to document standard methods and known procedures that must be frequently reproduced by different people. Other SOPs may be less detailed to describe approaches to problem-solving, or to provide a general outline of an approach to more complex processes, like research and development projects.
One piece of the lab documentation that is often neglected is addressing the ‘why’ questions. We are very comfortable documenting the ‘whats’ and ‘hows’ of lab work. This is the heart of a typical lab SOP. Unfortunately, we don’t very often also document the ‘whys.’ By capturing the ‘whys,’ we help document the assumptions, and we have a better chance of capturing some of the tacit information that will allow future staff to understand and teach the details of the work.
To ensure reproducible lab outcomes, a wide variety of good laboratory practices are needed. General practices such as lab cleanliness, calibrated equipment, controlled environment, and documentation of observations and results are good practices universal to nearly all kinds of labs. Specific practices correspond to the details of the science practiced. Mike Michaud is the environmental lab program manager for the City of Abilene, Texas. He suggests good practices of spike/recovery, triplicate experiments, and ensuring all results fall within the range of standard deviation defined for the standards. Training and expecting scientists to follow good lab practices will help drive greater reproducibility in results.
Sharing results of experiments with other practitioners is another way to ensure reproducible results. Round-robin testing is often used to compare the results on identical samples using the same method across different labs. Round-robin tests are a direct way to test reproducibility. The experiments are being completed by different people using different instruments in different labs having different environmental conditions. Round-robin tests provide data about the reproducibility of a method and provide lab managers with feedback about their lab’s ability to reproduce results.
Generating reproducible results is the key to good science. These reproducible results verify new discoveries, demonstrate competence in measurements, and enable clear communication of technical outcomes to other scientists. Lab managers have several ways to ensure and improve reproducibility in their labs, including: having high expectations about delivering excellent science, appropriately validating methods, using proper measurement uncertainty, providing effective training, ensuring effective documentation, expecting the implementation of good lab practices, and broadly sharing technical results.
- “Reproducibility and Repeatability in Science.” National Academies of Sciences, Engineering, and Medicine (2019), p46, THE NATIONAL ACADEMIES PRESS, Washington, DC
- NIST. “Measurement Uncertainty.” https://www.nist.gov/itl/sed/topic-areas/measurement-uncertainty
- Zach Scott. “Data Science’s Reproducibility Crisis.” https://towardsdatascience.com/data-sciences-reproducibility-crisis-b87792d88513
- Karl Popper. “The Logic of Scientific Discovery.” Routledge Classics, London, 1992
- European Medicines Agency, ICH Topic Q 2 (R1) Validation of Analytical Procedures, https://www.ema.europa.eu/en/documents/scientific-guideline/ich-q-2-r1-validation-analytical-procedures-text-methodology-step-5_en.pdf (1995)