Are your reagents for real?
While honest mistakes are inevitable in any human endeavor, the basic sciences, particularly biology, today find themselves drifting in a sea of irreproducible results.
A 2015 study by Leonard Freedman president of the Global Biological Standards Institute (GBSI) in PLOS Biology estimated that fully half of published biomedical research in the US, funded by $28 billion of taxpayer money, is irreproducible. Such work becomes the basis of grant renewals worth further billions of dollars, tenure awards at universities, drug approvals, and potentially even Nobel Prizes. Interestingly, top journals were not immune or even less likely than second-tier publications to suffer.
For example, a 2017 article in Nature Communications on reseach using estrogen receptor beta (ER-β), a 20-year-old biomarker for breast cancer, demonstrated that just one in 13 commercial antibodies used in that research—amazingly, the least-used product—actually targeted ER-β.
The factors most culpable, according to Freedman, were reagents and reference materials (36 percent), study design (28 percent), data analysis and reporting (25 percent), and laboratory protocols (11 percent).
That biological starting materials and reagents should be center stage in this controversy is not surprising. GBSI has for several years been calling for routine authentication of cell lines used in nearly every field of biomedicine.
As biological systems, cells suffer from genetic drift, contamination by other cells, mislabeling, poor handling, and a host of other issues that may result in vials or plates not containing what the producer says is inside or having lower potency than is advertised.
Large granting agencies and some journals have begun calling for (and in some cases requiring) cell line authentication for work they sponsor or publish, but researchers are essentially on an honor system, and, as recent reports suggest, honor is approximately as common in scientists as in the general population.
As products of living organisms, enzymes and antibodies, the most widely used biological reagents, carry the same authentication issues as do cells but with the added factor of the prep or manufacturing process.
Enzymes and antibodies are unique among biological molecules in their almost lifelike activity. Enzymes, of which millions exist in nature, convert one chemical into another with exquisite specificity; change one tiny physical characteristic on the substrate and the transformation does not proceed. Similarly antibodies, unique and complex molecules manufactured by animals’ immune systems, bind strongly and specifically to their targets.
Enzymes and antibodies are proteins isolated through complex manufacturing, and they must be handled with care from the raw-material stage through end use. Since so much of modern biology depends on these two reagent classes, their authentication is becoming the new frontier in scientific integrity.
The National Institutes of Health has published guidelines on the authentication of biological reagents proposed for use in NIH grant applications. Dr. Michael Lauer, deputy director for extramural research, notes that “research performed with unreliable or misidentified resources can negate years of hard work” and “it is imperative that researchers regularly authenticate key resources used in their research.” Lauer recognizes that vendor-sourced (as opposed to lab-generated) reagents may or may not have undergone fit-for-purpose testing, but regardless, in their grant applications, users “should also include a plan to independently verify the identity and activity of the product,” and if the reagent will undergo long-term storage, they should “consider the stability of the product” over time.
Value, not price
All reputable producers of biological reagents perform some type of quality control before their products ship. Abcam (Cambridge, UK), which provides a wide range of antibodies and reagents, employs extensive validation that includes DNA and peptide identity, activity, and the products’ fitness for purpose.
Abcam uses recombinant methods to generate many of its antibodies. Therefore, says Bruce Hamilton, PhD, head of new product development, “we ensure that we’re producing a reliable, defined sequence for the monoclonal antibody of interest.” Then, Abcam purifies its antibodies to high homogeneity using affinity media based on ligands extremely specific for this class of proteins.
The third validation level involves fit for purpose (specificity), which involves testing the product for performance against specific antigens as well as in actual assays. It is possible to perform binding studies between antibodies or enzymes and substrates to determine activity per mole, milligram of protein, or volume of reagent. This is the minimum one can expect from this type of reagent.
Abcam tests its antibody products in actual assays—for example, Western blots or immunohistochemical (IHC) assays. The company uses knockout cell lines, where the substrate gene has been silenced or deleted, to serve as a negative control for antibodies destined for cell-based assays. The elegance of this test is that every other variable—cells, culture conditions, readout—is controlled.
“Together these tests validate that the antibodies we say are in a product work in the way they’re intended to,” Hamilton says.
Quality of antibodies and other reagents is obviously critical for experimental reliability and reproducibility. Titer, or activity per volume or vial of product, is likewise an important consideration for labs interested in getting the most from these costly reagents.
“When we put product into a vial, we want to assure customers that they can anticipate how much they will need to conduct a certain number of assays,” Hamilton explains. “That number can be significantly different for a Western blot than for an IHC assay. Since we’re talking about biological molecules generated in living expression systems, antibodies may perform differently in different assays.”
Hamilton views the “problem” with biological reagents more in terms of reproducibility than authentication. The quality of biological reagents comes down to the products doing what they’re intended to do. “Antibodies and other biological reagents therefore need to be validated to the highest possible standards so customers have confidence in their work and know that the product will not vary from lot to lot,” explains Hamilton.
Minimal validation of biological reagents involves testing whether they perform as promised in specific assays. Mary Doers, senior director, manufacturing operations at Promega (Madison, WI), tells Lab Manager that her company performs analytical and functional tests “that have relevance to how our customers use our products. We often perform additional analyses to ensure batch-tobatch uniformity.”
While consumers take pleasure in price-shopping, that strategy works only when selecting among vendors that carry identical goods from the same manufacturer, like the latest cell phone or designer perfume. Biological reagents, even those with the same chemical name or product designation, are unique. They are extracted or isolated from living organisms whose cultivation and source may be different. Manufacturing differences also cannot be underestimated. As producers of biological drugs like to say, “The product is the process.” Products may differ noticeably, even when standard production methods are employed, between different groups at different locations, and seemingly trivial excursions from SOPs can lead to products of differing quality and performance.
That is why for biological reagents, value, not price, is the operative word.
“We believe that value-shopping is important,” Doers says, “because low price might not go hand in hand with high quality. Look for value; choose a reputable company from the start. Contact their technical services team, and get as much information as you can about the product.”
With so many reagents filling so many specific roles, biology is in need of a peermanaged system for evaluating reagents. Toward that end, GBSI has begun alpha testing a scorecard system to evaluate and rank research antibodies for specific applications based on their intrinsic on-target and off-target activity.
Freedman noted that earlier attempts to improve antibody performance resulted in welcome validation guidelines that too many researchers and antibody producers ignored.
The scorecard system will evolve through applicationspecific working groups representing stakeholders in preclinical life sciences, including academics, scientists at vendor companies, journal editors, and others.
The system will enable users to search an open database of scored antibodies to find the one most appropriate for their research. Users can contribute to the database by completing a scorecard for antibodies validated in their own lab.
After alpha testing the initial scorecards for Western blot, ELISA, and immunohistochemistry (the three most-used assays), participants will review the outcomes and determine what changes are needed. This step will be applied to scorecard development for the remaining applications. In 2018, GBSI plans to launch a website where users and producers can score antibodies and search for data on what GBSI hopes will be thousands of popular antibodies. GBSI will post scorecards for public comment and publish resulting guidelines and insights.
“It is time,” says Freedman, “to assess biological reagents based on their performance, not by name-brand marketing.”
As with cell line authentication, all stakeholders are responsible, to some degree, for biological reagents that don’t perform as advertised. Reliance on third-party sourcing of critical ingredients (or the product itself), shoddy manufacturing practices or quality control, and a lack of minimal fit-for-purpose testing are all factors, and to a degree the enablers are end users themselves.
Again, analogously to cell line authentication when it was a nascent issue, it’s an environment of hear no evil, see no evil. Freedman warns, “It’s a ‘buyer beware’ environment, but users also share some of the blame due to poor training on validating antibodies and their use for given applications.”