Scientists looked at all the published animal studies of sunitinib, a cancer drug successfully used to treat advanced kidney cancer, a rare type of stomach cancer and rare tumours of the neuroendocrine system.Photo courtesy of McGill UniversityBadly designed studies may lead to the efficacy of drugs being overestimated and money being wasted on trials that prove fruitless, according to new a study from McGill University in Canada.

The findings, to be published in the journal eLife, highlight the importance of ensuring that preclinical research can be reproduced by other scientists. Reproducibility helps confirm the validity of results before clinical trials in humans go ahead.

"Only a fraction of drugs that show promise in animals end up proving safe and effective in humans," says lead researcher Dr Jonathan Kimmelman, who directs the STREAM (Studies in Translation, Ethics and Medicine) research group.

"An important reason is because studies in animals are often not well designed, and because positive results have a higher chance of being published. They end up skewing what we think we know about the potential of a drug."

Blinding or randomization to avoid bias

The scientists looked at all the published animal studies of sunitinib, a cancer drug successfully used to treat advanced kidney cancer, a rare type of stomach cancer and rare tumours of the neuroendocrine system. They found evidence that studies that reported little or no anti-cancer effect were simply not published, leading anticancer effects of the drug to be overestimated by as much as 45%. The findings do not raise any concerns about the clinical use of sunitinib.

Few studies used practices like blinding or randomization, which help ensure personal expectations do not bias results. These practices, which are widely used elsewhere in medical research, prevent the experimenter from knowing which animals receive the drug and which receive the control. Often it was not even clear how many animals had been tested because the sample size was not reported.

Choosing the right animal model

The drug was tested against different types of cancer and all types tested showed statistically significant anti-cancer activity, a result which "strains credibility", according to Kimmelman. As well as over-estimating the effect of the drug on cancer-prone mice, the studies failed to observe the dose-dependent response to the drug that is known to occur in humans.

Finally, the studies failed to test the drug on a range of animal models, focusing instead on juvenile female mice with a compromised immune system. Malignancies tested in a wider range of animal types, such as mice that have spontaneously developed tumors, showed less extreme effect sizes.

What it means for patients

"Preclinical research is plagued by poor design and reporting practices, exposing patients to harmful and inactive agents, wasting time in the lab and driving up the price of drugs," says Kimmelman.

This lack of rigour may help explain why only 5 percent of agents that show anticancer activity in preclinical development are eventually licensed, while in cardiovascular disease, for example, the rate is 20 percent.

The authors of the cancer study have suggestions for addressing the problems they uncovered in animal research and for tackling problems highlighted by others - such as in vitro tests using drug levels that would be unachievable in humans due to toxicity.

"Our findings provide compelling reasons for developing and implementing guidelines for the design and reporting of preclinical studies in cancer, similar to those already in use for stroke, epilepsy and cardiology," says Kimmelman.