A troubling lack of replicability for cancer biology studies
After an ambitious project struggled to replicate high-profile studies, researchers are calling for a new focus on protocol and data sharing as essential steps for building confidence in the field
Why so few? The researchers reported extreme difficulty in even conducting many of the necessary experiments. None of them were described in enough detail to design a replication without clarification from the original authors, the report asserted, and 32% of those authors never responded to inquiries or declined to share reagents, code, or data. In some cases, extreme clarifications were needed to attempt the replications, and model systems often did not behave as they had been originally described. “Unnecessary friction in the research process may be slowing the advancement of knowledge, solutions, and treatments,” a write-up of the findings concluded.
For cancer research, the ability to move exciting findings from bench to bedside “requires us to understand what we’re moving along the way,” project leader Timothy Errington, PhD, director of research at the nonprofit Center for Open Science in Charlottesville, Virginia, tells CytoSource. If other researchers are not able to replicate the conditions under which the initial observations were made, he says, it is difficult to assess their credibility or whether the findings merit further investment.
Past efforts had pointed toward a replicability problem in the field of cancer biology. Dr Errington says that the finer level of detail from this study, including why some of the studies could not be replicated, has highlighted potential bottlenecks in the research process that could be targets for corrective action to improve the flow of knowledge.
As for the original experiments that the project could complete, Dr Errington says, the replication attempts resulted in significantly weaker evidence for the findings. Positive results, in fact, were, on average, half as likely to be replicable as null results, and the replication effect sizes for those positive results were 85% smaller than the original effect sizes (based on a relatively small sample size, however).2
“Surprising things should be less replicable than non-surprising things,” says Brian Nosek, PhD, executive director of the Center for Open Science, who notes that it was impossible to calculate a precise reproducibility rate because of the number of variables. Replication failures could be due to a false-positive result in the initial experiment, a false-negative result in the replication attempt, or differing conditions between the initial experiment and the replication attempt that created site-specific effects. Even so, Dr Nosek says that it was reasonable to suggest that the credibility of published findings in cancer biology may be less certain than was previously recognized.
Marcia McNutt, PhD, president of the US National Academy of Sciences in Washington, DC, notes that experiments using animal models can be especially tricky for independent groups to replicate because of the additional layer of variables in how the animals are housed, fed, and used in the research. Dr Errington adds that the replication project’s collaborators tried to recreate the same experimental conditions and steps as the original studies. “But naturally, it’s always different. If we can get similar results, despite that, that’s what we’re hunting for.” The ability of independent laboratories to replicate the results of cancer biology experiments despite all of the inherent variation “tells us we’re going in the right direction,” he says.
Conversely, if a successful replication attempt depends on highly technical knowledge or specialized equipment and techniques that are available only to select laboratories, Dr Errington says, that information is also valuable in assessing whether the knowledge can be transferred and have a broader clinical impact.
The self-corrective process of science depends on sharing data, which has long been identified as a major problem in a range of disciplines.3 Even so, Mike Lauer, MD, deputy director for extramural research at the National Institutes of Health, says that the cancer biology reproducibility project has yielded some “very striking findings.” He is troubled in particular by the revelation that one third of the scientists who had authored the original studies were not at all helpful to the researchers attempting to replicate their work. The results should not be taken to suggest that the science is untrustworthy, he says, but more work clearly is needed to encourage the open communication and data-sharing necessary to prevent an erosion of trust.
Creating a new culture of sharing
“What if we started having in our funding applications, ‘How many of your papers have open data? How many papers have open reagents? How many are pre-registered?’” –Timothy Errington, PhD
“For instance, what if we started having in our funding applications, ‘How many of your papers have open data? How many papers have open reagents? How many are pre-registered?’ Just asking that instantly shifts the mindset to, ‘Oh, wait a second—so that’s something you’re going to consider? Well, shoot; I’d better start doing this because I don’t want to leave that blank,’” he says.
Likewise, journals could begin asking authors for their data. At the very least, the questions could start a conversation about how to make open practices a more mainstream part of conducting research. “I think if you can get journals, institutions, and funders all asking for the same thing, then it becomes a reinforcing system that can start to shift the culture,” Dr Errington says.
Journals’ rigid formats sometimes work against reproducibility by encouraging authors to omit figures and details to fi t the space constraints. In one case, Dr Errington and his colleagues were able to replicate a study’s experiments only after the authors supplied an earlier draft of their article that included enough information to help to fill in the gaps.
Dr Errington says that more articles could include a brief description of methods along with a link to more extensive details to give other researchers a deeper understanding of the steps and nuances of the experiments. “That should be the bread and butter of what we’re doing as research, which is communicating that type of information for both the application of what we’re learning in the paper and for somebody else to pick it up and use it for some other purpose,” he says. “But it has to go past the paper.”
On the whole, for example, Dr Errington says that larger proteomic and genomic studies have improved their data sharing over the past few years. For a broader effort, he is excited about a new initiative called the Advanced Research Projects Agency for Health, housed within the US Department of Health and Human Services. The agency “will support transformative high-risk, high-reward research to drive biomedical and health breakthroughs” according to its website. Part of its mission, Dr Errington says, will be to push for more forward-looking approaches to how research is conducted and communicated, including an attempt to “bake” replication into the system from the start as a self-corrective measure that improves long-term efficiency.
US officials also might do well to follow the European Commission, says Dr Errington, which is “leaps and bounds” ahead in reconsidering the roles of publishers, preprints, research preregistration, and an infrastructure and reward system that more holistically supports information sharing and knowledge transfer from the start. “They’re thinking about more than just the data; they’re able to really look at the process,” he says.
Sharing data in an understandable and reusable way, Dr Errington notes, requires an investment: robust and well maintained repositories, for example, and funding incentives for researchers to expend the time and effort necessary to properly share their data. However, over time, he and other experts have concluded that the benefits to biomedical research could be invaluable.
