The biotech company Amgen Inc. and prominent biochemist Bruce Alberts have created a new online journal that aims to lift the curtain on often hidden results in biomedicine: failed efforts to confirm other groups’ published papers. Amgen is seeding the publication with reports on its own futile attempts to replicate three studies in diabetes and neurodegenerative disease and hopes other companies will follow suit.
The contradictory results—along with successful confirmations—will be published by F1000Research, an open-access, online-only publisher. Its new “Preclinical Reproducibility and Robustness channel,” launched today, will allow both companies and academic scientists to share their replications so that others will be less likely to waste time following up on flawed findings, says Sasha Kamb, senior vice president for research at Amgen in Thousand Oaks, California. The aim is “to help improve the self-correcting nature of science to benefit society as a whole, including those of us trying to create new medicines.”
Alberts, a former Science editor-in-chief and National Academy of Sciences president who is at the University of California, San Francisco, says the journal will be a place for data that other journals often aren’t interested in publishing because replication efforts lack novelty. “The whole idea is to lower the energy barrier for people doing this,” Alberts says.
A growing number of retractions—a few linked to fraud—have helped stir concerns about irreproducible results. Amgen itself raised the alarm 4 years ago when one of its scientists, who had recently left the firm, co-authored a commentary in Nature announcing that Amgen researchers could replicate just six of 53 landmark cancer biology studies. Bayer had published a similar analysis a few months earlier. The Amgen report drew criticism, however, because the company did not release any data or even reveal which studies it examined. (The former Amgen scientist, C. Glenn Begley, said one reason was confidentiality agreements with some of the original authors.)
The new channel, in contrast, will publish both methods and data from the replication attempts. “I do think it’s proper to show data and let people decide for themselves, and that’s the plan here,” says Kamb, who joined forces with Alberts after they attended a retreat on integrity in science early last year.
Putting the contradictory data out in the open will “force the original authors to clarify [the discrepancies],” Alberts says. “Maybe it was correct, but the original description of what they did was flawed.” Publishing the data will also give the replicators credit for the time they’ve spent, he and Kamb say.
Amgen chose the three papers initially highlighted on the site not because they were particularly noteworthy, but because the company’s scientists were ready to write them up, Kamb says. In one study, Amgen was not able to confirm a 2012 report in Science that a cancer drug, bexarotene, could clear β-amyloid plaques and reverse cognitive problems associated with Alzheimer’s disease in mice. Four other groups had already questioned the findings in technical comments in Science. But Kamb says Amgen’s contribution, which uses a different tool molecule to test bexarotene’s purported mechanism, adds “another very large brick.”
Amgen also reports that it failed to confirm a 2010 Nature study finding that blocking an enzyme called Usp14 helped cells degrade toxic proteins involved in Alzheimer’s and amyotrophic lateral sclerosis. The Amgen work “raises questions about the utility of Usp14 as a therapeutic target for neurodegenerative disease,” Kamb says. In their third paper, Amgen scientists report that a genetic flaw in a commercially available mouse model that lacks the gene for the protein Gpr21 led both an earlier Amgen team and an academic lab to erroneously report that Gpr21 influences body weight and glucose metabolism. (Amgen gave reporters summaries of the three studies on the condition that they not contact the original authors until after the new channel launched.)
Like other F1000Research “channels,” the new journal will first post the raw manuscripts, then invite peer reviews that will be added with the reviewers’ names. F1000Research charges author fees of $150—$1000 per paper depending on length. If the papers are accepted after peer review, they will be indexed in PubMed and “become part of the record,” Alberts says.
The project joins other efforts to promote reproducibility. The nonprofit Center for Open Science headed by psychologist Brian Nosek of the University of Virginia in Charlottesville is working with contract labs to replicate experiments from up to 50 high-impact papers in cancer biology. The project contacts the papers’ original authors for input on protocols and will also invite them to peer review the results before they are published in the open-access journal eLife. Such consultation has “social and cultural value,” Nosek says, but he doesn’t see it as a requirement and supports the new channel.
National Institutes of Health (NIH) in Bethesda, Maryland, Deputy Director Lawrence Tabak says NIH is also “supportive of the new channel” and is “encouraged by Amgen’s leadership role.” Microbiologist Arturo Casadevall of Johns Hopkins University in Baltimore, Maryland, is a fan as well, but he cautions that a failed confirmation doesn’t always mean original findings were incorrect. Ideally, the original authors will “try to understand the variables” that led to different conclusions, he says.