Recent years have seen animated debates about the transparency, reproducibility, and publication of research results. One of the many emerging initiatives trying to address bias in both research and publication is the option to get a research idea and protocol accepted by a journal before actually conducting the experiments, with the promise that the journal will later publish the results regardless of the outcome. The new publication format has yet to establish itself, but proponents argue that it could help young scientists in particular pursue good research ideas, develop solid methodological skills, and get papers on their CVs. So far, most of the journals offering the new publication format, known as a Registered Report, fall within specialist areas, such as psychology and neuroscience, but the U.K. Royal Society’s recent announcement that it has joined the initiative with one of its open-access journals will now make the option available to all major scientific fields.
What are Registered Reports?
The philosophy behind Registered Reports is that the new format will help improve research transparency and reproducibility by reducing publication bias—journals’ favoring of certain types of results—and, consequently, the questionable research practices that some scientists adopt to make their results more likely to be published. “Outcomes that are novel, or eye-catching are generally seen as more attractive and competitive than those that are null or ambiguous,” putting researchers under much career pressure to produce attractive results, says Chris Chambers, a cognitive neuroscientist at Cardiff University in the United Kingdom who became one of the founders of the Registered Reports concept a couple of years ago, in the Royal Society’s announcement. Such pressure, in turn, encourages researchers to engage in “a host of biased practices [including] reinventing history to ‘predict’ results that were, in fact, unexpected, or selectively reporting analyses that allow more publishable narratives,” continued Chambers, who is the editor in charge of handling Registered Reports at Royal Society Open Science, an online-only journal which covers all fields of science, engineering, and math. “Registered Reports not only reduce or eliminate bias, they also eliminate the need for biased practices in the first place,” Chambers says—although it remains to be demonstrated whether the new publishing model will actually succeed in achieving this aim and addressing all of scientists’ concerns.
Registered Reports are setting new standards in terms of statistical power and methodological rigor.
In a nutshell, here’s how the model works, based on a general description provided by the Center for Open Science’s (COS’s) Open Science Framework (OSF): Before researchers even begin the experiments, they submit a manuscript presenting a clear hypothesis that they plan to test and their proposed experimental methods and analyses. In a first peer-review phase, the journal evaluates the research question’s importance and the proposed experimental methods’ rigor and soundness. Upon acceptance, authors are invited to conduct the experiments, with the guarantee that their results will yield a publication regardless of whether they are positive or even statistically significant. A minority of the journals, including eLife, also publish the accepted protocols at this stage.
Once the experiments are complete and the researchers are ready to publish the results, the completed manuscript goes through a second-phase peer review that aims to check that two conditions have been fulfilled. First, the researchers must have stuck to the original research plan. Second, they must have conducted the experiments adequately as demonstrated by pre-agreed quality checks, such as positive controls and manipulation checks. At the time of publication of the full manuscript, the research data must also have been shared on a public archive such as OSF.
What could be in it for me?
At least in principle, “[i]n many situations [submitting a Registered Report] should only be a good thing for junior researchers because it alleviates the pressure to get ‘good results,’” writes Sam Schwarzkopf, a neuroscientist at University College London in the United Kingdom, in an email to Science Careers. Schwarzkopf initially raised questions about the concept of study pre-registration, but he now sees the new publication format as an initiative with much potential. In addition, the advance acceptance of manuscripts offered by Registered Reports “could push you ahead of the pack when it comes to applying for your next job because instead of having five papers listed optimistically in your CV as ‘in preparation,’ they would be ‘accepted in principle,’” Chambers writes in an email to Science Careers. Also, “Registered Reports are setting new standards in terms of statistical power and methodological rigor, which sends the signal that your work reaches the highest methodological standards, with potential benefits in terms of professional credibility and citations,” he adds.
Brian Nosek, COS executive director and a professor in the Department of Psychology at the University of Virginia in Charlottesville who studies the gap between research values and practices, also points out that using the Registered Report format can offer valuable feedback for junior researchers and help support their career development as they decide what research directions to pursue. “The biggest upside is that the researcher gets a lot of information about the publishability of a research idea before going through all the work of doing the study,” he writes in an email to Science Careers. If the study is given a green light, “Registered Reports offers a means of doing some work to prepare the idea, and then get assurance that following through will yield publishable work,” Nosek writes. If, on the other hand, the response is negative, that could also help young researchers by keeping them from investing much time and effort into pursuing dead ends, Nosek continues. “[G]etting rejected early is useful as a means of either improving the research designs, or recognizing that there are flaws in the designs that are difficult to overcome.”
The new publication format, which has primarily been designed to publish hypothesis-driven research, is however not suited to all forms of science. “Even if researchers have the option to declare how they changed plans or discovered new results after the registration, it still seems that this model imposes a rigid format to the scientific process that simply does not fit all situations and all fields equally,” writes Daniele Fanelli, a senior research scientist at the Meta-Research Innovation Center at Stanford, writes in an email to Science Careers. Fanelli, who studies bias and misconduct in science, welcomes the Registered Reports format as long as it does not become mandatory. Pre-registering studies may also be extremely cumbersome and not always necessary, especially for scientific work where a lot of data can easily be collected over a short period of time, Schwarzkopf adds.
It also has some potential downsides. “Because Registered Reports set high methodological standards, for example requiring high sample sizes and statistical power, researchers can find that their experiments take longer to conduct than they otherwise would under the status quo,” Chambers writes. This, in turn, can put them at a disadvantage, especially if they work in an endemically biased and underpowered field that allows researchers to produce low-quality papers quickly, he adds. On the plus side, however, while researchers who publish Registered Reports are likely to have fewer papers, these will be “larger and more credible papers.”
But perhaps the biggest drawback for young scientists right now is the uncertain career impact of participating in an experimental publishing model. According to Chambers, “[t]here are now over twenty journals offering the format, and we're seeing around one new journal taking them every month.” But, as he and more than 80 other scientists wrote in a 2013 editorial in The Guardian, for the initiative to really work, a much larger number of journals—and prominent journals in particular—need to join to make it a safe and desirable option for researchers working to establish their own careers. “To compete in the current academic system, young scientists need to be strategic in where they send their work, so as long as Registered Reports are offered only within specialist journals it places a ceiling on their desirability,” Chambers writes in his email. He adds that there are some signs of change in this direction, including Royal Society Open Science joining the initiative as the first generalist outlet.
Today, “[m]any career decisions are still made based on impact factors, etc. So there remains a pressure on junior scientists to submit their work to high impact journals rather than to one with the [Registered Reports] option,” Schwarzkopf agrees. He suggests that researchers make the decision about whether they want to use the Registered Reports option on a case-by-case basis, depending on the specifics of the project. For example, “[a]n experiment that requires flexible pre-testing but which can be conducted in [a] short time you should probably not submit” as a Registered Report, he says. Neither should junior scientists submit a sensational, ground-breaking idea, at least until Registered Reports become more commonplace, he recommends. “But if it's a solid idea, perhaps somewhat incremental, then [submitting as a Registered Report is] a good idea that I think will also benefit their career,” he says.
Because they guarantee publication and at the same time help novices improve their methods, Registered Reports may also be particularly well-suited to replication studies, Fanelli argues. And Chambers encourages young scientists to try the new publication model “for studies where you want to test a clear hypothesis and are concerned that a negative result could be hard to publish. Or for experiments in a field that is highly politicised [sic] or biased, so that finding the 'wrong result' under the status quo could delay publication of your paper for months or even years.”
Eventually, young scientists may find that it could pay off to take a risk. “In the close future I believe it will be important for hiring decisions to have a balanced CV that shows not only that your research has important impact but also that it is robust. So having at least a number [of Registered Reports] publications will actually benefit you,” Schwarzkopf writes. “Also, keep in mind that standards are changing,” Chambers argues. “Transparency and reproducibility are becoming more important career indicators, and they are only going to become more important in the years ahead. If you want to be leader in science one day then lead from the front, don't just follow the crowd.”