Sometimes greatness is hard to spot. Before going on to lead the Chicago Bulls to six NBA championships, Michael Jordan was famously cut from his high school basketball team. Scientists often face rejection of their own—in their case, the gatekeepers aren’t high school coaches, but journal editors and peers they select to review submitted papers. A study published today indicates that this system does a reasonable job of predicting the eventual interest in most papers, but it may shoot an air ball when it comes to identifying really game-changing research.
Studying peer review is difficult due to the confidential nature of the process, but sociologist Kyle Siler of the University of Toronto in Canada and colleagues were able to examine the peer-review history of 1008 articles that were submitted to three elite medical journals: Annals of Internal Medicine, The BMJ, and The Lancet. In total, just 62 of the manuscripts were accepted (6.2%), confirming just how difficult it is to be published in a top-tier journal. Editors “desk rejected” 722 of the manuscripts, meaning they never made it to the journal’s peer-review stage. However, 757 of the initially rejected articles eventually went on to be published elsewhere. This allowed Siler and his team to analyze if, like Jordan, the vetoed papers would go on to achieve greatness.
The researchers found that, by and large, the gatekeeping system was predictive of a paper’s eventual number of citations. Papers that were accepted outright by one of the three elite journals tended to garner more citations than papers that were rejected and then published elsewhere, they report online today in the Proceedings of the National Academy of Sciences. Additionally, papers that were desk rejected went on to receive fewer citations than papers that were approved by an editor, but then rejected during the subsequent peer-review process. “It’s a sign that these editors making snap decisions really quickly still have a nose for what quality is and isn’t,” Siler says.
There is a serious chink in the armor, though: All 14 of the most highly cited papers in the study were rejected by the three elite journals, and 12 of those were bounced before they could reach peer review. The finding suggests that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals, Siler says.
Part of the disparity may also arise from using citations as a measurement of quality. Although the most influential ideas often do serve as the basis for future research, highly technical topics with more niche appeal are less likely to be cited regardless of their scientific quality. Likewise, more accessible, “down-market” papers with practical applications may receive more citations, Siler says. “The analogy I use is, ‘Is Nickelback the best band in the world?’ No, not necessarily, but they’ve got that lowest common denominator, middle-of-the-road niche. The best research, in some cases, might be more esoteric.”
The initial rejection of all 14 of the most highly cited manuscripts indicates that the gatekeeping process is far from perfect. And Siler’s study is important, according to behavioral scientist Bill Starbuck of the University of Oregon in Eugene, because “anybody who comes out with some data which deals with this stuff, I think, is useful. I’d like to see more of these cages rattled. I’d like to see people rethink what they’re doing.”
One, perhaps radical, solution that seems to be gaining traction is to eliminate prepublication peer review altogether. With the rise of online journals, preprint servers, and academic social networks such as ResearchGate, page space is no longer at a premium, allowing more results to be published than ever before. “I think we’d be a lot better off if we just published everything. That’s probably the direction the world is actually going,” Starbuck says.
Low acceptance rates in journals may foster exclusivity and prestige, Siler says, but if his sample is representative of science as a whole, journals might also be missing important research that is esoteric or unconventional. In the case of the 14 overlooked items, “what the highly prestigious journals decide doesn’t dictate the ultimate value or recognition of the article,” Starbuck notes.