Read our COVID-19 research and news.

The National Institutes of Health is excluding more reviewers who have violated its policies on research integrity.

National Institutes of Health

NIH’s process for removing reviewers remains a mystery, watchdog finds

The National Institutes of Health (NIH) enlists some 27,000 scientists every year to help it judge more than 80,000 requests for funding. Although they volunteer their time, the reviewers are expected to uphold NIH standards on research integrity. If they don’t, NIH can remove them from the reviewer pool.

A new report by the Office of Inspector General (OIG) of NIH’s parent body, the Department of Health and Human Services, documents how NIH has begun to do exactly that. Some 77 reviewers have been put on a “Do Not Use” list as of November 2019 because they were found to have breached confidentiality, the report notes. An additional 47 were flagged after NIH found “potentially substantiated” evidence that they had failed to tell NIH about their affiliations with foreign institutions. And in December 2019 NIH Director Francis Collins said the agency had removed 55 scientists from the reviewer pool who were facing allegations of sexual harassment.

The OIG report praises NIH for taking such steps. But it thinks the agency should be doing even more to protect its grantsmaking process, including being more transparent on what criteria it is applying to exclude reviewers.

What NIH looks for

The 3 April OIG report focuses on policies affecting the current pool of reviewers, including the training they receive. It complements a report last fall on how NIH should vet potential reviewers to ensure the highest quality pool. Congress gave the watchdog office $5 million last year to do a deep dive into NIH’s grantsmaking machinery. Other studies in the works are examining how grantee institutions track foreign sources of funding and how NIH evaluates grant proposals.

NIH officials told OIG that they removed those scientists out of concern that their participation could taint the integrity of the grantsmaking process. (In late 2017, for instance, the agency announced it was rereviewing 60 applications after discovering rule violations on one review panel.) Undisclosed foreign influences, for example, “could lead peer reviewers to breach confidentiality.” And a man accused of harassing women might go out of his way to favor proposals by women to counter any perceived bias, according to a blog post last year by Noni Byrnes, head of NIH’s Center for Scientific Review (CSR).

However, not all scientists who break the rules are stricken from the reviewer pool. And OIG investigators didn’t get an answer from NIH when they tried to find out how agency officials make those distinctions.

A case in point involves NIH’s ongoing investigation into scientists who have foreign affiliations they might not have disclosed to the agency. NIH has identified 209 such scientists as of November 2019, according to the report, of whom 129 serve as reviewers. But only 47 have been banned from serving on review panels. “When OIG asked NIH about the remaining 82 peer reviewers—i.e. those who had potentially substantiated allegations but who had not been designated as Do Not Use—NIH did not respond.”

NIH officials declined to answer a list of questions from ScienceInsider on how scientists wind up on the Do Not Use list. In a written statement, however, they said that “a decision to flag an investigator as Do Not Use is made on a case-by-case basis and … requires a unanimous decision on the part of high-level officials at NIH, OER [the Office of Extramural Research], and CSR.”

Scientists are excluded for reasons unrelated to research integrity, they note. In her blog, Byrnes described how researchers have been barred for being chronically late in submitting their comments, having extensive conflicts of interest, or exerting an “undue influence” on NIH’s portfolio because of lengthy service on a review panel. Last week she declined to say more, including how many scientists are now on the excluded list, whether the number has grown significantly in recent years, and how and whether someone on the list can be reinstated.

Risk-based oversight

The new report repeats an earlier OIG recommendation that NIH keep a closer eye on those reviewers it believes are most likely to break the rules. Such “targeted, risk-based oversight” could be applied to those who review proposals containing “particularly sensitive information or having highly lucrative commercial purposes,” the report suggests. Taking a targeted approach would shrink the pool of scientists that need to be monitored, the report notes, saving NIH time and money.

But what reviewer or proposal characteristics might pose the greatest threat to the system? Nobody knows. For example, NIH declined to say whether scientists found to have breached confidentiality or failed to disclose foreign ties are any more likely to be reviewing proposals in the two “sensitive” areas that OIG has flagged, for example. Nor did officials say whether they even consider those categories to be risk factors.

One way that NIH is trying to uphold research integrity is by tracking the activities of reviewers, such as when they download a proposal. But the OIG report notes that the agency is not yet able to employ a digital tool, which NIH calls a Forensic Dashboard, “to identify evidence of potential breaches, in real time.”

The OIG report praises NIH for working with other federal agencies, including those responsible for law enforcement, national security, and intelligence. But it says NIH “can do more” to guard against “undue foreign influences” on its system of reviewing grant proposals. Collins promised to tell OIG this fall how NIH plans to implement the report’s recommendations.