It seems open revolt has its rewards.
Canada’s scientific community carved a major notch in its belt Wednesday by successfully forcing Canadian Institutes of Health Research (CIHR) President Alain Beaudet into a partial retreat from his controversial introduction of online peer review.
At an all-day meeting in Ottawa with roughly 50 practicing scientists, convened at the behest of federal Health Minister Jane Philpott to quell an uprising over CIHR grantsmaking reforms, Beaudet agreed to the introduction of a “hybrid” peer-review system. It will use online review to cull roughly 60% of grants from consideration, with the surviving 40% of proposals subject to face-to-face peer review to determine actual competition winners.
A contrite Beaudet emerged from the unprecedented gathering, conducted under the watchful eye of two of Philpott’s political staffers and the deputy minister of health, saying he’s hopeful the retreat will restore community faith in the CIHR.
But participants say the jury is still out. “The devil is going to be in the details,” said Jim Woodgett, director of research at the Lunenfeld-Tanenbaum Research Institute at Toronto, Canada’s Mount Sinai Hospital, who raised the alarm about Beaudet’s reforms by crafting a 27 June open letter to Philpott that attracted more than 1300 signatures. The new online peer-review system “was a failed experiment” that shook community faith in fairness of grant selection processes, he told ScienceInsider. “Once you’ve lost confidence in a system, you’ve got to put a lot of effort into it to restore it. … There certainly is a lot of disappointment with the leadership in CIHR in persisting with what was clearly a model which wasn’t working very well. They had lots of signs and lots of warnings about that.”
“People feel much comfortable if they’re not getting their grants [if] human beings have seen them and discussed them,” says Mona Nemur, the University of Ottawa’s vice president for research. “I think that we have a compromise. I think that the most important thing is that people feel that you have to put the person, human, back in the peer-review system. It’s not just a computer algorithm and so on. And I think this how the rest of the world functions. We’re embracing what everyone else is doing.”
Beaudet’s retreat should restore some community faith, said Bill Tholl, president and CEO of HealthCareCAN in Ottawa, the national federation of hospital and academic health care organizations. “I’d say we’re 50% there if this was a 100-yard dash.”
Still to be resolved are such issues as the size and membership of the expert panels that will actually meet in old-fashioned face-to-face peer review to hammer out the winners (at an estimated cost of $1.54 million annually), the extent to which teleconferencing can or will be used in synchronous discussion in early stage triage of grant applications, and the mechanisms to be used to match application to reviewers. (The new system used a computer algorithm developed by U.K. scientific publishing giant and analytics provider Elsevier.) A working group will be immediately convened to iron out those details within the next few weeks.
The question will be “how quickly this can be implemented,” said Kristin Baetz, president of the Canadian Society for Molecular Biosciences. The next CIHR competition for its two major grant programs—“Foundation scheme” and “Project scheme” grants—is slated for this fall. (Under Beaudet’s reforms, CIHR’s former array of grant programs was obliterated in favor of a regime that awarded Foundation scheme grants worth between $38,800 to $1.16 million annually for up to 7 years to established researchers to conduct open-ended research, or Project scheme grants worth between $38,800 to $581,000 annually for up to 5 years for more focused research projects.)
The restructuring of CIHR grants was not discussed at Wednesday’s gathering, Beaudet said.
Asked whether he’d given thought to resigning in the face of community outrage over his reforms, Beaudet essentially dodged the question, responding that “today is a day to rejoice. We’ve achieved something together.”
Beaudet added that the new hybrid system “goes a very big step forward in restoring the confidence of the community. ... There was a strong feeling (in the room) that the CIHR is responsive and that the community can really trust CIHR to do the best possible work in reviewing.”
The number of grants that will be reviewed face-to-face will be increased “by a factor of 10,” Beaudet said, adding that he believes the hybrid system will “improve the quality of peer review.”
“We’re looking at a variety, a scope, a breadth of topics in health research now that really needs that there’s no custom fitting, or custom reviewing. It means that we really need to ensure that every single application has the right expertise.”
As for the revolt, Beaudet noted “there are a lot of things that in theory can look absolutely superb and then in practice, it doesn’t quite turn out the way you wanted it.”
Beaudet also indicated he expects no outcry following this weekend’s announcement of the winners of its most recent grant competition, because 100 borderline applications were submitted for face-to-face peer review following the community uprising. “Every grant that we had reason to believe did not have optimal online review, or an optimal number of reviewers, was evaluated face-to-face.”