James Woodgett

Jim Woodgett of the Lunenfeld-Tanenbaum Research Institute at Toronto, Canada’s Mount Sinai Hospital has been leading an effort to change how Canada’s leading biomedical research funder does business.

Courtesy of Sally Szuster

Update: Canada's health funder agrees to meet with researchers outraged by peer-review changes

*Update: On Tuesday, officials at the Canadian Institute of Health Research (CIHR) agreed to a request from Jane Philpott, Canada’s minister of health, to meet with representatives of more than 1000 researchers who have demanded that the agency reverse changes to its grantsmaking process, including a decision to replace face-to-face peer review meetings with an online system.

CIHR President Alain Beaudet said the agency will heed Philpott’s 5 July request to “convene a working meeting in the very near future with key representatives of the research community, including those who have raised this issue publicly, to find common ground and move forward with solutions that address the issues raised with regard to the quality and integrity of CIHR's peer review system.” The meeting is expected to occur on 13 July.

In a 5 July statement, the besieged Beaudet acknowledged a need for the agency to endeavor to restore the community’s faith. “CIHR embarked on the reform of its open funding programs and peer review process to improve the sustainability of our health research system, the transparency and fairness of our granting processes, and the quality and impact of the research supported by federal investments,” he wrote. “However, the online system implemented to ensure an unbiased and tailored evaluation of each research proposal has raised serious concerns among applicants and reviewers alike. These concerns must be addressed since CIHR can only be successful if it has the support and confidence of the research community. This working meeting represents an important next step toward re-establishing that support and confidence.”

Beaudet also indicated CIHR is postponing the deadline on a forthcoming grant competition until after the working group meets, and Philpott said the results of the meeting will be expected to inform the findings of a recently launched Advisory Panel for the Review of Federal Support for Fundamental Science. That review, being undertaken by a nine-member panel headed by former University of Toronto President David Naylor, has been asked to produce recommendations by the end of the year on the architecture of basic science funding and support in Canada. The review encompasses the operations of CIHR and the nation’s two other granting councils, as well as programs like the Canada Foundation for Innovation, Canada Research Chairs, and Genome Canada.

“I have already ensured that the issues surrounding CIHR and its reforms will be considered by the panel, and I look forward to their advice,” wrote Philpott, who also noted that “I expect CIHR to ensure that the very best health research across all pillars [biomedical; clinical; health systems and services; and social cultural, environmental factors that affect the health of populations] is funded according to the highest international standards of research excellence.”

Here is ScienceInsider’s original 1 July story on the controversy:

Someday, historians might call it the Peer Uprising.

Nearly 1000 Canadian researchers are demanding that the government immediately reverse “radical” changes that the nation’s main biomedical research funder has made to its grantsmaking process, arguing that they are wreaking havoc on the science community. In particular, the researchers want the Canadian Institutes of Health Research (CIHR) to reinstate face-to-face meetings of peer-review panels, which the agency has ended in favor of an online system for evaluating grant proposals.

The new system “shows deep flaws and erroneous assumptions” and has led to shoddy review practices, the critics charge in a 27 June open letter to Jane Philpott, Canada’s minister of health. The letter expresses “extreme alarm” about the peer-review changes, as well as other CIHR reforms that the signers say have created “unpredictability, causing widespread lack of confidence in the review process and the agency itself. There is little hope that the best ideas and projects will be funded. We posit this represents a fundamental failure of CIHR’s primary mandate.”

Controversial reforms

The letter—so far signed by more than 900 researchers, including many prominent scientists—represents the latest salvo against a controversial reform effort launched roughly 4 years ago by CIHR President Alain Beaudet. Responding to recommendations made in 2011 by an international review panel led by Elias Zerhouni, the former head of the U.S. National Institutes of Health, the agency launched a three-pronged reform effort that revamped its funding streams, the way researchers submitted proposals, and the way proposals are reviewed.

Under the reforms, CIHR essentially obliterated its array of existing grant programs, replacing them with a regime that essentially awards two types of grants: “Foundation scheme” grants that provided $38,800 to $1.16 million annually for up to 7 years for work on more open-ended research (with duration and size pegged to how “established” a researcher was); and “Project scheme” grants of $38,800 to $581,000 annually for up to 5 years for more focused research projects. The idea, council officials said, was to find better ways of adequately supporting truly elite researchers, and reducing the pressure to constantly chase new grants.

The agency also remade its grantsmaking process, saying it wanted to improve “quality, fairness and transparency.” Under the old system, applications were distributed among 53 discipline-based expert panels for peer review, each of which met in person annually in Ottawa to rank the applications. Under the new system, applications are tossed into a central pool and each is assigned to four reviewers, who are supposed to participate in a virtual, “asynchronous” electronic discussion through an internet chat room mediated by 140 scientists known as “virtual chairs.” Proponents argued the changes would help to eliminate reviewer fatigue and promote approvals of multidisciplinary applications.

Competition complaints

Critics, however, were skeptical. And they say their fears were confirmed after the council held its first competition for Foundation grants in 2015. Many blue-chip researchers received “tiny” awards, whereas the success rates and award levels for both female and younger researchers were significantly lower than for male, more elderly counterparts, says geneticist Janet Rossant of The Hospital for Sick Children and the University of Toronto in Canada, and a former member of CIHR’s governing council. “The criteria, I think, became very much skewed not toward outstanding science, but outstanding leadership,” which put many investigators at a disadvantage, says Rossant, a winner in the competition.

CIHR said it would mitigate the bias against younger researchers by allocating them 15% of available Foundation grants, yet they received just 4% of the money awarded in the competition, adds Michael Hendricks, co-founder of the Association of Canadian Early Career Health Researchers and assistant professor of biology at McGill University in Montreal, Canada. That shortfall was compounded by CIHR’s redistribution of its overall pool of monies, (following the restructuring of its programs), which Hendricks says “took almost half of the open, operating grants funds off the table for new investigators. … We’re looking at probably over a 30% loss, in dollars, in funding support.”

As a result of such concerns, earlier this year the U15 group of research-intensive Canadian universities asked CIHR for a “temporary moratorium on further changes to research funding programs” and the convening of a national summit to discuss the changes. But Beaudet, who was appointed to a second 5-year term as CIHR president by former Conservative Prime Minister Stephen Harper in 2013, rebuffed the request.

Peer-review backlash

Worries about the new virtual peer-review system crystallized earlier this year, after CIHR conducted its first Project grant competition. Tight funding had caused CIHR to cancel two previous competitions, so “the majority of this country’s health researchers applied to the latest competition,” notes the 27 June letter, which was drafted by Jim Woodgett, director of research at the Lunenfeld-Tanenbaum Research Institute at Toronto’s Mount Sinai Hospital. “Despite warnings, CIHR scrambled to find sufficient reviewers, at least some of whom were inexperienced or unqualified.”

The shortage of reviewers prompted a “certain level of desperation” at CIHR, Woodgett says, forcing the agency to rely on a matching algorithm that assigned applications to reviewers based on keywords. That resulted in reviewers being asked to assess applications in disciplines that they knew nothing about, for example, “a health systems researcher given a cell biology grant,” and biomedical proposals assigned to clinicians or even hospital librarians. Virtual chairs reported that some peers got applications “with just 2 or 3 days’ notice.” Some reviewers didn’t bother to provide written reviews or participate in the online discussions, and applicants were given no chance to submit revisions in response to feedback.

The new system may be more cost-efficient, but “the fundamental issue here is really the abandonment of scientists evaluating other scientists in front of other scientists,” Woodgett argues. “There was essentially a degradation in the overall quality of the review process.”

A call for a summit

To fix the problem, CIHR needs to beat a hasty retreat from its reforms, say the signers of the 27 June letter and a related 29 June letter from a large group of CIHR University Delegates, who serve as intermediaries between Canadian universities and the agency. The critics want Philpott to immediately meet with a representative group of scientists to iron out concerns, and for CIHR to revert to traditional peer review until the new methodology has been assessed by international experts. The government should also convene a national summit to develop a new “health and life sciences action plan,” argues Bill Tholl, president and CEO of HealthCareCAN in Ottawa, the national federation of hospital and academic health care organizations.

Beaudet declined ScienceInsider’s request for comment. But he has posited the criticism his agency is getting is linked to funding issues, and will dissipate if the new Liberal Prime Minister Justin Trudeau’s government boosts CIHR’s budget. “The impact of this underfunding has been erroneously interpreted by some as resulting from the reforms, which in turn has increased the angst amongst stakeholders,”Beaudet said in his 16 February response to the U15.

That explanation, though, didn’t prevent Canada’s uprising. And it’s arguable whether additional funding will dissipate concerns about the perceived assault on the time-honored tradition of face-to-face peer review.

*Update, 6 July, 12:00 p.m.: This story has been updated with the most recent developments.