Douglas Fisher, an associate professor of computer science at Vanderbilt University, started his 3-year rotation as program director at the National Science Foundation (NSF) in 2007. That fall, he convened and chaired in-person panels to decide the fate of NSF grant proposals in information sciences. But when he started planning ahead for a spring meeting, he recalls by e-mail, “I could find NO room available to hold my panels on the dates I wanted or what I regarded as reasonable alternatives.” So instead, he ran the panel with a remote component, which allowed for phone and web-based videoconferencing. It worked well, and after that, he always gave panelists a choice: Those who wanted to attend in person could do so, and the others could phone in. After that, panelist acceptance rates shot up from 20% to 70%, and his panels included most of his top-choice candidates.
The limitation that everyone has to be at the same place at the same time to get something done no longer exists.
Allocating research and research-training funds is the prime responsibility of NSF, which funds research in several disciplines, and the National Institutes of Health (NIH), which funds research in the biomedical sciences. In the past, almost all the scientists who peer reviewed those funding applications flew in to the nation’s capital (or nearby), stayed for a couple of days, and pored over proposals behind closed doors. They accepted inconveniences—the wear and tear of travel and time away from family and regular work—in order to pay back into a system that they had benefited from.
That payback is getting cheaper. “The idea of flying people across the country to discuss proposals seems primitive and environmentally irresponsible,” says Fisher, who has championed remote panels ever since that first experience. Donald Hantula, a program director in NSF’s decision, risk and management science directorate, adds, “The limitation that everyone has to be at the same place at the same time to get something done no longer exists.” When Hantula is back at his home institution—Temple University in Philadelphia, Pennsylvania—he studies the effectiveness of virtual teams and work groups.
Both funding agencies now make liberal use of the greener alternative. Last year at NSF, there were 1874 panels with a remote component—1337 included virtual participants, and 28.6% were entirely virtual—easily surpassing NSF’s 5% target. At NIH, nearly 13% of the 54,000 proposals reviewed by study sections within the Center for Scientific Review (CSR) had a video or Internet-mediated component. (About 7000 proposals are reviewed within the various other institutes and centers.)The technology used in these remote reviews can range from old-fashioned conference calls on landlines through video-enabled conferencing systems, all the way to virtual online resorts in Second Life. NIH also employs Internet-assisted meetings: online chats that last a specific number of days, allowing reviewers to consult the literature before posting comments and responses.
Advantages of remote peer review
Why are the agencies embracing remote peer review? For one thing, remote meetings save money on airfares, hotels, and meals for panelists, and that money can be made available to fund research. Remote review saved NSF roughly $5 million per year on average over the past 3 years. Richard Nakamura, NIH’s CSR director, estimated last year’s savings at $3.9 million—but he insists that’s not the point. The point, rather, is to provide the best possible peer review. “Our primary goal is to bring together groups of reviewers who have the right expertise and the appropriate breadth and diversity so NIH can find and fund the most promising research. We use electronic review platforms when they allow us to engage reviewers that we might otherwise be able to recruit.” In the past, clinicians and entrepreneurs have been hard to recruit. For those populations, remote review is a big advantage, Nakamura says.
Parents of young children, too, often find travel difficult. Diane Barber, professor and chair of cell biology at the University of California, San Francisco, says that virtual panels would have been a nice option when her children were growing up. That ties in with the motivation of José L. Muñoz, NSF’s chief technology officer. The goal, he says, is to broaden participation without compromising the quality of the review process. Remote review, he says, works best for people who previously could not attend because of physical limitations, family responsibilities, or other constraints. The new technology also makes it easier to include international panelists, Muñoz says.
William Bainbridge, co-director of the human-centered computing cluster at NSF, has conducted more than 25 review panels in the online world Second Life. “I theorize, but cannot prove, that the use of such a virtual environment reduces some of the features of in-person social interaction that give some individuals more power than others based on size, emotional gestures, and facial expressions, and other aspects of communication style rather than intellectual merit,” he writes in an e-mail. “This may reduce pressures toward consensus, sustain a more democratic discussion, and encourage logic more than emotion in decision-making.”
Virtual panels have disadvantages. Maintaining confidentiality is a challenge for electronically facilitated discussions. Panelists using video must contend with difficulties we’ve all encountered: Poor lighting and bad camera angles are the bane of videoconferencing, Hantula says. “With video, microgestures and nonverbal cues should be available, but they are not. The camera could be pointing up a panelist’s nostril.” Despite these disadvantages, his research (which is not on NSF or NIH panels specifically) shows that in such panels, the work gets done. Panelists appear to work harder to communicate, but applications are read, and decisions are taken efficiently.
This is difficult to quantify, but participants say that technology-mediated panels make for a less engaged discussion. In hybrid panels, some virtual panelists are perceived as distracted or not “wholly present,” in the words of an NSF reviewer who prefers not to be identified. Remote panelists strain to be part of the conversation. NSF policy says that in-person panels are the best choice when there are more than 10 panelists, the proposals are complex, or the subject is multidisciplinary.
Remote review panels work fine when everyone agrees, but they are not as good as in-person panels when there is a major disagreement, says Suzanne Pfeffer, professor of biochemistry at Stanford University in California and a regular NIH panelist. “It is important that panel members build trust among themselves and hear each other with great care. In my experience, that dynamic is lost when reviews are carried out via online cameras,” she says. In-person participation, though, isn't always better: That essential dynamic can also be lost in face-to-face meetings, if the moderator doesn’t handle the discussion well and force the issue to a resolution, she adds.
Some virtual-panel critics lament the loss of social interaction that is an essential part of in-person panels. At the end of sessions, people disconnect and go back to their regular lives, instead of spending time with their peers on the panel. Young investigators, in particular, could miss networking opportunities and collaborations that can ensue from hallway conversations and shared dinners, those critics argue. Fisher, though, believes the purported social benefits of in-person panels are exaggerated.
A seasoned panelist for both NIH and NSF, who prefers not to be identified, concurs. Working in person with professional colleagues and socializing afterward are good ways to get to know them, she notes—but neither occasion presents huge networking opportunities. In her own career, she says, she has never started collaborations after a day spent evaluating grants. “I'm sure it happens, it just has never happened to me,” she says. So for her, virtual participation is worth the tradeoff. “I love being able to go to the gym in the a.m. and still be at my desk in time to call in to a review meeting. I also appreciate not traveling in general.”