Sebastian Doniach is a professor of physics and applied physics at Stanford University and has held the post of director and chair of the faculty at the Stanford Synchrotron Radiation Laboratory in Stanford, California. He recently wrote to Science summarizing his concerns about the proposed changes to NIH's review process (Science's Compass, 17 September, p. 1850). Doniach believes that NIH reviewers disregard physics-based research proposals despite their importance to biology.
Below is a copy of the letter he sent to the Panel on Scientific Boundaries for Review detailing his 10 main points of concern about their proposed changes:
Here are some comments on your report, a copy of which I downloaded from the CSR website.
I feel your statement: "In addition, certain segments of the research community, including clinical researchers, behavioral scientists, bioengineers, and developers of technology and instrumentation, believe that they are inadequately served by the existing system" should be EXPANDED to include "physicists".
I realize that your "Working Group On Review Of Bioengineering And Technology And Instrumentation Development Research", defined "bioengineering and technology" as encompassing areas such as biotechnology, functional genomics, informatics, chemistry and physics, nevertheless they did not discuss the problems experienced by physicists engaged in basic research on the frontier of physics and biology from the present system of study sections. It was perhaps unfortunate that the only physicist on this Working Group, Steve Koonin, has never published a paper in any field relating to biology. (He did recently author a report from JASON on the Genome project, but has not himself worked in bio-related problems).
Perhaps because of the composition of the working group, they tended to focus on issues on the interface between engineering and biology and not the physics-biology interface, the nature of which has been eloquently described by Harold Varmus. This lack of discussion of the latter interface makes itself felt also in the report of your committee.
Many of the physics-biology issues alluded to above are cultural in origin, however I find the section in your report on the relationship between sections and disciplines to be muddled and quite confusing from this perspective. It's a standard practice of study sections to say, in effect, this proposal is submitted by a physicist, hence is not worthy of NIH funding, regardless of the importance to biology of the proposed research. What they are doing is claiming "this is discipline-based research (i.e. physics ? since it is being proposed by a physicist) and therefore does not fit in to the mission of NIH". It 's my belief that the issue of "discipline-driven" versus "scientific problem-driven" is totally irrelevant to the assessment of whether proposed research will further the fundamental mission of the NIH. I realize the writer(s) of this section of your report had this in mind, but feel they have muddied the waters by the way they expressed this issue. A study section which wants to "protect its turf" will seize on the statement "fostering the health and propagation of disciplines is not the function of NIH, but rather of universities" as an excuse to shoot down basic research proposals by people who they perceive as being outside their own discipline.
Your report makes the statement that "Peer reviewers thus need not be scientific "competitors" of the applicant, or even be studying the same disease or organ system". I do not believe this statement is nearly strong enough - In areas where the number of researchers is quite limited, a direct competitor on a study section has in practice the power to keep a researcher out of NIH funding essentially indefinitely (so long as they are on the section) - See section 4 below. I feel your report should make a much stronger statement - that reviewers who are direct competitors should excuse themselves from participation in the review, and, if it should turn out that they have not done so, this in itself would be grounds for an appeal of the rating.
Your report nowhere goes into procedural issues of how ratings are generated by study sections. But this is in fact a significant issue in the mechanism by which study sections can protect their sub-field from "intruders", particularly when the subject matter is unfamiliar to many members of the study section. It's my understanding that in practice the ratings are proposed by the two principal readers of a given proposal and by the two subsidiary readers. Then although this rating can in principle be altered as a result of discussions by the study section, the level of modification which occurs in practice is quite minimal. The result, particularly when direct competitors are involved (see remarks above) is that one person can essentially block "outsiders" from her/his subfield without much moderation by other members of the section.
Because of the potential for bias against new and unfamiliar ideas induced by the above practice I believe the peer review system of the NIH as presently constituted, has inherent tendencies to protectionism. Therefore I would like to suggest that the present study section system be reorganized to depend much more on mail reviews from a broad spectrum of outside reviewers (as practiced by the NSF). This will help alleviate the problems which arise when a given study section does not have sufficiently objective expertise to adequately assess a given proposal, or where the "resident experts" tend to represent a built-in cartel.
As discussed in appendix II, the CSR is exploring flexible ways to overcome the present obstacles to reviewer recruitment. I feel that adding mail reviews to the present system could help alleviate some of these problems since it would spread the reviewing burden more widely.
I applaud your exhortations to study sections to broaden their criteria for evaluation of research styles. In particular your statement :" Peer reviewers should eschew the common current tendency to find fault or to identify minor errors. Instead, they should strive to assess the potential impact of the proposed research and to encourage good ideas and novel concepts, even if they appear to be risky" is very relevant to defects in the current study section culture. I'd like to suggest that the current practice of finding fault based on "inadequate reference to the literature" be strongly discouraged. As is well known in the community, reviewers use this as an excuse to fault proposals which do not adequately refer to the panel member's own work. In addition to being unachievable (since panel members are often replaced by substitutes whose identity is not known at the time of the submission date) it is a standard tool for eliminating proposals from scientists who are considered to be "outsiders" to a given study section's normal range of poposers.
I am concerned that the Panel's recommendations are really not being absorbed by the Study section SRA's at the present time. Your statement in section III, Cultural changes, that "Adopting them need not await the proposed reorganization" should be promulgated forthwith and that the recently instituted group of oversight "baby sitter" study section members should insist that the cultural changes you propose be applied right away.
I feel that you do not adequately cover protein-protein or protein substrate interactions in the proposed IRG list. These are at length/time scales intermediate between the CHEMICAL BIOLOGY AND BIOPHYSICS IRG and the MOLECULAR APPROACHES TO CELL FUNCTIONS AND INTERACTIONS IRG. Things like HIV protease inhibitors, for example, or Kinesin-tubulin interactions do not appear to be covered by any of the subheads listed under these two headings. Obviously they could be made to fit in with one or the other, but it would be better if this was made more explicit (in terms of when the time comes to pick panel members for these IRG's).
Concerns are being expressed by current SRA's that the proposed new organization of study sections will be "too vertical" and thus have insufficient expertise in some of the basic sciences, such as protein crystallography for example. In fact, as expressed above, it is this predominance of "expertise" which often leads to some of the worst features of protectionism in the present system. In my view, proper use of an outside mail review system coupled with a panel assessment of these reviews would be much more effective at arriving at "objective" assessments than is the present system. Therefore, the new set of integrated review groups and their dependent study sections will be much more effective if complemented by a mail review process.
Departments of Applied Physics and Physics
 Harold Varmus, "The Impact Of Physics On Biology And Medicine"
Plenary Talk At The Centennial Meeting Of The American Physical Society,
Atlanta, Ga., March 22, 1999