May estimates (1) return on investment for a single year (1991) based on the yearly average number of citations over the period 1981-94 (4). However, expenditure in 1991 will have little effect on citations before about 1997 because there is commonly a 4-year lag before papers emerge from the funded research and at least a further 2-year period before the citation peak is reached. The preferred analysis would be to compare expenditure figures for each year with citations achieved (say, 4 to 6 years later) and to track this over time. Because May takes average citation data as the numerator and a single funding year as the denominator, his analysis also becomes sensitive to the funding year selected. There is no "correct" funding year that one can use, but we would argue that an earlier year than 1991 would be better (5). We have therefore looked at government expenditures in the G7 countries for a range of earlier years (Table 1) (6) and compared these with May's citation data to calculate Apparent Cost Effectiveness (citations per £million).

Table 1. Government funding of R&D for civil objectives in real terms (adjusted to 1991 prices in £billion) and, in italics, as a percentage of gross expenditure on R&D (GERD), 1986-91, for the G7 countries (4). Country R&D funding per year 1986 1987 1988 1989 1990 1991 U.S.A. 12.44 13.20 13.41 14.52 15.27 16.93 13.4 14.0 14.0 14.9 15.4 16.5 Germany 6.44 6.45 6.40 6.62 6.66 7.95 35.9 33.7 32.6 32.2 32.5 35.2 Japan 5.40 5.55 5.65 5.86 5.95 6.32 18.0 17.4 16.4 15.6 14.8 14.8 France 5.33 5.16 5.14 5.34 5.36 5.81 42.0 39.0 37.7 36.7 35.3 36.5 Italy 3.51 3.86 4.09 3.85 4.09 4.31 57.8 59.0 59.5 53.3 53.5 52.5 U.K. 3.21 3.16 3.09 3.00 2.95 2.79 26.4 25.6 24.7 23.2 22.8 22.5 Canada 1.77 1.63 1.65 1.73 1.75 1.90 40.2 36.7 37.3 37.8 36.7 37.9

Our sensitivity analysis (Fig. 1) shows that the United Kingdom moved into first position ahead of the United States only if expenditure in 1991 is used as the denominator. This is because, in contrast to the situation in other G7 countries, U.K. government civil expenditure on R&D has been steadily falling in real terms (7). This might have been expected to have led to a decline in U.K. scientific output. In practice, the reduction in government expenditure has largely been made up by an increase in funding from other sources, including private nonprofit funders such as charities, and the part of spending by industry and foreign sources that is for research in the public domain (that is, which leads to research publications in the open literature).

The importance of nongovernment funders varies with the field of science. In particle physics, government support dominates, while in biomedical science, private funders play a bigger role. Biomedical science represents one of the United Kingdom's great strengths, with outputs increasing from 23,354 papers in 1988 to 29,391 papers in 1994 (8). Research charities have played a key role in the growth in numbers of research papers in this field. Over the period 1988-94, there was a 60% increase in the number of funding acknowledgements to U.K.-based charities in that country's biomedical literature. By 1994 charities were the acknowledged funding source on about 25% of national U.K. biomedical science publications. In comparison, the growth in the number of papers acknowledging support from the main government funding agency, the Medical Research Council, was only 5% over the same period. Despite this growth in charity output, the United Kingdom's world share of publications in biomedicine has declined marginally (9), mainly because of more rapid growth in other countries. Without the contributions of charities, it is clear that the United Kingdom's international position in biomedical science would be worse than it is now.

It is doubtful that the increase in private sources of funding is sustainable indefinitely. So, if government funding continues to decline, it is likely that there will be a reduction in the United Kingdom's gross scientific output and impact. Given the evidence that a strong, locally funded science base is important for technology innovation in industry (10), our results would argue for a sustained or increased commitment by government funding to science in the United Kingdom.

Jonathan Grant Grant Lewison
Unit for Policy Research in Science and
Medicine (PRISM),
Wellcome Trust,
210 Eston Road,
London NW1 2BE, United Kingdom

REFERENCES AND NOTES

1. R. May Science 275, 793 (1997).
2. May's Policy Forum was derived in part from a benchmarking study [The Quality of the UK Science Base (Office of Science and Technology, U.K. Department of Trade and Industry, London, 1997)], which in turn draws heavily on an analysis of Australian research [Australian Science: Performance from Published Papers (Australian Government Publishing Service, Canberra, 1996)].
3. The G7 countries are Canada, Germany, France, Italy, Japan, the United Kingdom, and the United States.
4. In (1), the citation period is given as 1981-84 [table 1 in (1)], but in the U.K. report and in the Australian study [(2), tables 5 and 2.1, respectively) it is given as 1981-94. We have assumed that the dates in May's Policy Forum are a typographic error.
5. One way of estimating which funding year is relevant is to analyze the age distribution of articles cited in a leading general science journal. For the 1994 editions of the journal Nature, the median age of papers cited is 3 to 4 years. If the age distribution of papers cited in 1994 is similar to the age distribution of papers cited in all previous years back to 1981, then the median year of publication, for papers cited in Nature between 1981-94, is 1985/6. The research reported in the cited papers would plausibly have been funded several years previously. Therefore, if Nature is representative of science in general (spanning many disciplines), it is most reasonable to compare funding in the early to mid 1980's with citations achieved during 1981-94.
6. Figures for government civil expenditure on R&D and Gross Expenditure on R&D (GERD) for 1986-91 were taken from Table 7.7 and 7.2, respectively, from the government report Science, Engineering and Technology Statistics, 1996, Her Majesty's Stationery Office, London, where published (1996). The figures were adjusted to 1991 prices with the use of GDP deflators presented in footnote 2 of Table 2.2 of the same publication.
7. Government civil expenditure on R&D continued to decline in the United Kingdom to £2.65 billion (1991 prices) in 1994 (4).
8. J. Anderson MRC News, (Autumn/Winter 1996), pp. 14-16.
9. Science & Engineering Indicators 1993 (National Science Foundation, Washington, DC, 1993), appendix table 5-23, and The European Report on Science and Technology Indicators 1994 (European Commission Publications, Luxembourg, 1994), table I.11.B.
10. J. Anderson, N. Williams, D. Seemungal, F. Narin, D. Olivastro, Technology Anal. Strategic Manage. 8(2), 135 (1996); F. Narin and D. Olivastro, Proceedings of the Sixth Conference of the International Society for Scientometrics and Informetrics, Jerusalem, Israel, 16-19 June 1997, pp. 305-312.
11. We are grateful to J. Anderson and D. Seemungal for helpful discussions and to R. Cottrell for performing some of the analysis. Funded by the Wellcome Trust.

Response: First, in that part of my Policy Forum which focused on the ratio of outputs (measured by papers or citations) to inputs (in terms of funding), I emphasised the marked--more than twofold--differences between such ratios for the United States, United Kingdom, and Canada (along with other countries such as Switzerland and Sweden) compared with the other four of the G7 nations (1). Although readers may have misperceived my message as one of U.K. chauvinism, the intent was to air some tentative speculations on the reasons for these differences in "cost effectiveness," which have sparked discussion and controversy (2). The main feature of figure 1 in the comment by Grant and Lewison is to confirm that these marked differences among the G7 have persisted over the extended interval 1986-91; the gap is so marked that the key to the figure is inserted between the top three lines and bottom four.

Table 2. Citations per £million spent. Similar to Table 1, except the outputs are citations to papers published in 1993 and 1996. The latter have, on average, many fewer citations. Country Ratio for 1993/1990 Ratio for 1996/1993 SBRD HERD SBRD HERD Canada 106 183 4.7 7.9 France 57 113 2.5 4.6 Germany 63 111 2.5 4.5 Italy 44 84 2.1 4.1 Japan 26 42 1.5 1.5 Netherlands 101 172 4.5 7.6 Sweden 112 123 5.4 6.1 Switzerland 164 196 7.1 8.4 United Kingdom 130 215 6.2 10.1 United States 96 139 4.1 6.2

Second, Grant and Lewison correctly observe that my "output/input" calculations are rough, in two respects: Government civil expenditures on R&D are an unsatisfactorily coarse measure of what produces the output of the "science base"; and there are time lags between inputs of relevant funding and outputs of papers or, even more, citations. But, having made these telling points, Grant and Lewison do not pursue them. Instead, they repeat my rough calculation, dividing the average citations over the span 1981-94 by total government civil R&D spends for a range of years, with the conclusions noted above.

Taking on board Grant and Lewison's constructive criticisms, I have made better estimates of "output/input" ratios (Tables 1 and 2). The first problem is how to measure "input." Government civil spend on R&D includes both too much and too little: too much in that it includes money spent on R&D to underpin policy, which does not typically lead to publications as might be counted by the Institute for Scientific Information (ISI); too little in that support for the "science base" from charities and "business enterprise" (industry, and so forth) are omitted. But consistently collected data about appropriate expenditure are not easily compiled. The Organization for Economic Cooperation and Development (OECD) publishes statistics for expenditure on R&D performed in higher education (HERD) (3), which is probably a better measure of "input" than total government civil R&D spend; but although roughly 80% of the U.K. scientific papers come from universities (including teaching hospitals), such patterns vary from country to country. A better "input" measure is arguably the total "science base" expenditure on R&D (SBRD), defined as all R&D carried out in universities and nonprofit making institutions, irrespective of funding sources, including intramural government funded civil R&D, mostly at government research establishments. The U.K. Office of Science and Technology has put together estimates of such expenditure for several countries, drawn from published OECD statistics on the breakdown of national gross expenditure on R&D (GERD) data, but these numbers arguably also suffer from problems of comparability owing to national structural differences, although possibly less so in aggregate than the OECD HERD numbers (4). The second, and easier, problem is how to take account of time lags between inputs and outputs. If the output is scientific papers, I divide the total papers (5) published in any one year by the input (HERD, or SBRD, or total government civil spend) 3 or 4 years earlier. If the output is citations, then one correspondingly counts citations to the papers in question.

Table 1. Papers per £million spent. Ratio between output of papers (5) to input of SBRD or HERD expenditure 3 years earlier (3, 4). Ratios are given at two time points: 1993/90 and 1996/93. Country Ratio for 1993/1990 Ratio for 1996/1993 SBRD HERD SBRD HERD Canada 16.4 28.4 14.3 24.0 France 9.3 18.4 9.1 16.5 Germany 9.8 17.2 7.9 14.1 Italy 7.8 14.8 8.5 16.4 Japan 4.9 8.0 7.0 7.0 Netherlands 13.5 22.9 13.1 22.2 Sweden 15.4 16.9 16.7 18.7 Switzerland 17.9 21.4 15.3 17.9 United Kingdom 18.0 29.8 17.5 28.6 United States 11.3 16.4 10.1 15.2

I have calculated (Table 1) the ratios between outputs of scientific papers, in 1993 and 1996, to inputs of HERD or SBRD expenditures 3 years earlier, for an admittedly arbitrarily chosen set of countries (the G7 plus Switzerland, Sweden, and the Netherlands). The output in citations to the papers published in 1993 and in 1996 is also presented (Table 2) (obviously, 1996 papers have on average accumulated fewer citations than 1993 ones). The patterns in these two tables are striking and fairly stable over the 3-year interval of inputs, 1990-93.

The second part of the comment by Grant and Lewison underlines the important contribution made by charities to the U.K. "science base," especially in biomedical areas. I strongly endorse their views. In the U.K. in 1993, funding from the Wellcome Trust and other charities accounted for roughly 10% of the total "science base" income, a higher proportion than for any other country listed (Tables 1 and 2). Similarly, business enterprise support also accounts for about 10% of the total U.K. "science base" expenditure, again a higher proportion than for other countries listed (Tables 1 and 2).

Any attempt to compare output/input ratios among countries, whether by scientific papers or by citations, is bedeviled by all manner of difficulties in compiling truly comparable statistics and by ineluctable biases inherent in counting papers or, even more, citations (6). Even so, some of the differences in such crude measures of the cost efficiency of research among the G7 and other countries (Tables 1 and 2) are so large as to defy explanation as statistical artifacts. These patterns deserve more attention than they have so far received (7).

Robert M. May
Office of Science and Technology,
Albany House,
94-98 Petty France,
London, SW1H 9ST, United Kingdom

REFERENCE AND NOTES

1. R. M. May, Science 275, 793 (1997) [Free Full Text] , first column on p. 796.
2. G. R. Barreto, ibid. 276, 882 (1997) [CrossRef]; S. Bauin, ibid., p. 883; I. Gómez, M. Bordons, J. Camí, ibid., p. 884; S. Herskovic, ibid., p. 884; R. M. May, ibid., p. 885.
3. Main Science and Technology Indicators (Organization for Economic Cooperation and Development, Paris, 1997).
4. These estimates, and accompanying discussion, will be published soon. I will also display them on the website: http://www.open.gov.uk/ost/osthome.htm
5. The papers, and citations, are for the 20 fields of science, engineering, and medicine used in the Australian study and in the partly derivative OST study: See reference 2 of the comment by Grant and Lewison.
6. See the lengthy notes 6 and 7 in (1).
7. I thank J. Grant, G. Lewison, and J. Anderson of the Wellcome Trust for their helpful collegiality and courtesy, and R. Dowdell, K. Root, D. Barker, and T. Quigley of OST for their help with Tables 1 and 2.