The rate at which scientists in Italy cite themselves and their compatriots is rising faster than in 10 other developed countries, according to a new study. The surge in Italy’s clubby citation behavior is likely the result of a 2010 law requiring productivity standards for academic recruitment or promotion, the study authors say.
The findings are a cautionary tale for research administrators who rely too much on citation metrics in allocating resources and making decisions on career advancement, says study author Giuseppe De Nicolao, an engineer at the University of Pavia in Italy. Linking professional advancement to citation indicators can prod scientists into unintended behaviors and make the metrics unreliable, he says.
The findings are “disturbing,” says Ludo Waltman, a bibliometric expert at Leiden University in the Netherlands who was not involved in the study. To limit questionable citation practices, Waltman says, the Italian evaluation system should exclude self-citations and consider factors such as a researcher’s experience and activities in addition to citation counts.
After the 2010 law was passed, Italy began to regulate academic recruitment and promotion using indicators such as citation counts. It was intended to address concerns about nepotism and a lack of meritocracy.
Under the policy, academics can’t seek a job or a promotion as an associate or full professor unless they meet at least two of three indicators of research productivity. In fields such as medicine and natural sciences, these indicators include the number of publications, the number of citations received, and h-index—a combined measure of productivity and citation impact.
Previous studies have found that the 2010 policy induced a rise in self-citations. However, those studies didn’t look at the proportion of each country’s publications cited by other scholars within the same country, says Alberto Baccini, a scientometrics expert at the University of Siena in Italy. These intranational citations could reveal “citation clubs,” a subtle form of manipulation in which groups of scientists cite each other to boost their citation scores, Baccini says.
So Baccini, De Nicolao, and their team set out to develop an indicator of inwardness, which measures both self-referential and intranational citations. The researchers scoured Elsevier’s Scopus database, one of the world’s largest for research literature, for citation counts between 2000 and 2016 for researchers in the G-10, a group of 11 developed countries. To calculate a nation’s inwardness, the team counted citations by a country’s authors to papers authored in that country and divided this figure by the total number of citations accrued by the country.
All of the nations showed modest rises in inwardness over time, which can be explained, paradoxically, by a growth in international collaborations. These expand the number of papers from participating countries that could be cited. Take, for example, a paper co-authored by research collaborators in Italy and France: Any citation to this paper from an Italian- or French-authored publication will count as an intranational citation for both Italy and France.
Beginning in 2010, however, Italy’s inwardness started to increase rapidly, surpassing France, Japan, and the United Kingdom, the researchers found. The surge could not be attributed to collaborations, because Italy’s rate of growth for international collaborations between 2000 and 2016 was anemic compared with other nations. By 2016, about 31% of the Italy’s citations came from authors within its borders—more than any other country except the United States, a research powerhouse where many intranational citations are expected.
Because the trends changed after the introduction of the 2010 policy, it’s likely that authors in Italy adopted opportunistic behaviors, including massively citing their own work and that of colleagues, to reach their country’s policy targets, the researchers report today in PLOS ONE.
Marco Seeber, a science policy researcher at the University of Ghent in Belgium, says the growth in Italy’s inwardness is “striking.” In March, Seeber investigated the country’s use of citation metrics and found substantial increases in self-citations after the 2010 policy. “The policy was motivated by worthy intentions,” he says. “But bibliometric indicators should be used to inform rather than determine evaluations.”
Seeber says it’s unclear how much of Italy’s inwardness is from self-citation versus citation clubs. To uncover these clubs, one would need to examine individual papers to discriminate legitimate from bogus citations, he says.
John Ioannidis, a physician-scientist at Stanford University in Palo Alto, California, suspects citation clubs are behind Italy’s trend. Ioannidis, who created a database that revealed hundreds of extreme self-citing researchers, says the new study provides yet another example of how metrics can be misused. He notes that self-citations are necessary if a study builds on previous work by the authors or their colleagues. “But if someone has amassed more than half of their citations from themselves or their co-authors, that’s pretty weird,” he says. “You have to take a closer look.”