Last fall, the Allen Institute for Artificial Intelligence in Seattle, Washington, launched a challenge to Google Scholar, PubMed, and other online search engines by unveiling a service called Semantic Scholar. The program, originally trained on 2 million papers from the field of computer science, was intended to provide a search engine, driven by artificial intelligence (AI), to actually understand—to a limited extent—the content of published literature. Its corpus has grown to 4 million papers. And today, the institute is adding a new capability to Semantic Scholar with an equally ambitious aim: measuring the influence that a scientist or organization has had on subsequent research.
The tool, which focuses only on computer science for now but will expand to neuroscience by the fall and then to other subjects, can rank papers, authors, and institutions by a specific influence score. For instance, the tool finds that the most influential computer science is happening at the Massachusetts Institute of Technology in Cambridge. No surprise there. But the most influential computer scientist? It's Michael I. Jordan of the University of California, Berkeley, a pioneer of AI that few outside of his field recognize. "He's known as the Michael Jordan of machine learning," quips Oren Etzioni, director of the Seattle-based Allen Institute that created Semantic Scholar. (Click here for a list of the top 50 authors, and here for a list of the top 50 domains.)
If outsiders find Semantic Scholar’s rankings trustworthy, its numbers could get used by hiring and tenure committees. That’s because influence is a hard thing to measure. The old way is to count citations. Such counts have become a cornerstone of the academic publishing industry's metrics, with Thomson Reuters, Elsevier, and others selling access to bibliographic databases that enable users to run the numbers.
But not all citations are created equal—for instance, being cited as the inspiration for an entire paper is very different than a brief mention in its methods section. So the raw count can paint a misleading portrait of a scientist’s impact. And researchers complain that traditional citation-based metrics have helped create a "publish or perish" mentality, pressuring them to spit out papers as quickly as possible, regardless of the importance of the findings, in the hopes of racking up citations.
What is needed, some say, is a more direct measurement of a paper's actual influence on future research. So Etzioni's team built a new tool into Semantic Scholar that enables the creation of an “influence graph.” Most of the papers in its database are in PDF format, which is easy for a human to read but just looks like a blob of unstructured text to a computer. Reading more like a human requires machine learning, a technique that helps a computer make accurate guesses. For example, it must not only discover the different sections of the paper—introduction, methods, results—but discern the tone of how papers are being cited. So the Allen Institute team used machine learning to train a statistical model that detects all these features. Then the computer steadily improved its model by comparing its guesses with those of human experts who curated a sample of the papers.
For now the system only measures "direct influence" between papers that cite each other, says Etzioni, but future versions will account for the indirect influence of papers that cite papers that, in turn, cite other papers, and so on.
The tool debuted today at www.semanticscholar.org. Science asked Jeff Clune, a computer scientist at the University of Wyoming in Laramie, to take it for a test drive.
The first thing Clune did was look at his own neighborhood in the influence graph. "It is extremely fun," he says. "I can see which scientists have most influenced my own career, which scientists I have inspired the most, and the same for any other scientist." Most of the results were exactly what Clune expected—his mentors influenced him and he influenced his graduate students and postdocs—but there were some surprises. He wasn't familiar with the name of someone, for example, that has been doing extensive follow-up research inspired by Clune’s papers.
But besides the entertainment value, Clune believes the tool could have value in the academic hiring and promotion process. The committees that make those decisions are pressured to not just rank candidates by the success of previous work, but to predict each candidate’s future impact. Semantic Scholar tries to reveal what's "hot" in the field by measuring the "velocity" and "acceleration" of bodies of work, measurements of how rapidly others are citing certain work, and whether that is trending. Departmental committees are “already calculating that on the fly," says Clune, so those numbers will get used, he predicts.
But that aspect also worries him. To some extent, Semantic Scholar is "a black box," Clune says. "Will people understand where the numbers are coming from?"
Etzioni acknowledges the murkiness of how the algorithms produce the results. "It's always a trade-off in machine learning," he says. "One thing that helps is that you can see examples of where the numbers are coming from when you hover over the numbers."
Meanwhile, the Michael Jordan of machine learning is taking his top ranking in stride, but deflects credit. "Despite the mythologies that have historically been built up around particular individuals in science and mathematics, I personally believe that it is the networks that researchers belong to that are most predictive of their success," he says. "My own personal network is full of great people who have had great ideas."
Etzioni’s team is now setting Semantic Scholar loose on a massive corpus of research papers focusing on the brain. That tool and the influence rankings for brain researchers will debut at the Society for Neuroscience meeting in San Diego, California, on 12 November.