Gambling is frowned upon in many circles. But what if the gamblers are researchers betting on how each other's experiments will turn out, and the results are used to improve science itself? A group of psychologists has found that their collective gambling—with real money—predicted the outcome of attempts at replicating experimental results better than their own expert guesses. They propose that this type of gambling setup, known as a prediction market, could become part of how science gets done.
Science is inefficient. By most estimates, about half of the results that get published in peer-reviewed journals are false positives—the main findings turn out to be a fluke. The best way to reach consensus on which results to trust is to replicate experiments in different labs. The problem is that few scientists want to spend their time directly replicating each other's work. If only there was a way to gauge which experiments were more likely to be false positives.
That's where prediction markets come in. The "wisdom of the crowd" can be a powerful tool for estimating an unknown measurement, but a motivated crowd is even sharper. For example, the average guess of 100 people about the weight of a cow or the number of rice grains in a jar is usually close to correct, even if most of the guesses are far from the mark. But when those people have skin in the game, for example by placing bets instead of just guessing, the accuracy is often far higher.
A team led by Anna Dreber, a behavioral economist at the Stockholm School of Economics, set up prediction markets based on the Reproducibility Project: Psychology, which has orchestrated the replication of studies from top psychology journals. They recruited psychologists involved with those replications to take part in two prediction markets—first in November 2012 and then in October 2014—that each ran for 2 weeks. The researchers used a commercial online prediction market service. The markets had about 50 traders each and endowed the traders with $100 each for placing bets. (The money came from Dreber's grant.) About a dozen of the traders were actually involved with some of the experimental replications, so they weren't allowed to place bets on those.
In a prediction market you can change your bets on the fly. Here's what it looks like: The market starts with every option set at $0.50. Let's say you have a strong hunch about two particular options—experiment X will succeed and Y will fail. So you start placing your bets on those two options. But other people have the same hunch and the price of those options quickly rises. Once the price reaches $0.80, you look elsewhere and get interested in experiment Z. Most people have bet on it failing, so the price is only $0.15. You think its chances are more like 50/50, so that's a bargain. You sell off some of your experiment X shares and plop that down with your remaining money on a success for experiment Z. This horse-trading continues until the close of market—a predetermined time known to all. Once the real results from the experiments come in, the correct bets pay off.
Before the markets opened, every trader took a survey, declaring the chance of success (0% to 100%) for each experimental replication. If the prediction market really does work, then it has to be not just better than random but better than this survey at predicting the outcomes. Otherwise, you could just ask scientists rather than go through the fuss of betting.
The markets arranged by Dreber’s team were bustling. Contracts changed hands 2500 times, with bets placed about as frequently in favor of studies as against them. And once the dust settled, the price of betting that a study would succeed ended up at an average of $0.55. That means that the psychologists expected that about half of the replications would fail. But were they accurate?
Overall, they were too optimistic: Only 39% of the experiments successfully replicated. But the prediction market was still far more accurate than the survey. As individuals, the psychologists did not guess the outcomes significantly better than random: Only 58% of their individual premarket guesses proved correct. But based on the market collectively—predicting success or failure based on whichever bet was more expensive by the end—gambling got it right 71% of the time, the team reports today in the Proceedings of the National Academy of Sciences.
Researchers who took part in the markets aren't sure what guided their decisions. "I would check the market on my walk into work in the morning," says Marcus Munafò, a psychologist at the University of Bristol in the United Kingdom. "I just bought and sold positions on the basis of my feeling for whether the current market position was over- or under-priced for each study." In the end, he nearly doubled his money to $180. Munafò spent it all on music—his theme song for scientific success is now Green Onions by Booker T. & the M.G.’s.
Running such markets is not easy, cautions John Ioannidis, a biologist at Stanford University in California who tried—and failed—to set up a prediction market for genetic experiments. "It was challenging to get investigators to play in the market." Although the results show that prediction markets really can predict, he wonders about fields such as disease genomics where, unlike psychological studies, "the pathways are extremely convoluted: It may be more difficult for prediction markets to make useful predictions, since we are facing an unpredictable black box." Another limitation is that "in order to judge whether the market has made successful choices or not, you need to do the replication anyhow, so you don't really avoid spending these extra resources."
Dreber has a plan to get around that: Scientists could bet on a large number of experiments, but only a random sample are then chosen for replication. "The bets for the studies where no outcome gets determined could be closed at the market price," she says. Then science really would become a horse race.