Human gamers know just how hard it is to win a new spin on the classic computer game Quake: In a mazelike arena, they must work with other players to capture floating flags—all while dodging deadly gunfire. Now, for the first time, artificial intelligence (AI) has mastered teamwork in a complex first-person video game, coordinating its actions with both human and computer teammates to consistently beat opponents.
“The scale of the experiments is remarkable,” says Michael Littman, an AI expert at Brown University. Getting AI agents to work together is incredibly tough, he says.
Although AI can drive cars and easily defeat the world’s greatest chess and Go players one on one, researchers have struggled to get it to master teamwork. The practice may seem intuitive to us, but predicting how others will behave—a crucial component of working on a team—adds a whole new level of complexity and uncertainty for AI to deal with.
In the new study, researchers got AI bots to teach each other to work as a team. Their classroom was a simplified version of 1999 first-person shooter, Quake III Arena. The game involves two teams that navigate around a 3D map to retrieve a flag from their opponent’s base and return it to theirs. The team with the most captures after 5 minutes wins. Players also fire a laser to tag enemies, sending them back to their home base.
To train the AI to work as a team, the scientists created 30 different bots and pitted them against each other in a series of matches on randomly generated maps. The bots trained using brain-inspired algorithms called neural networks, which learn from data by altering the strength of connections between artificial neurons. The only data the bots had to learn from was the first-person visual perspective of their character and game points, awarded for things like picking up flags or tagging opponents.
Initially the bots acted randomly. But when their actions scored points, the connections that led to the behavior were strengthened through a process called reinforcement learning. The training program also culled the bots that tended to lose and replaced them with mutated copies of top performers inspired by the way genetic variation and natural selection help animals evolve.
After 450,000 games, the researchers arrived at the best bot, which they named For The Win (FTW). They then tested it in various matches with a mirror FTW, an FTW bot missing a crucial learning element, the game’s in-built bots, and humans. Teams of FTW bots consistently outperformed all other groups, though humans paired with FTW bots were able to beat them 5% of the time, they report today in Science.
The FTW bots learned to play seamlessly with humans and machines, and they even developed classic cooperative strategies, says study co-leader Max Jaderberg, an AI researcher at Google-owned DeepMind in London. Those strategies included following teammates in order to outnumber opponents in later firefights and loitering near the enemy base when their teammate has the flag to immediately grab it when it reappears. In one test, the bots invented a completely novel strategy, exploiting a bug that let teammates give each other a speed boost by shooting them in the back.
“What was amazing during the development of this project was seeing the emergence of some of these high-level behaviors,” Jaderberg says. “These are things we can relate to as human players.”
The approach is still a long way from working in the real world, Jaderberg adds. But the advance is good for more than computer games. If AI can learn to work in teams, it could make everything from self-driving cars that avoid crashes by coordinating with each other to robotic surgical assistants that help out doctors during procedures.
Still, Littman warns against extrapolating too much from a relatively simple computer simulation. “It could be that the details of this particular game require only a very narrow slice of what we think of as teamwork,” he says. And that, he says, means there’s no guarantee the same approach would teach AI to work as a team on other tasks.