An artist’s conception of a world overloaded by augmented reality.

Still from HYPER-REALITY (2016) by Keiichi Matsuda

Smart windshields could transform your driving experience, but what happens when they get hacked?

In the future, driving a car could feel like playing a video game. “Smart” windshields might digitally transform your view of the road, highlighting lane markers, warning about collisions, and even displaying navigational arrows on the pavement. But what happens if a hacker distorts your view or a pop-up advertisement suddenly covers a stop sign? Researchers at the University of Washington (UW) have designed a prototype system that could block dangerous or annoying distractions in augmented reality (AR).

Franziska Roesner, a computer scientist at UWSeattle who co-authored the study, says the team wanted “to help people think about the security and privacy issues before the technologies are widespread and set in stone.” Their solution, which her UW co-author Kiron Lebeck named Arya—after the Game of Thrones character—is a program that manages graphics from multiple software applications on a single device, much like an air traffic controller supervising a set of runways. For an app to land a graphic on a piece of the screen’s real estate, it first needs the OK from Arya. To make sure graphics from hacked or poorly written apps don’t interfere with each other—or the real world—Arya enforces 10 rules (with many more possible). Among them:

  • Don’t allow any AR objects to block other AR objects.
  • Don’t obscure pedestrians or road signs.
  • Avoid the abrupt movement of AR objects.

The program deals with offending content by moving it, hiding it, or making it see-through. “Ultimately the guidelines they developed are a really good starting point,” says Missie Smith, an industrial engineer at Virginia Polytechnic Institute and State University in Blacksburg who works on AR for driving, and was not involved in the research.

The researchers tested Arya on virtual devices in three virtual locations. At “home,” where people might want to play AR games, have AR pets, or watch an AR TV, Arya removed a giant spider blocking the user’s view, slowed down a highly distracting virtual cat, and then prevented a pop-up message from obscuring the same cat. At the “office,” Arya increased the transparency of virtual objects obscuring people and exit signs. On the “road,” Arya hid an email pop-up and deleted virtual ads hiding pedestrians in the road. The images in all three scenes came from model AR apps similar to Pokémon Go that displayed graphics over the “real” world they had created.

More rigorous testing found that even when adjusting 48 offending graphics 30 times a second, Arya’s computations did not slow the virtual interface, the researchers will report later this month at the IEEE Symposium on Security and Privacy.

Arya could be baked into AR operating systems designed by Microsoft, Facebook, or Apple. But the researchers say they’d first want to develop more complex and intelligent guidelines. For example, instead of hiding content, it could suggest alternate locations to the app. It could also prioritize content.

Blair MacIntyre, a computer scientist at Georgia Institute of Technology in Atlanta who studies AR, sees a need for a system like Arya in safety-critical places such as cars, factories, hospitals, or even battlefields, where graphics might hide important features. But such a system might seem too heavy-handed in other situations, or it could have unintended consequences. Some apps might not work properly, or advertisers might become frustrated with limitations on ad placement. MacIntyre worries about overregulation in the digital realm. “If we start adding controls over what can be displayed where,” he says, “it just seems like a path that is fraught with danger.”

He also doesn’t want virtual game characters disappearing every time he happens to glance at a stop sign near his house. “As I’m standing in my front yard doing ninja moves on the zombies,” he says, “that would just be absurd.”