Read our COVID-19 research and news.

A representation of a neural network.

Akritasa/Wikimedia Commons

Brainlike computers are a black box. Scientists are finally peering inside

Last month, Facebook announced software that could simply look at a photo and tell, for example, whether it was a picture of a cat or a dog. A related program identifies cancerous skin lesions as well as trained dermatologists can. Both technologies are based on neural networks, sophisticated computer algorithms at the cutting edge of artificial intelligence (AI)—but even their developers aren’t sure exactly how they work. Now, researchers have found a way to "look" at neural networks in action and see how they draw conclusions.

Neural networks, also called neural nets, are loosely based on the brain’s use of layers of neurons working together. Like the human brain, they aren't hard-wired to produce a specific result—they “learn” on training sets of data, making and reinforcing connections between multiple inputs. A neural net might have a layer of neurons that look at pixels and a layer that looks at edges, like the outline of a person against a background. After being trained on thousands or millions of data points, a neural network algorithm will come up with its own rules on how to process new data. But it's unclear what the algorithm is using from those data to come to its conclusions.

“Neural nets are fascinating mathematical models,” says Wojciech Samek, a researcher at Fraunhofer Institute for Telecommunications at the Heinrich Hertz Institute in Berlin. “They outperform classical methods in many fields, but are often used in a black box manner.”

In an attempt to unlock this black box, Samek and his colleagues created software that can go through such networks backward in order to see where a certain decision was made, and how strongly this decision influenced the results. Their method, which they will describe this month at the Centre of Office Automation and Information Technology and Telecommunication conference in Hanover, Germany, enables researchers to measure how much individual inputs, like pixels of an image, contribute to the overall conclusion. Pixels and areas are then given a numerical score for their importance. With that information, researchers can create visualizations that impose a mask over the image. The mask is most bright where the pixels are important and darkest in regions that have little or no effect on the neural net’s output.

For example, the software was used on two neural nets trained to recognize horses. One neural net was using the body shape to determine whether it was horse. The other, however, was looking at copyright symbols on the images that were associated with horse association websites.

This work could improve neural networks, Samek suggests. That includes helping reduce the amount of data needed, one of the biggest problems in AI development, by focusing in on what the neural nets need. It could also help investigate errors when they occur in results, like misclassifying objects in an image.

Other researchers are working on similar processes to look into how algorithms make decisions, including neural nets for visuals as well as text. Continued research is important as algorithms make more decisions in our daily lives, says Sara Watson, a technology critic with the Berkman Klein Center for Internet & Society at Harvard University. The public needs tools to be able to understand how AI makes decisions. Algorithms, far from being perfect arbitrators of truth, are only as good as the data they’re given, she notes.

In a notorious neural network mess up, Google tagged a black woman as a gorilla in its photos application. Even more serious discrimination has been called into question in software that provides risk scores that some courts use to determine whether a criminal is likely to reoffend, with at least one study showing black defendants are given a higher risk score than white defendants for similar crimes. “It comes down to the importance of making machines, and the entities that employ them, accountable for their outputs,” Watson says.