In the early 2010s, three things came together: much larger datasets, much faster hardware, and improved neural network techniques. The result was a sudden leap in performance that researchers call the deep learning revolution. “Deep” refers to networks with many layers of artificial neurons, each learning to detect more complex features than the one before.
The persistent believers
Through the years when neural networks were unfashionable, a small group kept working on them, including Geoffrey Hinton in Toronto, Yann LeCun in New York and Yoshua Bengio in Montreal, supported in part by Canadian research funding. In 2006, Hinton and colleagues showed new ways to train deep networks one layer at a time, renewing interest. In 2019 the three received the Turing Award, often called the Nobel Prize of computing, for their work on deep learning.
Data: ImageNet
Learning systems need examples. The computer scientist Fei-Fei Li recognized that progress in computer vision was limited by small datasets. Starting in 2007, her team built ImageNet, a database of over fourteen million images labeled by category, such as dog breeds, vehicles and household objects. The labeling was done largely by thousands of paid online workers. From 2010, an annual competition challenged researchers to build systems that could classify ImageNet images correctly.
Hardware: graphics chips
Training large networks requires enormous numbers of simple calculations performed in parallel. Graphics processing units, or GPUs, designed to render video game images, turned out to be ideal. In 2007 the company Nvidia released software that made it easier to use its graphics chips for general computing. Researchers began training networks many times faster than before.
AlexNet, 2012
In the 2012 ImageNet competition, a deep convolutional network built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, later called AlexNet and trained on two graphics cards, achieved an error rate of about fifteen percent on the main task, far better than the roughly twenty-six percent of the next best entry. The margin stunned the field. Within a few years, nearly every leading entry used deep learning, and by 2015 systems were matching or exceeding human performance on this specific benchmark. Technology companies rushed to hire deep learning researchers.
Speech, translation and assistants
Deep learning quickly transformed other fields. Speech recognition error rates dropped sharply, making voice assistants practical; Apple had introduced Siri in 2011, and others followed. In 2016, Google switched its translation service to a neural network system, producing much more fluent translations. In 2012, a Google research team showed that a large network trained on frames from YouTube videos, without labels, learned on its own to recognize cats, among other things.
DeepMind and AlphaGo
The London company DeepMind, founded in 2010 by Demis Hassabis, Shane Legg and Mustafa Suleyman, and bought by Google in 2014, combined deep learning with reinforcement learning. In 2013 to 2015, it showed a system that learned to play dozens of classic Atari video games from the raw screen pixels, reaching human-level performance on many.
Its most famous achievement was in Go, the ancient board game, which has far more possible positions than chess and had resisted brute-force search. Experts had thought a computer champion was at least a decade away. In March 2016, in Seoul, DeepMind's AlphaGo defeated Lee Sedol, one of the world's top players, four games to one. In the second game, AlphaGo played move 37, an unconventional move that commentators first thought was a mistake, but which proved brilliant. In the fourth game, Lee Sedol won with a creative move of his own. A later version, AlphaZero, learned chess, shogi and Go entirely from self-play, without human examples, and surpassed all previous programs.
Generating images
In 2014, the researcher Ian Goodfellow introduced generative adversarial networks, in which two networks compete, one generating fake images and the other trying to detect them, so that the generator steadily improves. Such methods soon produced realistic faces of people who did not exist. By 2017, the term “deepfake” appeared for realistic fake videos made with these tools, raising concerns about misinformation and abuse.
Problems emerge
As deep learning spread, its weaknesses became clearer. Researchers showed that tiny, invisible changes to an image could fool a network into misclassifying it, called adversarial examples. Networks were often black boxes, hard to explain. And they reflected biases in their data. In 2018, the researchers Joy Buolamwini and Timnit Gebru, in a study called Gender Shades, found that commercial facial analysis systems misclassified the gender of darker-skinned women far more often than lighter-skinned men, with error rates of up to about thirty-five percent compared with under one percent.
Worked example: why 2012?
Neural networks had existed since the 1950s, and backpropagation since the 1980s. Why did the breakthrough come in 2012?
- Data. The internet and projects like ImageNet produced millions of labeled examples, far more than ever before.
- Compute. GPUs made training many times faster and cheaper.
- Techniques. Improvements such as better activation functions and methods to prevent overfitting made deep networks trainable.
- Persistence. A few researchers kept working on an unfashionable idea through decades of skepticism.
Many inventions wait for their moment: the idea exists, but the surrounding conditions are not ready. When they align, progress can appear sudden. The deep learning revolution was decades in the making, and its breakthrough depended as much on data, hardware and patience as on any single brilliant idea.