Language had long been one of AI's hardest challenges. In 2017 a new kind of neural network design changed that, leading within five years to systems that could write essays, answer questions and hold fluent conversations.

Attention Is All You Need

Earlier neural networks for language read text one word at a time, in order, which made them slow to train and prone to forgetting earlier words in long passages. In June 2017, eight researchers at Google published a paper with the title “Attention Is All You Need.” It introduced the transformer, an architecture built around a mechanism called attention, which lets the network consider all the words in a passage at once and weigh how much each word should influence the interpretation of every other. For example, in the sentence “The animal didn't cross the street because it was too tired,” attention helps the model link “it” to “animal” rather than “street.” Because transformers process words in parallel, they could be trained efficiently on enormous amounts of text using GPUs.

Pretraining on the internet

Researchers discovered a powerful recipe. First, pretrain a large transformer on huge amounts of text, from books, websites and other sources, by having it repeatedly predict the next word, or a hidden word, in a passage. To make good predictions, the model must learn grammar, facts and patterns of reasoning. Then fine-tune it for particular tasks. In 2018, Google released BERT, which improved search and many language tasks. The research lab OpenAI, founded in 2015 as a nonprofit, released a series of models called GPT, for Generative Pre-trained Transformer.

GPT-2 and GPT-3

In February 2019, OpenAI announced GPT-2, which could generate surprisingly coherent paragraphs. It initially withheld the full model, citing concerns about misuse such as generating fake news, then released it in stages. Critics called this a publicity move; others saw it as a first experiment in responsible release.

In 2020, GPT-3 arrived, with about 175 billion parameters, the adjustable numbers inside a network, far larger than its predecessor. It could write essays, poems and computer code, translate and answer questions, often given only a few examples in the prompt, without special training for each task. Researchers called this few-shot learning.

Scaling laws

In 2020, researchers at OpenAI published findings on scaling laws: the performance of language models improved predictably as models, datasets and computing power grew larger. This encouraged companies to build ever bigger models, investing in enormous data centers. Some researchers also observed that larger models seemed to gain new abilities, such as solving arithmetic problems or following complex instructions, that smaller ones lacked, though whether such abilities truly emerge suddenly is debated.

Teaching models to follow instructions

Raw language models predict text; they do not necessarily do what users want, and they can produce harmful or false content. In 2022, OpenAI described a method called reinforcement learning from human feedback, or RLHF. Human reviewers compared model responses and rated which were more helpful and appropriate, and the model was trained to prefer the kinds of answers people rated highly. This made models much better at following instructions and more cautious, though not reliably accurate. Other approaches, such as training models with written principles, were developed by labs including Anthropic, founded in 2021 by former OpenAI researchers with a focus on AI safety.

Science: AlphaFold

Not all breakthroughs involved language. Proteins, the molecular machines of life, fold into complex shapes that determine their functions, and working out a protein's shape from its sequence of building blocks had been a grand challenge of biology for fifty years. In 2020, DeepMind's AlphaFold 2 predicted protein structures with accuracy comparable to laboratory methods in an international competition. In 2021 and 2022, DeepMind and the European Bioinformatics Institute released predicted structures for over two hundred million proteins, nearly all known to science, freely available to researchers worldwide. In 2024, Demis Hassabis and John Jumper of DeepMind shared the Nobel Prize in Chemistry for this work, with the biochemist David Baker. The same year, Geoffrey Hinton and John Hopfield received the Nobel Prize in Physics for foundational work on neural networks.

Images from words

In 2021 and 2022, systems such as OpenAI's DALL-E, Midjourney and Stable Diffusion began generating detailed images from text descriptions. Stable Diffusion, released in August 2022, was openly available, allowing anyone to run it. Artists raised objections that these systems were trained on their work without permission or payment, beginning legal and ethical disputes that continue.

Worked example: what does a language model do?

Understanding the basic mechanism helps explain both the power and the flaws of large language models.

1. Training. The model reads vast amounts of text and adjusts billions of parameters to get better at predicting the next piece of text.
2. Generation. Given a prompt, it predicts a likely next word, adds it, then predicts the next, and so on.
3. Power. Because predicting text well requires capturing grammar, facts, style and reasoning patterns, the model can do many tasks it was never explicitly taught.
4. Flaws. It can produce fluent text that is false, often called a hallucination, because sounding plausible is not the same as being true. It may reproduce biases in its training text.

Whether this amounts to understanding is still debated, echoing Turing and Searle. For practical purposes, the key lesson is to treat a language model's output as a draft to be checked, not an authority, especially on facts that matter.