Attention Is All You Need: the paper that built modern AI

In June 2017, eight researchers at Google published a short paper with a confident title: "Attention Is All You Need". It introduced the Transformer architecture, and it more or less founded the era of AI we live in. Strip the branding from any modern model — Claude, GPT, Gemini, Llama, Mistral — and underneath you find the same structural pattern: stacked attention layers relating every word to every other word, in parallel.

Before 2017, language models read sequentially. Recurrent networks processed one word at a time with a running memory, which made training slow — nothing parallelized — and memory short, losing the start of a long passage by its end. The paper's bold move was to throw recurrence out entirely and keep only attention, a mechanism that had previously been a bolt-on helper. The result trained faster, scored better, and — decisively — kept improving as you added data and compute. That scaling property is the quiet engine behind nearly every capability jump since.

The intuition fits in one sentence. In "The bank was next to the river," you know which bank because river appears nearby; self-attention is that move, made numerical — every token scores the relevance of every other token, the scores become weights, and each word's representation is rebuilt from what it attends to. Stack those layers and the model stops tracking just neighboring words and starts tracking structure, reference, and meaning across an entire document. No distance bias means a word at the end of a long context can reach the first word as easily as the previous one — which is how models hold long conversations together.

The paper's framing was narrow — English-to-German translation — but the architecture proved absurdly general: BERT for understanding in 2018, GPT for generation soon after, then vision, then protein folding. At twelve pages it's also one of the most readable landmark papers in the field, worth the hour even for non-specialists.

Nearly a decade on, attention really did turn out to be all we needed.