Simple Machine Learning
A field journal · v.01
← Back
FiledSeptember 16, 2026·5 min read·By The Editor

How Transformers Work (the AI kind)

A transformer is a next-word predictor built from one repeated block — every word looks at every other word, then thinks on its own. Attention is the only genuinely new idea, and it fits in one picture.

llmarchitectureintuition

This is about the transformer in AI — the architecture behind GPT, Claude and Gemini. If you are here about the electrical device, you want a different page entirely.

A transformer is a next-word predictor built from one repeated block. The block does two things: every word looks at every other word, then every word is processed on its own. Stack that block a few dozen times and you have the architecture running essentially every modern language model.

The genuinely new idea is the looking part, called attention. Start there.

Attention: the one new idea

Take the sentence "The animal did not cross the street because it was tired." What does it refer to?

You know instantly: the animal. Streets do not get tired. To answer, you connected a word to another word seven positions earlier, using meaning rather than proximity.

Figure 1 · attention, the only new idea
Attention weights for the word 'it'The sentence 'The animal did not cross the street because it was tired'. Lines run from the word 'it' back to every earlier word, thickest to 'animal', showing where the model looks to resolve the pronoun.TheanimaldidnotcrossthestreetbecauseitwastiredWHAT DOES “IT” REFER TO?thicker line = more attention paid↑ the word doing the looking↑ 62% of its attentionswap “tired” for “wide” and the thick line moves to “street”
Every word gets to look at every other word and decide how much each one matters to it. Here, it pays most of its attention to animal — which is how the model resolves the pronoun. Nothing told it to do this; the weights that produce it were learned.

That is attention. Every word gets to look at every other word and decide how much each one matters. The connections are not rules anybody wrote — they are the product of weights learned by gradient descent, exactly like every other weight in the model.

In plain English

Older architectures read left to right, carrying a summary as they went — so by word 40 the memory of word 3 had gone thin. Attention gives every word direct access to every other word, at full strength, all at once. Nothing has to survive a long journey.

How a word decides where to look

Each word produces three vectors. The metaphor that makes it stick is a search desk.

Figure 2 · query, key, value
Query, key and valueA search-desk metaphor: the query is what you are looking for, each key is a label on a drawer, and the value is what is inside the drawer you open.QUERYwhat I am looking for“I am a pronoun. Who is my noun?”KEYwhat each word advertises“I am an animal, a subject, singular.”VALUEwhat gets handed backthe actual content that flows onwardTHREE VECTORS PER WORDmatch every query against every key → those scores are the attention weights
Each word produces three vectors. Comparing one word's query against every word's key — with a dot product, naturally — gives the attention weights in Figure 1. Those weights then blend the values together.

To find where word i should look, compare its query against the key of every word — using a dot product, which measures exactly the "how much do these two agree" that is wanted here. Big score, big attention. Then use those scores to take a weighted blend of the values.

Attention(Q,K,V)=softmax ⁣(QKTdk)V\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V
In plain English

Read it right to left. QKT is every query dotted with every key — all the match scores at once. ÷√dk keeps those numbers from getting huge when the vectors are long. softmax turns them into percentages that sum to 1. Multiplying by V blends the values in those proportions. That formula is the whole mechanism.

Models run many of these in parallel — heads — because a word needs to track several relationships at once: what it agrees with grammatically, what it refers to, what modifies it.

The block, and the stack

Figure 3 · the architecture, entire
The transformer block, stackedTokens are embedded, position is added, then the same two-part block — attention followed by a small feed-forward network — repeats dozens of times before a final layer turns the result into next-word probabilities.text → tokens+ positionattentionevery word looks at every otherfeed-forwardeach word thinks on its own↑repeat thiswhole block12–100+ timesnext-word probabilitiesONE BLOCK, REPEATEDthat is the whole architecture
There is far less here than the name suggests. Two operations — look around, then think alone — stacked several dozen times. Depth is the only thing that separates a small transformer from a large one.

After attention comes a small feed-forward network applied to each position separately. If attention is gathering context, this is digesting it. Slightly counter-intuitively, most of a transformer's parameters live here rather than in attention.

Then the block repeats. Twelve times in a small model, a hundred or more in a large one. Early layers handle surface patterns; later layers handle abstraction. Depth is essentially the only structural difference between a small transformer and a frontier one.

The part everybody forgets: position

Attention has no sense of order. It sees a bag of words, all equally reachable. "Dog bites man" and "man bites dog" would be identical to it.

The fix is to add a position signal to each token's vector before the stack begins. That is the entire job of positional encoding — and it is why the concept exists at all.

Why this architecture took over

  • It parallelises. Recurrent models had to process word 1 before word 2. Attention scores every pair simultaneously, which is what makes training on a GPU cluster tractable. This is the real reason transformers won.
  • Distance costs nothing. Word 1 and word 1000 are one step apart.
  • It scales predictably. Add layers, add data, get a better model — reliably enough to justify spending hundreds of millions on a training run.

The cost: comparing every word with every other word is quadratic. Double the context, quadruple the work. That single fact is why context windows were small for years and why so much research goes into making attention cheaper.

The vocabulary

Attention head
One independent 'who should I look at' calculation. A layer runs many in parallel — one may track grammar, another long-range reference.
Multi-head attention
All those heads at once, their answers concatenated.
Feed-forward
A small two-layer network applied to each position separately. Most of the model's parameters live here, not in attention.
Residual connection
Add the input back to the output. Lets gradients reach the bottom of a hundred-layer stack without vanishing.
Layer norm
Rescales the numbers between steps so they stay in a sane range.
Positional encoding
The only reason the model knows word order, since attention itself is order-blind.

The short version

  • A transformer predicts the next token. Everything else is plumbing.
  • Attention lets every word look at every other word and weight what matters.
  • Query/key/value: what I want, what I advertise, what I hand over. Matching is a dot product.
  • The block — attention, then feed-forward — repeats dozens of times. Depth is the difference.
  • Position must be added explicitly; attention is order-blind.
  • It won because it parallelises, not because it is clever.
← Archive— end of essay —Front page →