You have Q · K · V and the dot products. Then what?
The prompt is paris is in… and we want the contextualized vector for the last token, "in". Each token has already been turned into a Query, a Key, and a Value, and the Query has been dot-producted against the Keys. Press Next and watch the rest of a whole block happen one move at a time — scores become weights, the weights blend the Values, twelve heads combine, the result feeds the MLP, and every piece adds back into the running vector.
01Inside one block
From dot products to a finished block
Ten moves: the attention half (the 12 heads), then the MLP half. The first two moves are the projections and dot products; the player makes them concrete, then keeps going all the way to the handoff. Use Next / Back or the arrow keys.
↓that was 1 of 12 heads — all 12 run in parallel, each with its own Q/K/V
123456789101112
concatenate all 12 outputs → 12 × 64 = 768
↓mix the heads with the output projection W_O
attn_out · all heads (768)
↓add it back onto the original "in"
x_in + attn_out = contextualized "in"
in′ (contextualized)
↓sublayer ② — the MLP reads in′ alone (no other tokens)
in′768
→ up →
3072neurons + GELU
→ down →
mlp_out768
↓add mlp_out back onto in′
in′ + mlp_out = in″
in″ (this block's output)
↓in″ drops into the next block — still a 768-dim vector, not a word
LAYER 2
in″ arrives
→ runs the whole block again — its own 12 heads + its own MLP
A common misreading
It's tempting to chain Q · K · V as if Values get dot-producted too. They don't. The dot product is only Q · K → a score. The Values are never dotted — they're averaged, using the weights softmax produced. Q·K decides how much; V is what gets pulled in.
token vector x Key Value Query attn_out in′ (contextualized) mlp_out in″ (block output)
Why "in" can see "paris" but the model still works left-to-right: "in" is the last token, so it's allowed to look back at paris, is, and itself (causal masking blocks only future tokens). Every token actually does this in parallel — each builds its own Query and its own contextualized vector. We're just following "in" because it's the one whose next word we want to predict.
Two "dots", and they're not the same operation. In step 3, Q · K is a dot product — two vectors go in, one number comes out (the score). In step 5, 0.71 × V_paris is scalar multiplication — one number scaling every component of a vector, which stays a vector. So attention is: dot-product to get numbers (how much), then scale-and-add the Values by those numbers (what). The Values are never dotted against anything.
What the toy leaves out (on purpose). Real GPT-2 wraps each sublayer in a LayerNorm: the running vector is rescaled to a standard size just before attention reads it, again just before the MLP reads it, and once more at the very top before the word guess. It also divides every Q·K score by √64 = 8 before softmax, and adds a small learned bias at each projection. All of this is stability bookkeeping — it keeps training well-behaved but doesn't change the story on this page.
02The ten moves
The whole block in one list
1 · vectors inEach token is a vector. "in" carries no context yet.
2 · project× three learned matrices → each token's K, V, Q.
3 · scoreQ_in · each K → one relevance number per token.
4 · softmaxScores → weights summing to 100%.
5 · blend VWeighted average of the Values = one head's attn_out.
6 · 12 heads → W_OAll heads in parallel, concat (12×64=768), mix → full attn_out.
7 · add backx_in + attn_out = in′ (residual add).
8 · MLPin′ → up 3072 → GELU → down 768 = mlp_out. One MLP per block.
10 · next blockin″ (a vector, not a word) rises; repeat with fresh weights.
Two sublayers, one block. Steps 1–7 are the attention sublayer (the 12 heads); steps 8–9 are the MLP sublayer. Every one of GPT-2's 12 blocks is exactly this: attention, then MLP, each adding back into the residual stream. For what those 3072 neurons actually store and how GELU gates them, see How the MLP works ›
03Where the numbers come from
One loss, three different gradients
W_Q, W_K, W_V are not trained by three separate objectives — there's a single next-token loss for the whole model. They end up holding different numbers because each sits at a different point in the forward pass, so the gradient that flows back to each is different. Below: the forward pass in grey, the one loss at the bottom, and the gradient travelling back up in red — splitting because V plugs in later (at the blend) than Q and K (at the score).
forward pass ↓ grey · gradient backward ↑ red dashed · a single loss at the bottom
W_Q & W_K — via the scoreThey only touch the loss throughQ · K → softmax → the attention weights. Their gradient answers: “should this token have attended more or less?” Their roles are symmetric, and each one's update is computed through the other's current values — W_Q gets nudged toward the Keys it should match, W_K toward the Queries.
W_V — via the blendV enters later, at the weighted sum. Its gradient answers a different question: “once we decided to look at paris, what should we have pulled from it?” A shorter, separate route back to the same loss.
The error signal splits. Suppose the true next word was “France” and the model gave it too little probability. That one error flows backward and divides: part travels to W_Q/W_K — “point attention harder at paris” — and part travels to W_V — “make paris’s value carry more France-ness.” Same loss, different destinations → different learned numbers. Nobody ever supervises Q, K, or V directly; their roles emerge from this one objective plus the architecture's built-in asymmetry.
04The residual stream
One running vector — and why the matrices can spin
The residual stream is the thing you've been watching all along — it's what we kept calling in → in′ → in″. Each token gets exactly one 768-dim vector that flows from the embedding all the way to the top. Every sublayer reads the stream, computes an update, and adds it back. Nothing ever replaces the vector — it only accumulates. ("Residual" because each sublayer contributes a residue on top of what's already there; "stream" because it flows through all 12 blocks.)
one token's residual stream, top to bottom — sublayers branch off to read, and merge back to add
Why this matters for "where concepts live": the stream is the model's shared workspace — every head and every MLP neuron communicates with every later one only by writing into this vector. A "concept" is a direction in this 768-dim space (you can read the stream at any height by de-embedding it — a trick known as the logit lens). It is usually a diagonal direction across many coordinates — not one slot, and not one matrix column.
Now the spin. Attention scores only ever use Q · K — which means the model only ever sees the productW_Q · W_Kᵀ. Like 20 = 4×5 = 10×2: the product is pinned, the factors aren't. Rotate the columns of W_QandW_K by the same rotation R, and the transpose makes the two spins meet and cancel — every individual column changes, the model's output doesn't budge by a single bit.
spin both matrices by the same R → every column different, output identical → a column can't "be" a concept
Meaningful unitsW_Q·W_Kᵀ — a head's who-attends-to-whom map. W_V·W_O — its what-gets-copied map (same spin trick applies to that pair too). Directions in the residual stream. MLP neurons — GELU acts on each coordinate separately, so that basis can't be spun without changing the output.
Not meaningfulIndividual columns or rows of W_Q, W_K, W_V. The head's 64 internal axes are private scratch coordinates — like the factors in 20 = 4×5 = 10×2, they can be traded around freely as long as the product stays fixed.
This is why interpretability tools show whole-head properties. For an attention head, they show the attention pattern (the QK map in action) and what the output pushes toward (the OV map in action) — never "column 17 of W_Q." For the MLP, they can show individual neurons and what each writes, because the MLP's basis is pinned by its nonlinearity.
05Base model → assistant
The full training pipeline, one stage at a time
Everything so far — the whole block, all 12 layers — describes a base model, the product of pretraining. That's a raw next-word predictor: it knows the world but only autocompletes. Turning it into a helpful assistant (ChatGPT, Claude) takes a few more stages. Step through them — and watch the same “What is the capital of France?” prompt get answered better as the model learns not just facts but behavior.
Stage 1 / 5
Stage
—
stage 1Pretrain
→
stage 2SFT
→
stage 3Reward
→
stage 4RLHF
→
resultAssistant
🔒 the same model weights carry through every stage — each one just keeps nudging the same K/Q/V/MLP numbers. Only the DATA and the SIGNAL change.
Data in~10 billion → trillions of tokens of raw internet text (books, web, code)
Signalnext-token cross-entropy — the single loss from §03, applied across all of it at once
Gainsgrammar, facts, reasoning patterns — the knowledge. But its behavior is pure autocomplete.
Compute~99% of the entire training budget is spent right here
“What is the capital of France?”
base model · just continues the pattern“What is the capital of Germany? What is the capital of Spain? What is the largest…”
This is exactly where GPT-2 stops — a pure base model, no post-training.
Data inthousands → millions of human-written (instruction → ideal answer) pairs
Signalthe same next-token loss — nothing new, just run on this curated data instead of the web
Gainsthe assistant format: see a question → give an answer. Adds behavior, barely any new knowledge.
“What is the capital of France?”
after SFT · answers the question“The capital of France is Paris.”
Same weights as stage 1, gently nudged. Sometimes called instruction tuning.
Data inhumans rank pairs of answers (“A is better than B”) — tens of thousands to millions
Signaltrain a separate model to predict which answer humans prefer → a scalar reward score
Gainsan automatic judge of answer quality — a tool for stage 4, not the assistant itself
“Explain gravity to a 6-year-old.”
answer A · humans prefer ✓ (reward: high)“Gravity is what pulls things down — it’s why a ball drops when you let go.”
answer B · dispreferred ✗ (reward: low)“Gravity is the curvature of spacetime described by the Einstein field equations Gμν = 8πTμν…”
The reward model learns human taste so it can grade millions of answers without a human in the loop.
Data inthe model’s own generated answers, scored by the reward model (or preferences directly)
Signalreinforcement / preference loss — shift the weights toward higher-scoring answers
Gainshelpful, honest, harmless: tone, hedging, formatting, and sensible refusals
“Write an insult for my coworker.”
aligned assistant · helpful + harmless“I’d rather not write an insult — but if there’s a real problem, I can help you raise it as clear, firm feedback.”
Stages 3–4 together are alignment / preference tuning. Newer methods like DPO skip the separate reward model and learn straight from the preference pairs.
Architectureidentical to stage 1 — same 12 blocks, same K/Q/V tensors you visualized above
Knowledgefrom pretraining (stage 1) — the ~99% of compute
Behaviorfrom SFT + RLHF (stages 2–4) — a comparatively thin layer of compute on top
finished assistant“The capital of France is Paris — its largest city and political center, on the Seine in the north. Want anything to do or see there?”
Nothing about the machinery changed — only what the same weights were trained on, and with what signal.
The one-line version
Pretraining installs the knowledge (almost all the compute); SFT and RLHF install the behavior (a thin layer on top). Same architecture, same tensors, start to finish — only the data and the loss change at each stage.
Where each loss comes from. Stages 1–2 both use the next-token cross-entropy you already understand from §03 — the only difference is which text they train on. Only at stage 4 does the objective itself change, from “predict the next token” to “produce the answer humans score highest.”