How an LLM turns your words into its next word.
Follow one tiny prompt through the full transformer path—from text, to numbers, to context, to a generated answer.
See the whole loop in six clicks
Use this as a quick explanation, then open any full section below when someone wants the “how.”
Cut the sentence into familiar chunks
The model starts by splitting text into tokens. Each token gets a vocabulary ID the model can work with.
Turn each token into a list of numbers
An embedding gives every token a numeric representation. Learned patterns in those numbers place related ideas in useful directions.
Add the order of the words
The same words in a different order can mean something else. Position information tells the model where every token sits.
Let each token look at the useful context
Attention scores which earlier tokens matter right now, then blends their information into a context-aware vector.
Refine that context, layer after layer
Each transformer layer repeats attention and an MLP. The running token representations become more useful for the final prediction.
Score the next token, choose one, repeat
The final vector produces a probability for every vocabulary token. One is selected, appended, and the loop runs again.
Split the text into tokens
The prompt is first split into tokens: common chunks of text. Each token maps to an integer ID in the model's vocabulary. From here on, the model works with those IDs, not the raw letters.
Look up a vector for each token
Each ID points to a row in a learned embedding table. That row is a vector: the token represented as numbers. Real models use thousands of numbers per token; here we show 4 so the math fits on the page.
New to vectors, dot products, and matrices? How vectors work covers the math on this page with small numbers.
Add where each token sits
Embeddings know what a token is, but not where it appears. Without position, "where is italy" and "italy is where" would have the same three vectors. Adding a positional vector gives each token an order slot.
why use a position vector instead of the number 2?
Let tokens give each other context
Attention is the transformer's main move. Each token is projected into three vectors: a Query, a Key, and a Value. A Query asks what matters, Keys are matched against it, and Values are the information that gets blended in. Below we follow the last token, italy.
In a real model, each token's vector is first multiplied by three learned matrices — WQ, WK, WV — giving three views of the same token. To keep every number on this page checkable, this toy skips that step and uses each token's vector directly as its own Q, K and V. The projection step is shown move-by-move in How attention works.
3.64
5.17
1.82
2.58
0.27
0.57
0.20, 1.60]
Kᵀ means the Keys are flipped into columns. That lets one matrix multiply compare every Query with every Key. We only need italy's row for the next-token prediction.
1.65·0.21 + 0.21·0.87 + 0.01·0.84 + 1.55·1.30 = 2.55 → repeat per column to fill italy's score row.
do the math — the italy↔italy score & softmax
Now pull the content. The weights say how much to use from each token. Multiply each Value by its weight, then add the rows. The result is italy's new context vector.
do the math — the weighted sum, dimension by dimension
↑ that's one contextualized vector — the output of attention for one head, in one layer.
Want to watch a full block run — all 12 heads, the MLP, the residual adds — one move at a time? How attention works.
Pass the contextualized tokens through the layer stack
Step 4 gives each token a context-aware vector. A real transformer does not stop there: each layer runs many attention heads, mixes their outputs, sends each token through an MLP (also called the FFN, feed-forward network), then passes the result to the next layer. The same pattern repeats until the top of the stack.
+ mix WO
per token
Real models run this in many heads and many layers. Each head has its own Q/K/V matrices and can learn a different relationship; concatenation puts the head outputs side by side, and WO mixes them so later layers can combine what different heads found. The MLP then transforms each token's vector independently, and a residual add — the update is added onto the token's running vector rather than replacing it — folds the result back in before the next layer treats it as its new input. Numbers are illustrative.
Turn the vector back into a word
After the layer stack, each token has a fully contextualized vector. To generate the next token, the model uses the vector at the last position: here, the vector for italy.
During training, every row predicts its own next token. During generation, the current last row is the one used.
That last vector is multiplied by the output matrix to score every possible next token. Those raw scores are logits. A final softmax turns them into probabilities, the model chooses a token, appends it, and repeats.
Follow-up questions reuse cached Keys and Values
An LLM does not keep human-style memory between turns. The app sends the conversation so far again as one long token stream. To avoid recomputing all of it, the system keeps the Keys and Values from earlier tokens. New tokens make fresh Queries that attend back into that cache.
Adding a token adds a row (its Query) that reaches across all the cached columns (Keys/Values). The old Query rows already ran and handed off their output — nothing ever reads them again — so only the K/V columns are worth keeping. (europe also drops its own K/V in as a new column for whatever comes next.)
↑ "europe" (pos 9) can attend straight to the "Europe" the model itself wrote at pos 6 — that's how turn 2 "knows" turn 1.
what's actually stored vs recomputed
What newer architectures try to change
Everything above is the standard transformer path used by models such as GPT, Llama and Claude. New architecture work usually targets the two expensive parts: step 4, where attention compares tokens to other tokens, and step 6, where generation normally happens one token at a time.
Mamba & SSMs
Instead of building the all-pairs attention matrix from step 4, a state-space model scans left to right and updates a fixed-size memory at each token. The model decides what to keep and what to forget as it moves.
Changes: attention's O(n²) cost becomes closer to linear work with length.
Diffusion LLMs
Instead of emitting one token at a time, a diffusion language model starts with a block of masked placeholders and refines many positions in parallel over several passes.
Changes: generation can fill a block in a fixed number of refinement passes instead of one full pass per word.
Titans & neural memory
Attention is a precise short-term lookup over whatever tokens are still in the window. Titans (arXiv:2501.00663) adds a small network that compresses older context into its own weights as the sequence runs — surprising tokens update it more — and the model queries that network for information from beyond the window.
Changes: context isn't limited to the attention window — and unlike the KV cache (which just stores exact past K/V to skip recomputation), this memory learns and compresses.