The other half of every layer

Attention mixes tokens. The MLP thinks about each one.

You already have attention: Query · Key · Value lets tokens look at each other and pull in context. But that is only half of every layer. The other half — the MLP — takes each contextualized token on its own and runs it through a little look-up engine. It is where the model does most of its computing, and where facts like "France → Paris" are stored.

Two halvesAttention talks, the MLP thinks. Up · squish · downTwo matrices and a bend. The squishThe nonlinearity is the point. MemoryEach neuron is an if-then rule.

01Two halves

Every layer does two different jobs

A transformer block is two sublayers stacked back to back. Attention moves information sideways — between token positions — so each token can gather what it needs from the others. The MLP (multi-layer perceptron, also called the feed-forward layer) then works straight down on each token separately: it never looks at the other tokens, it just processes the vector attention just handed it.

Both write their result back into the same running vector — the residual stream — by adding to it. So one block is really: x = x + attention(x), then x = x + mlp(x).

one block = attention (mix across tokens) → MLP (process each token)

ATTENTION — communication Looks at other tokens. Mixes information across positions. What flows in is dynamic: the Keys and Values come from whatever tokens are in the sentence. "Gather the context I need."

MLP — computation Looks at one token's vector. Never sees the others. What it knows is fixed: the same learned weights run on every position. "Now think about / look up what that context means."

A clean way to hold it: attention looks things up in the context; the MLP looks things up in its memory.

follow one token ("in") down through a block — one running vector, two updates added on

1Does the MLP take in what attention did? Yes — its input is in′, which is in + attn_out. It reads the result attention wrote — but only that one vector. It never sees Q/K/V or the other tokens.

2Where does its output go? Added straight back onto the vector: in′ + mlp_out = in″. That in″ is what leaves Layer 1 and enters Layer 2.

There's really just one running vector for "in" (the residual stream). Attention and the MLP each read it, compute an update, and add the update back — that's the x = x + attention(x), then x = x + mlp(x) from above. (paris and is ride the identical path in parallel.)

02Up · squish · down

An MLP is two matrices with a bend in between

Despite the fancy name, an MLP is the oldest, simplest neural network. It does exactly three things to a token's vector: expand it to a much wider list of numbers, bend each number through a nonlinearity, then shrink it back to the original width. In GPT-2 the vector is 768 numbers wide; the MLP blows it up to 3072 and back.

the three steps (GPT-2 small sizes)

768

token vector in

→× W_up
(c_fc)

3072

neurons

→GELU
squish

3072

activations

→× W_down
(c_proj)

768

add to residual

1Up-project (a vector × matrix, exactly like the matrix step on the Vectors page). 768 numbers become 3072. Think of it as asking 3072 questions about the token at once — one per neuron.

2Squish each of the 3072 numbers through GELU. This is the only nonlinear step, and it is what makes the MLP more than just another matrix (next section).

3Down-project back to 768 and add the result into the residual stream — the same stream attention writes to.

"Multi-layer perceptron" just means "a couple of fully-connected layers with a nonlinearity between them." The transformer drops one of these after every attention sublayer.

03The squish

The bend in the middle is the whole point

Why bother with the nonlinearity? Because without it, the two matrices would collapse into one. (x × W_up) × W_down is just x × (one combined matrix) — no more powerful than a single linear step. The GELU bend in the middle breaks that collapse and lets each neuron act like a switch: it stays near zero until its input is positive enough, then it turns on.

GELU: off for negative inputs, on (≈ pass-through) for positive ones

Read it like a gate:

−Strongly negative input → output ≈ 0. The neuron stays silent.

0Around zero → a soft, gentle turn-on (GELU is smooth, not a hard step).

+Positive input → output ≈ the input. The neuron fires, roughly passing its score through.

So each of the 3072 neurons is independently deciding "does my pattern appear in this token? if yes, speak up."

This is the source of the model's expressive power. Stacking linear steps gets you nothing new; stacking linear → bend → linear, over and over, lets the network approximate essentially any function.

04Memory

Each neuron is a tiny "if you see X, add Y" memory

Here is the bridge from attention. A single neuron does almost the same dance you already know — a dot product to get a score, then it scales a vector — except its Key and Value are learned weights baked into the model, not pulled from other tokens.

one neuron, on a toy 4-number token (the real ones are 768 wide)

the tokenafter attention

0.9

0.1

0.8

0.2

"capital-of-France-ish" context

step 1 · KEYwhat this neuron looks for (W_up row)

· token · key = 0.9 + 0.8 = 1.7 (a high score → pattern present)

step 2 · GELUthe gate

GELU(1.7) ≈ 1.62 → this neuron fires.

step 3 · VALUEwhat it writes (W_down column)

1.62 ×

1.5

2.4

added to the residual

A neuron whose KEY matched a different pattern (say [0,1,0,1]) would score only 0.1 + 0.2 = 0.3, barely fire, and write almost nothing. Out of 3072 neurons, the handful whose key matches this token are the ones that speak.

The whole MLP output is just the sum of every neuron's contribution: Σ GELU(token · key_i) × value_i. It behaves like a giant book of rules — "if the context looks like the capital of France, add a nudge toward Paris." That is literally where factual knowledge is stored: in the key/value weight pairs, learned during training.

which neurons fire for "...capital of France is" (illustrative)

#1828 → Paris

fires

#2649 → Europe

fires

#771 → plural noun

quiet

#3030 → past tense

quiet

Reality check. In a real model, individual neurons are polysemantic — most respond to a messy blend of unrelated things rather than one clean concept (a problem called superposition). The clean "Paris neuron" above is a teaching idealization; the aggregate behavior of the whole MLP is far more interpretable than any single neuron.

05Scale

Most of the model, by weight, is MLP

Because each MLP holds two big matrices (768×3072 and 3072×768), the feed-forward layers contain about two-thirds of GPT-2's parameters — more than attention. When people say large models "store knowledge," they mostly mean these matrices. Attention is the smaller, flashier part that routes information; the MLPs are the bulk memory.

parameters per layer, GPT-2 small

MLP

~4.7M

Attention

~2.4M

Across all 12 layers that is roughly 57M parameters in MLPs versus 28M in attention — and the gap only widens in bigger models.

06Putting it together

How a token gets answered

Back on the main explainer, "pass the contextualized tokens through the layer stack" is this two-step dance repeated 12 times:

Attention gathersThe "is" token pulls in "France" and "capital" — assembling what we're talking about.

MLP recallsFed that context, MLP neurons fire and write facts toward it — capital-of-France → Paris.

Repeat & read outEach layer refines the running vector; the final one is de-embedded into the next word.

So the one-line answer to "what is an MLP?": it is the per-token look-up engine that turns the context attention gathered into stored knowledge and computation — two matrices, a bend, and 3072 little if-then memories, run on every token at every layer.