The math behind the explainer

Vectors are just lists of numbers with rules.

In the LLM page, every token becomes a vector. The model adds vectors, scales them, compares them with dot products, and multiplies them by matrices. This page shows those operations with small numbers.

AddCombine matching components. MultiplyScale or gate components. DotTurn two vectors into one score. MatrixTurn one vector into another.
01 Vector

A vector is an ordered list

A vector is a row of numbers where position matters. In 2D, you can draw it as an arrow. In an LLM, a vector may have thousands of dimensions, so we usually draw it as a row of cells.

same vector, two views
[2, 1] 1 2 1
as cells
2
1
= two coordinates: right 2, up 1

For language models, the numbers are not screen coordinates. They are learned features. A 4D toy token vector might look like this:

0.21
-0.13
0.84
0.30
02 Addition

Add matching positions

Vector addition is component-by-component. The first number adds to the first number, the second to the second, and so on. The vectors must have the same length.

[2, 1] + [1, 3] = [3, 4]
2
1
+
1
3
=
3
4
1first component: 2 + 1 = 3
2second component: 1 + 3 = 4
[2,1] [1,3] sum [3,4]

In the LLM explainer, positional vectors are added to token embeddings this way: embedding + position = input vector.

03 Multiplication

Multiplication can mean a few things

When people say "multiply vectors," check which operation they mean. The two common simple versions are scaling by one number and element-wise multiplication.

A. scalar multiplication stretches or shrinks the whole vector
2 ×
2
1
=
4
2

Every component gets multiplied by the same number: 2 x 2 = 4, and 2 x 1 = 2. In attention, a weight like 0.57 scales a whole Value vector.

B. element-wise multiplication multiplies matching positions
2
1
3
⊙
4
-2
0.5
=
8
-2
1.5

Element-wise multiplication is often used for masks and gates: each position can be kept, weakened, flipped, or zeroed independently.

04 Dot product

A dot product turns two vectors into one score

The dot product is "multiply matching positions, then add the results." It is the key comparison operation in attention: a Query dot Key gives a relatedness score.

[2, 1] dot [1, 3] = 5
2
1
·
1
3
= (2×1) + (1×3) =
5
1multiply matching positions: 2 x 1 = 2, and 1 x 3 = 3
2add the products: 2 + 3 = 5
a = [2,1] b = [1,3] angle matters
how to read the score
same direction
large +
sideways
near 0
opposite
negative

A bigger positive dot product can mean the vectors point in similar directions, or that one vector is simply much longer. A score near zero means they do not line up much. A negative score means they point against each other.

raw dot product includes length, not just direction
[2, 1] dot [2, 1] = 5 Same direction, same length. As a direction match, this is perfect.
[2, 1] dot [100, 0] = 200 Higher raw score because [100, 0] is huge, even though it does not point in exactly the same direction.
[2,1] same direction [100,0] scaled down to fit

Both of these are correct. Raw dot product is:

A[2,1] dot [2,1] = 2×2 + 1×1 = 5
B[2,1] dot [100,0] = 2×100 + 1×0 = 200
Fix if you only care about direction: use cosine similarity, which divides by both vector lengths. Then [2,1] vs [2,1] scores 1.00, while [2,1] vs [100,0] scores about 0.89. Standard transformer attention uses scaled dot products, so vector length can still affect attention unless the architecture normalizes Q and K.
05 Vector x matrix

A matrix turns one vector into a new vector

A matrix is a grid of numbers. When a vector multiplies a matrix, each output number is a dot product between the input vector and one column of the matrix.

[2, 1] x matrix = [2, 3, 8]
2
1
×
1
0
3
0
3
2
=
2
3
8
1column 1: [2,1] dot [1,0] = 2
2column 2: [2,1] dot [0,3] = 3
3column 3: [2,1] dot [3,2] = 8
input [2,1] col 1 [1,0] col 2 [0,3] col 3 [3,2]

Vector diagram view: the matrix is three column vectors. The input vector is dotted with each column.

c1[2,1] lines up with [1,0] enough to score 2
c2[2,1] lines up with [0,3] enough to score 3
c3[2,1] lines up strongly with [3,2], so it scores 8
What happens to the original 2D direction? Because this matrix has three columns, the output has three numbers. It is no longer a 2D arrow on this same plane; it is a 3D feature vector made of three scores. If the matrix were 2 x 2 instead, the output would stay 2D and you could draw it as a new arrow: rotated, stretched, squashed, or sheared by the matrix.

This is why learned matrices are so central in LLMs. A matrix is a bank of learned recipes. Each column asks one question about the input vector, and the answers become the next vector.

06 LLM connection

Where these operations appear in the explainer

The LLM page uses these same operations repeatedly. The real vectors are wider and the matrices are huge, but the mechanics are the same.

operation map
Addition Token embedding + position vector gives the input to the network.
Dot product Query dot Key creates an attention score for each token pair.
Scaling + sum Attention weights scale Value vectors, then the rows add into one context vector.
Vector x matrix Learned matrices make Q/K/V: x × WQ = Q, x × WK = K, and x × WV = V.