Chapter 40: Sequence Models & Positional Encoding Mathematics

Chapter 40: Sequence Models & Positional Encoding Mathematics

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 40 of 48

Attention has no idea what order its inputs arrived in. That is not a bug in the implementation — it is a theorem about the operation, and without an injected position signal a transformer cannot tell “dog bites man” from “man bites dog”. This chapter proves the permutation-equivariance of attention, then derives the sinusoidal encoding and RoPE from the rotation identity that makes relative position work.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Why position is needed Asserted, never shown. Permutation equivariance demonstrated numerically.
Sinusoidal encoding Presented as a magic formula. Shown to be a rotation: PE(p+k) = R(k) PE(p).
RoPE Called “better sinusoidal”. Relative-distance property derived and verified.
Extrapolation Claimed for every scheme. Separated into definition-domain and empirical claims.

Core learning objectives. After this chapter you should be able to:

  1. Prove that attention is permutation equivariant.
  2. Construct the sinusoidal encoding and state its frequency ladder.
  3. Derive the rotation identity PE(p+k) = R(k)·PE(p).
  4. Explain why RoPE makes attention depend on relative position.
  5. Describe additive versus multiplicative position schemes.
  6. State honestly what each scheme does and does not extrapolate.

2. Attention Cannot See Order

Two sentences with reversed word order feeding the same set-based attention sum
Figure 40.1: The attention sum is over a set, so the row order is irrelevant.
Let P be a permutation matrix acting on the sequence axis.

attn(PX) = softmax(PQ K^T P^T / sqrt(d)) (P V)
         = P attn(X)

The permutation passes straight through. The operation is
EQUIVARIANT, not invariant: permute the input and the
output is permuted the same way, with the same values.

So two sentences containing the same tokens in different
orders are indistinguishable to the model unless something
breaks that symmetry.
import numpy as np

rng = np.random.default_rng(0)

def attn(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    S = Q @ K.T / np.sqrt(Q.shape[1])
    S = S - S.max(axis=1, keepdims=True)
    A = np.exp(S); A /= A.sum(axis=1, keepdims=True)
    return A @ V

d, n = 4, 5
X = rng.normal(size=(n, d))
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))

base = attn(X, Wq, Wk, Wv)
perm = np.array([2, 0, 4, 1, 3])
Xp = X[perm]

got = attn(Xp, Wq, Wk, Wv)
want = base[perm]
print("attn(PX) == P attn(X):", np.allclose(got, want))
print("max abs difference  :", f"{np.abs(got-want).max():.2e}")
print()
print("shuffling tokens shuffles the output rows and nothing else.")
print("Row 0 of the output for order A:", np.round(base[0], 4))
print("Row for the same token in order B:", np.round(got[list(perm).index(0)], 4))

The difference is exactly zero, not small. No amount of training can teach a pure attention block to distinguish orderings, because the two computations are literally the same matrix operations.

3. The Sinusoidal Encoding

Four sine waves at geometrically decreasing frequencies plotted against position
Figure 40.2: A geometric frequency ladder gives fine and coarse resolution at once.
PE(p, 2i)   = sin( p / 10000^(2i/d) )
PE(p, 2i+1) = cos( p / 10000^(2i/d) )

Low i  -> high frequency -> distinguishes adjacent positions
High i -> low frequency  -> distinguishes distant blocks

Wavelengths form a geometric series from 2*pi to 10000*2*pi,
so the encoding resolves position at every scale.
import numpy as np

def pe(n, d):
    pos = np.arange(n)[:, None]
    i = np.arange(d)[None, :]
    angle = pos / (10000 ** (2 * (i // 2) / d))
    out = np.zeros((n, d))
    out[:, 0::2] = np.sin(angle[:, 0::2])
    out[:, 1::2] = np.cos(angle[:, 1::2])
    return out

P = pe(50, 16)
print("shape:", P.shape)
print("row 0 [0:6]:", np.round(P[0, :6], 4))
print("row 1 [0:6]:", np.round(P[1, :6], 4))

# every position is unique
dists = np.linalg.norm(P[:, None, :] - P[None, :, :], axis=-1)
off = dists + np.eye(50) * 1e9
print("minimum distance between any two encodings:", f"{off.min():.6f}")

# nearby positions are near; distant ones need not be
for k in (1, 2, 4, 8, 16, 32):
    print(f"gap {k:3d}: distance {np.linalg.norm(P[0] - P[k]):.4f}")

# every row has the same norm, by the sin^2+cos^2 identity
norms = np.linalg.norm(P, axis=1)
print("all row norms equal:", np.allclose(norms, norms[0]), "norm^2 =", round(norms[0] ** 2, 6))

Each pair of coordinates is a point on the unit circle, so every row has squared norm d/2 exactly. The encoding injects position without changing the overall scale of the token vector — which matters, because a large positional offset would drown the token embedding.

One property people often assume and should not: the distance between two encodings is not monotone in the gap. It rises steadily for small gaps, then saturates and fluctuates around gap 16–32, because the high-frequency components wrap around and become uncorrelated while the low-frequency ones are still separating. If that were not so, a linear model could read position off as a distance; instead, position has to be recovered from the pattern across all frequencies, which is exactly what the rotation identity below allows.

4. The Rotation Identity

A heatmap of the encoding matrix over position and dimension, fast on the left and slow on the right
Figure 40.3: No two rows repeat, so position is fully recoverable from the encoding.
For one frequency w, the pair (sin wp, cos wp) satisfies

  [sin(w(p+k))]   [cos wk  sin wk] [sin wp]
  [cos(w(p+k))] = [-sin wk  cos wk] [cos wp]

That is a rotation by wk. So

  PE(p + k) = R(k) PE(p)

with R(k) block-diagonal, one 2x2 rotation per frequency.
The offset k is encoded as a rotation, not as an additive
shift — and rotations compose additively in angle, which
is exactly what we want from positions.
import numpy as np

def pe_pair(p, w):
    return np.array([np.sin(w * p), np.cos(w * p)])

def rot(theta):
    return np.array([[np.cos(theta), np.sin(theta)],
                     [-np.sin(theta), np.cos(theta)]])

w = 0.35
for p, k in [(0, 3), (7, 5), (11, 100)]:
    lhs = pe_pair(p + k, w)
    rhs = rot(w * k) @ pe_pair(p, w)
    print(f"p={p:3d} k={k:3d}  |PE(p+k) - R(k)PE(p)| = {np.linalg.norm(lhs-rhs):.2e}")

print()
print("rotations compose:", np.allclose(rot(w*2) @ rot(w*3), rot(w*5)))

# the full 16-dim encoding inherits the property blockwise
def pe_row(p, d=16):
    i = np.arange(d)
    a = p / (10000 ** (2 * (i // 2) / d))
    out = np.zeros(d)
    out[0::2] = np.sin(a[0::2]); out[1::2] = np.cos(a[1::2])
    return out

def rot_full(k, d=16):
    R = np.eye(d)
    for j in range(d // 2):
        wj = 1.0 / (10000 ** (2 * j / d))
        R[2*j:2*j+2, 2*j:2*j+2] = rot(wj * k)
    return R

print("full encoding:", np.allclose(pe_row(9 + 4), rot_full(4) @ pe_row(9), atol=1e-12))

This is the property that makes the sinusoidal scheme more than a lookup table: a fixed relative shift corresponds to a fixed linear map, independent of where you started.

5. RoPE: Rotating the Queries and Keys

Three vectors rotated by increasing angles on a circle, one per position
Figure 40.4: RoPE rotates q and k by their own position angle; only the difference survives.
Sinusoidal ADDS the encoding to the token.
RoPE ROTATES q and k instead, and never touches v.

q_m -> R(m theta) q_m
k_n -> R(n theta) k_n

q_m . k_n = q_m^T R(m theta)^T R(n theta) k_n
          = q_m^T R((n - m) theta) k_n

Only the difference appears. The attention score depends
on relative position and the token content — the absolute
position has cancelled out.
import numpy as np

def rot2(theta):
    return np.array([[np.cos(theta), -np.sin(theta)],
                     [np.sin(theta),  np.cos(theta)]])

theta = 0.4
rng = np.random.default_rng(1)
q = rng.normal(size=2)
k = rng.normal(size=2)

print(f"{'m':>3}{'n':>4}{'m-n':>5}   score")
for m, n in [(0, 0), (5, 5), (7, 3), (12, 8), (20, 16), (3, 7), (100, 96)]:
    score = (rot2(m * theta) @ q) @ (rot2(n * theta) @ k)
    print(f"{m:3d}{n:4d}{m-n:5d}   {score:+.8f}")

# all pairs with the same gap give the same score
s1 = (rot2(7 * theta) @ q) @ (rot2(3 * theta) @ k)
s2 = (rot2(102 * theta) @ q) @ (rot2(98 * theta) @ k)
print("\ngap +4 at positions (7,3) and (102,98):", np.isclose(s1, s2))

# and rotation preserves the vector norm
print("norm preserved:", np.isclose(np.linalg.norm(rot2(3.7) @ q), np.linalg.norm(q)))

The score for gap +4 is identical at positions (7,3) and (102,98). This is the entire justification for RoPE: relative position falls out of the algebra rather than being approximated, and the vector length is untouched so attention logits do not drift with position.

6. Comparing the Schemes

A table comparing additive, rotary and bias-based positional schemes
Figure 40.5: Two families — those that modify vectors, and those that modify scores.
Question Answer
Does sinusoidal extrapolate to longer sequences? The formula is defined for any length, but accuracy well beyond the training range degrades — the claim is about the definition, not about performance.
Does a learned table extrapolate? No. There are no rows beyond the trained maximum.
Can RoPE be combined with length extension? Yes — scaling the rotation base or the angle is the standard trick, and it needs fine-tuning to work.
Why do most LLMs use RoPE? It encodes relative position exactly, adds no parameters, and leaves the value stream unchanged.

7. Worked Examples

7.1 Shape

PE is an (n, d) matrix added to an (n, d) token tensor.
One row per position, no parameters, no learning.

7.2 Why d must be even

Each frequency occupies a sin/cos pair, so d/2 distinct
frequencies are available. An odd d leaves a half pair.

7.3 Choosing the base

10000 is the original default. A larger base spreads the
frequencies, which is the usual move when extending context.

7.4 ALiBi

Add -slope * (m - n) to the attention scores before softmax.
No parameters, no vectors touched, strong local bias.

7.5 What position does not fix

Position identifies WHERE a token is, not what it means.
A model still needs content features to use the position.

8. Verify It in Python

import numpy as np

# --- 1. attention is permutation equivariant ------------------
rng = np.random.default_rng(0)
def attn(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    S = Q @ K.T / np.sqrt(Q.shape[1])
    S -= S.max(axis=1, keepdims=True)
    A = np.exp(S); A /= A.sum(axis=1, keepdims=True)
    return A @ V

n, d = 6, 4
X = rng.normal(size=(n, d))
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))
perm = np.array([3, 0, 5, 1, 4, 2])
base = attn(X, Wq, Wk, Wv)
assert np.allclose(attn(X[perm], Wq, Wk, Wv), base[perm], atol=1e-14)
print("attn(PX) == P attn(X) — permutation equivariance holds")

# --- 2. sinusoidal encoding properties ------------------------
def pe(n, d):
    pos = np.arange(n)[:, None]; i = np.arange(d)[None, :]
    a = pos / (10000 ** (2 * (i // 2) / d))
    out = np.zeros((n, d))
    out[:, 0::2] = np.sin(a[:, 0::2]); out[:, 1::2] = np.cos(a[:, 1::2])
    return out

P = pe(64, 32)
assert P.shape == (64, 32)
norms = np.linalg.norm(P, axis=1)
assert np.allclose(norms, np.sqrt(32 / 2))
print(f"every row has norm sqrt(d/2) = {norms[0]:.6f}")

# --- 3. no two positions collide ------------------------------
D = np.linalg.norm(P[:, None, :] - P[None, :, :], axis=-1) + np.eye(64) * 1e9
assert D.min() > 1e-3
print(f"closest two encodings differ by {D.min():.6f}")

# --- 4. distance grows with gap, then saturates ---------------
# It does NOT grow forever: the fast components wrap around and
# decorrelate while the slow ones are still separating.
gaps = [np.linalg.norm(P[0] - P[k]) for k in (1, 2, 3, 4, 8, 16)]
assert all(a < b for a, b in zip(gaps, gaps[1:]))
far = np.linalg.norm(P[0] - P[32])
print("distance grows with gap:", [round(g, 3) for g in gaps])
print(f"then saturates: gap 32 gives {far:.3f}, less than gap 16's {gaps[-1]:.3f}")
assert far < gaps[-1]

# --- 5. rotation identity -------------------------------------
def rot(t):
    return np.array([[np.cos(t), -np.sin(t)], [np.sin(t), np.cos(t)]])

def pair(p, w):
    return np.array([np.sin(w * p), np.cos(w * p)])

for p, k in [(0, 3), (7, 5), (11, 100)]:
    lhs = pair(p + k, 0.35)
    rhs = np.array([[np.cos(0.35*k), np.sin(0.35*k)],
                    [-np.sin(0.35*k), np.cos(0.35*k)]]) @ pair(p, 0.35)
    assert np.allclose(lhs, rhs, atol=1e-12)
print("PE(p+k) == R(k) PE(p) verified")

# --- 6. full encoding inherits it blockwise -------------------
def pe_row(p, d=32):
    i = np.arange(d); a = p / (10000 ** (2 * (i // 2) / d))
    out = np.zeros(d)
    out[0::2] = np.sin(a[0::2]); out[1::2] = np.cos(a[1::2])
    return out

def rot_pair(t):
    """Rotation acting on a (sin, cos) ordered pair — note the sign layout."""
    return np.array([[np.cos(t), np.sin(t)], [-np.sin(t), np.cos(t)]])

def rot_full(k, d=32):
    R = np.eye(d)
    for j in range(d // 2):
        wj = 1.0 / (10000 ** (2 * j / d))
        R[2*j:2*j+2, 2*j:2*j+2] = rot_pair(wj * k)
    return R

assert np.allclose(pe_row(9 + 4), rot_full(4) @ pe_row(9), atol=1e-12)
print("full 32-dim encoding satisfies PE(p+k) = R(k)PE(p)")

# --- 7. RoPE depends only on the gap --------------------------
theta = 0.4
q = rng.normal(size=2); k = rng.normal(size=2)
gap = lambda m, n: (rot(m * theta) @ q) @ (rot(n * theta) @ k)
assert np.isclose(gap(7, 3), gap(102, 98))
assert np.isclose(gap(5, 5), q @ k)          # gap 0 is the unrotated dot product
assert not np.isclose(gap(7, 3), gap(3, 7))
print("RoPE score depends on m-n only, and is direction-sensitive")

# --- 8. rotation preserves magnitude --------------------------
for t in (0.1, 1.0, 3.7, 12.0):
    assert np.isclose(np.linalg.norm(rot(t) @ q), np.linalg.norm(q))
print("rotation preserves the vector norm")
print("all assertions passed")

9. Practice Questions

  1. Prove that attention is permutation equivariant.
  2. Why is equivariance not the same as invariance, and why does it matter here?
  3. Write the sinusoidal encoding for position 3 in 8 dimensions.
  4. Why does every row of the encoding have the same norm?
  5. Show that PE(p+k) = R(k)·PE(p) for a single frequency.
  6. Derive the RoPE score decomposition.
  7. Why does RoPE leave the value vectors alone?
  8. What does ALiBi add, and to what?
  9. Which schemes can be defined beyond the training length?
  10. Why does an odd embedding dimension break the encoding?

Leave a Reply

Your email address will not be published. Required fields are marked *