Chapter 42: Attention QKV Mathematics: Queries, Keys, Values & Multi-Head Projections

Chapter 42: Attention QKV Mathematics: Queries, Keys, Values & Multi-Head Projections

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 42 of 48

Multi-head attention is not “more attention” — it is attention run in parallel subspaces and then recombined. This chapter derives the parameter accounting that makes heads free, shows why the output projection W_O is the only place heads can interact, explains the KV cache as a memory-for-compute trade, and quantifies exactly how the quadratic term dominates at long context.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Head splitting Described as “multiple attentions”. Shown to be a reshape of one channel dimension.
Cost “Heads are expensive” — false. Parameter count proved identical to one wide head.
W_O Treated as an afterthought. Identified as the sole cross-head mixing point.
KV cache Called a speed optimisation. Derived as a time-to-memory trade with explicit bytes.

Core learning objectives. After this chapter you should be able to:

  1. Describe the reshape that turns one attention into h heads.
  2. Prove that splitting into heads costs no extra parameters.
  3. Explain what concatenation loses and what W_O restores.
  4. Compute the KV cache size for a given model.
  5. Derive why attention is quadratic while projections are linear.
  6. State what FlashAttention changes and what it cannot.

2. One Attention, Sliced

A 512-wide input split into four 128-wide heads, concatenated and projected back to 512
Figure 42.1: Heads split the channel dimension; they do not multiply the work.
d_model = 512, h = 8 heads, d_head = 64

Instead of one 512 -> 512 projection per Q, K, V:

  Wq, Wk, Wv each (512, 512)
  reshape to (h, d_head) along the channel axis
  run h independent attentions of width 64
  concat back to 512
  multiply by W_O (512, 512)

Parameter count, per projection:
  one head  : 512 x 512            = 262,144
  h heads   : 8 x (512 x 64)       = 262,144

Identical. Splitting is a reshape, not an expansion.
import numpy as np

d_model, h = 512, 8
d_head = d_model // h

one_head = d_model * d_model
many_heads = h * (d_model * d_head)
print(f"one head : {one_head:,} parameters")
print(f"{h} heads  : {many_heads:,} parameters")
print("identical:", one_head == many_heads)

# the reshape is literally a view
rng = np.random.default_rng(0)
X = rng.normal(size=(4, d_model))          # 4 tokens
proj = rng.normal(size=(d_model, d_model))
Q = X @ proj
Qh = Q.reshape(4, h, d_head).transpose(1, 0, 2)
print("\nQ      :", Q.shape)
print("per head:", Qh.shape)
print("reshape preserved every value:", np.allclose(Qh.transpose(1, 0, 2).reshape(4, d_model), Q))

# each head attends in its own 64-dim subspace
scores = [Qh[i] @ Qh[i].T / np.sqrt(d_head) for i in range(h)]
print("per-head score matrices:", [s.shape for s in scores])

The reshape is a view, not a copy — no information is created or destroyed. What changes is that the eight attention problems are solved independently, which is the entire mechanism by which heads can specialise.

3. Concatenation and the Output Projection

Three head outputs concatenated into a 384-wide vector then projected to 512
Figure 42.2: Concatenation keeps heads in fixed slots; W_O is what lets them mix.
After the heads:  concat = [O_1 | O_2 | ... | O_h]   (n, d_model)

Without W_O, channel 0..63 always belongs to head 1, 64..127 to
head 2, and so on. The blocks never communicate.

output = concat W_O

W_O is a dense d_model x d_model matrix, so every output channel
is a learned mixture of every head. That single matmul is the only
cross-head interaction in the whole block.
import numpy as np

rng = np.random.default_rng(1)
n, d_model, h = 5, 512, 8
d_head = d_model // h

heads = [rng.normal(size=(n, d_head)) for _ in range(h)]
concat = np.concatenate(heads, axis=1)
print("concat shape:", concat.shape)

W_O = rng.normal(size=(d_model, d_model)) / np.sqrt(d_model)
out = concat @ W_O
print("output shape:", out.shape)

# demonstrate that heads are isolated without W_O
isolated = np.concatenate(heads, axis=1)
changed = [hh.copy() for hh in heads]
changed[0] = changed[0] * 0.0
print("\nchanging head 0 without W_O affects channels:",
      np.nonzero(np.any(np.concatenate(changed, axis=1) != isolated, axis=0))[0][[0, -1]])

mixed = np.concatenate(changed, axis=1) @ W_O
print("changing head 0 with W_O affects all",
      int(np.sum(np.any(mixed != out, axis=0))), "of", d_model, "channels")

Zeroing head 0 without W_O touches exactly 64 channels. With W_O it touches all 512, because each output channel draws on every head.

4. Do Heads Actually Specialise?

Three attention heatmaps showing a local diagonal pattern, a first-token pattern and a content-match pattern
Figure 42.3: Nothing assigns head roles — the patterns emerge from gradient descent.
Common patterns observed in trained models:
  - attend to the previous token
  - attend to the first token (a sink)
  - attend to a syntactically related token
  - broad, near-uniform averaging

Head diversity must be MEASURED, not assumed. If every head
produces the same attention map, the model is paying for h
attention computations and receiving one.
import numpy as np

def head_similarity(A1, A2):
    """1.0 means the two attention maps are identical."""
    a, b = A1.ravel(), A2.ravel()
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

rng = np.random.default_rng(2)
n = 6

def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

# a "collapsed" pair: identical by construction
A_dup = softmax(rng.normal(size=(n, n)))
B_dup = A_dup.copy()
print("collapsed heads similarity:", round(head_similarity(A_dup, B_dup), 6))

# a "diverse" pair: independent
B_ind = softmax(rng.normal(size=(n, n)))
print("independent heads similarity:", round(head_similarity(A_dup, B_ind), 4))

# a deliberately structured pair: local vs global
local = np.zeros((n, n))
for i in range(n):
    local[i, max(0, i - 1)] = 0.85
    local[i] = local[i] / local[i].sum()
glob = np.full((n, n), 1 / n)
print("local-vs-global similarity:", round(head_similarity(local, glob), 4))

The similarity score is the cosine between flattened attention maps. Collapsed heads score 1.0 by definition; genuinely different mechanisms score well below it. Measuring this is how you detect wasted heads.

5. The KV Cache

A table comparing cached and uncached generation cost and memory
Figure 42.4: Caching K and V turns quadratic generation into linear time.
Generating token t+1 needs attention over tokens 1..t.
Their keys and values do not change, so recomputing them is waste.

Without cache: O(n^2) key and value projections overall
With cache:    O(n) projections, O(n) memory

Cache bytes = 2 * n * d_model * layers * batch * bytes_per_elem

The 2 is for K and V. Q is never cached because it is used once.
import numpy as np

def kv_cache_gb(n, d_model, layers, batch=1, bytes_per=2):
    return 2 * n * d_model * layers * batch * bytes_per / 1e9

configs = [
    ("7B,  32 layers, d=4096", 4096, 32),
    ("13B, 40 layers, d=5120", 5120, 40),
    ("70B, 80 layers, d=8192", 8192, 80),
]
print(f"{'model':28}{'n=2k':>10}{'n=8k':>10}{'n=32k':>10}")
for name, d, L in configs:
    row = "".join(f"{kv_cache_gb(n, d, L):10.2f}" for n in (2048, 8192, 32768))
    print(f"{name:28}{row}  GB")

print()
print("At 32k context the 70B cache alone exceeds most single-GPU")
print("memories, which is why grouped-query attention exists:")
print("share K and V across heads, shrink the 2 * n * d term.")

Grouped-query attention keeps h query heads but only g key/value heads, dividing the cache by h/g. That is the single most common memory optimisation in modern LLMs, and it is a direct consequence of the byte accounting above.

6. Quadratic Versus Linear

A quadratic attention curve against a linear projection curve, with a ratio table
Figure 42.5: The n² term dominates only once n is large — and then absolutely.
Attention:   O(n^2 d)     quadratic
Projections: O(n d^2)     linear in n

Crossover at n = d. Below it, projections dominate; above it,
attention dominates. For d = 4096 the crossover is at 4096
tokens, which is why long-context work is attention-bound.
import numpy as np

for d in (768, 4096):
    print(f"d_model = {d}")
    for n in (512, 4096, 32768):
        attn = n * n * d
        proj = n * d * d
        print(f"  n={n:6d}  attention/projection = {attn/proj:8.2f}x")
    print(f"  crossover at n = d = {d}")
    print()

At n = d the two costs are equal, and every doubling of context length quadruples the attention term while only doubling the projection term. This is why “just use a longer context” is never free.

7. Worked Examples

7.1 Head dimension must divide d_model

512 / 8 = 64. If h does not divide d_model, the reshape
silently drops or duplicates channels.

7.2 Multi-query attention

h query heads, 1 key head, 1 value head.
Cache shrinks by a factor of h; quality drops slightly and
training is less stable, which is why GQA interpolates.

7.3 Total parameters of an attention block

4 * d_model^2 for Wq, Wk, Wv, W_O, independent of h.

7.4 Why Q is not cached

Each new token produces one query used exactly once.
Past queries are never revisited.

7.5 FlashAttention’s place

It reduces memory traffic, not arithmetic. The n^2 FLOPs are
unchanged, so it does not remove the scaling wall — it moves it.

8. Verify It in Python

import numpy as np

rng = np.random.default_rng(0)

# --- 1. heads cost the same as one wide head -----------------
for d_model, h in [(512, 8), (768, 12), (4096, 32)]:
    d_head = d_model // h
    assert d_model * d_model == h * (d_model * d_head)
    assert d_model % h == 0
print("splitting into heads is parameter-neutral for all configs")

# --- 2. the reshape is lossless ------------------------------
n = 4
Q = rng.normal(size=(n, 512))
Qh = Q.reshape(n, 8, 64).transpose(1, 0, 2)
assert Qh.shape == (8, n, 64)
assert np.allclose(Qh.transpose(1, 0, 2).reshape(n, 512), Q)
print("reshape round-trips exactly")

# --- 3. per-head attention then concat reproduces shapes -----
def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def multi_head(X, Wq, Wk, Wv, h):
    n, d_model = X.shape
    d_head = d_model // h
    outs = []
    for i in range(h):
        sl = slice(i * d_head, (i + 1) * d_head)
        Qi, Ki, Vi = X @ Wq[:, sl], X @ Wk[:, sl], X @ Wv[:, sl]
        A = softmax(Qi @ Ki.T / np.sqrt(d_head))
        outs.append(A @ Vi)
    return np.concatenate(outs, axis=1)

d_model, h = 16, 4
X = rng.normal(size=(5, d_model))
Wq, Wk, Wv = (rng.normal(size=(d_model, d_model)) for _ in range(3))
H = multi_head(X, Wq, Wk, Wv, h)
assert H.shape == (5, d_model)
print("multi-head output shape:", H.shape)

# --- 4. each head block depends only on its own slice --------
H2 = multi_head(X, Wq, Wk, Wv, h)
Wq_mod = Wq.copy(); Wq_mod[:, :4] *= 0.0     # zero head 0's queries
H3 = multi_head(X, Wq_mod, Wk, Wv, h)
d_head = d_model // h
assert np.allclose(H3[:, d_head:], H2[:, d_head:])
assert not np.allclose(H3[:, :d_head], H2[:, :d_head])
print("zeroing head 0's queries leaves the other heads untouched")

# --- 5. W_O mixes every head into every channel --------------
W_O = rng.normal(size=(d_model, d_model))
out = H @ W_O
pert = H.copy(); pert[:, :d_head] = 0.0
out2 = pert @ W_O
assert np.all(out2 != out)
print("W_O spreads a single head's change across all", d_model, "channels")

# --- 6. parameter count of a full attention block ------------
d = 4096
block = 4 * d * d
print(f"attention block Wq+Wk+Wv+W_O at d=4096: {block:,} parameters")
assert block == 4 * 4096 ** 2

# --- 7. KV cache arithmetic ---------------------------------
def kv_gb(n, d_model, layers, batch=1, bpe=2):
    return 2 * n * d_model * layers * batch * bpe / 1e9

c1 = kv_gb(4096, 4096, 32)
c2 = kv_gb(8192, 4096, 32)
assert np.isclose(c2, 2 * c1)
print(f"doubling context doubles the cache: {c1:.2f} GB -> {c2:.2f} GB")

# GQA with 8 groups instead of 32 heads divides the cache by 4
assert np.isclose(kv_gb(4096, 4096, 32) / 4, kv_gb(4096, 1024, 32))
print("GQA with 8 of 32 KV heads divides the cache by 4")

# --- 8. quadratic beats linear past the crossover -----------
d = 4096
cross = d
for n in (d // 2, d, 2 * d, 8 * d):
    attn, proj = n * n * d, n * d * d
    assert (attn > proj) == (n > cross)
print("attention exceeds projections exactly when n > d =", cross)
print("all assertions passed")

9. Practice Questions

  1. Show that h heads of width d/h cost the same as one head of width d.
  2. What would break if h did not divide d_model?
  3. Why does concatenation alone not let heads interact?
  4. What exactly does W_O do?
  5. Compute the parameter count of an attention block at d = 768.
  6. Derive the KV cache size formula.
  7. Why is Q never cached?
  8. How does GQA reduce the cache, and what does it cost?
  9. At what sequence length does attention overtake the projections?
  10. Does FlashAttention change the asymptotic complexity?

Leave a Reply

Your email address will not be published. Required fields are marked *