Chapter 48: Capstone: The Complete Mathematics of a Transformer

Chapter 48: Capstone: The Complete Mathematics of a Transformer

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 48 of 48

This chapter assembles everything into one system. We trace a token from integer to prediction through every mathematical object the course has built, account for where the parameters and FLOPs actually go, budget the memory a training run needs, and collect the handful of formulas worth carrying around. Nothing here is new — the point is that it now fits together.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Assembly Chapters never composed into one object. A token traced end to end with every operation named.
Budgeting Parameter counts quoted without memory context. All four training budgets computed for a real config.
Verification No end-to-end check anywhere. A full forward pass implemented and shape-checked.
Retention 48 chapters with no summary. The ten numbers worth memorising, derived where used.

Core learning objectives. After this chapter you should be able to:

  1. Trace a token through a complete transformer.
  2. Account for parameters by component.
  3. Compute the four training memory budgets.
  4. Estimate training FLOPs and wall-clock time.
  5. Identify which chapter each practical failure traces back to.
  6. Reproduce the core formulas from memory.

2. One Block, End to End

A transformer block shown as normalise, attend, add, normalise, feed-forward, add
Figure 48.1: Two sub-layers, two normalisations, two residual adds. That is the whole design.
id            -> integer from the tokeniser
x  = E[id]                            (d,)        ch 46
x  = x + pos                          (d,)        ch 40

for each of L layers:
    h = LayerNorm(x)                  (d,)        ch 43
    a = MultiHeadAttention(h)         (d,)        ch 41, 42
    x = x + a                         residual    ch 37
    h = LayerNorm(x)
    f = W2 relu(W1 h)                 (4d)        ch 37, 38
    x = x + f

logits = x W_out                      (V,)        ch 46
p      = softmax(logits / T)          (V,)        ch 35, 38
loss   = -log p[target]                           ch 35, 44
import numpy as np

rng = np.random.default_rng(0)

def layer_norm(x, eps=1e-5):
    mu, var = x.mean(-1, keepdims=True), x.var(-1, keepdims=True)
    return (x - mu) / np.sqrt(var + eps)

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def attention(X, Wq, Wk, Wv, causal=True):
    n, d = X.shape
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    S = Q @ K.T / np.sqrt(d)
    if causal:
        S = S + np.triu(np.full((n, n), -np.inf), k=1)
    return softmax(S) @ V

def block(X, P):
    h = layer_norm(X)
    X = X + attention(h, P["Wq"], P["Wk"], P["Wv"])
    h = layer_norm(X)
    ff = np.maximum(h @ P["W1"], 0.0) @ P["W2"]
    return X + ff

n, d, V, L = 5, 16, 40, 3
P = {}
for k in ("Wq", "Wk", "Wv", "W1", "W2"):
    P[k] = rng.normal(size=(d, d)) / np.sqrt(d)
P["W1"] = rng.normal(size=(d, 4 * d)) / np.sqrt(d)
P["W2"] = rng.normal(size=(4 * d, d)) / np.sqrt(4 * d)
E = rng.normal(size=(V, d)) * 0.1
W_out = rng.normal(size=(d, V)) / np.sqrt(d)

ids = np.array([3, 11, 7, 2, 19])
X = E[ids]
for _ in range(L):
    X = block(X, P)

logits = X @ W_out
probs = softmax(logits)
print("input ids     :", ids)
print("embedding     :", E[ids].shape)
print("after blocks  :", X.shape)
print("logits        :", logits.shape)
print("probs row sums:", np.round(probs.sum(axis=1), 12))
print("argmax per pos:", probs.argmax(axis=1))

Every shape is preserved through the block — (n, d) in, (n, d) out — which is what allows the stack to be repeated L times. The only shape changes are at the embedding (integer to vector), inside the feed-forward (d to 4d and back), and at the output projection (d to V).

3. Where the Parameters Go

A parameter budget table showing the feed-forward taking twice the attention parameters
Figure 48.2: The feed-forward holds twice the attention parameters.
Per layer, with model width d:

  attention   Wq, Wk, Wv, W_O   4 d^2
  feed-forward W1 (d x 4d)      4 d^2
               W2 (4d x d)      4 d^2
                                -----
  total per layer               12 d^2
  plus two norms                4 d

Total = 12 L d^2  +  V d          (embedding, if untied)
def budget(d, L, V, tie=True):
    attn = 4 * d * d
    ff = 8 * d * d
    norms = 4 * d
    body = L * (attn + ff + norms)
    emb = V * d
    head = 0 if tie else V * d
    return dict(attn=L * attn, ff=L * ff, norms=L * norms, emb=emb, head=head,
                total=body + emb + head)

for name, d, L, V in [("small", 768, 12, 50_257),
                      ("7B-class", 4096, 32, 128_000),
                      ("large", 8192, 80, 128_000)]:
    b = budget(d, L, V)
    print(f"{name:10s} d={d:5d} L={L:3d}")
    print(f"  attention {b['attn']/1e9:8.2f}B   feed-forward {b['ff']/1e9:8.2f}B")
    print(f"  embedding {b['emb']/1e9:8.2f}B   norms        {b['norms']/1e9:8.3f}B")
    print(f"  total     {b['total']/1e9:8.2f}B")
    print(f"  ff share of body: {b['ff']/(b['attn']+b['ff']):.1%}")
    print()

# what 12 L d^2 predicts
d, L = 4096, 32
print(f"12 L d^2 = {12*L*d*d/1e9:.2f}B (body only, excludes norms and embedding)")

The feed-forward is always exactly twice the attention by parameter count, because 8d² against 4d². Attention dominates compute at long context but the feed-forward dominates capacity at every context.

4. The Four Memory Budgets

Four bars for weights, gradients, optimiser state and activations, with the optimiser largest
Figure 48.3: Training memory is three to five times the model size before activations.
Training a model with N parameters:

  weights          N * bpe
  gradients        N * bpe          (one per parameter)
  optimiser        N * 2 * 4        (Adam: m and v, in fp32)
  activations      n * d * L * b * bpe * k

The last term has a constant k of order 10, because every
layer stores several tensors for the backward pass.

Inference needs only the first term plus the KV cache.
def mem_gb(n_params, bpe=2, optimiser_bytes=8):
    return (n_params * bpe * 2 + n_params * optimiser_bytes) / 1e9

def act_gb(n, d, L, batch, bpe=2, k=10):
    return n * d * L * batch * bpe * k / 1e9

N = 7e9
print(f"7B model, bf16 weights + Adam:")
print(f"  weights + grads : {N*2*2/1e9:8.2f} GB")
print(f"  optimiser       : {N*8/1e9:8.2f} GB")
print(f"  subtotal        : {mem_gb(N):8.2f} GB")
for n, batch in [(2048, 8), (8192, 8), (8192, 32)]:
    a = act_gb(n, 4096, 32, batch)
    print(f"  activations n={n:5d} b={batch:2d}: {a:7.2f} GB  -> total {mem_gb(N)+a:7.2f} GB")

print()
print("Every trick targets one line:")
print("  ZeRO / FSDP        shard weights, grads and optimiser across devices")
print("  gradient checkpointing   store fewer activations, recompute on backward")
print("  LoRA                freeze weights and train a low-rank update")
print("  8-bit optimiser     cut the largest single line")

At 8k context with batch 32 the activation term is 215 GB on its own. This is not a tuning problem — it is why sharding and checkpointing are mandatory rather than optional above a certain size.

5. Cost and Time

The eight stages of the course shown as a dependency chain from foundations to systems
Figure 48.4: Every stage depends on the ones below it.
Training:   6 N D FLOPs                     ch 46
Inference:  2 N FLOPs per generated token    ch 46

Time = FLOPs / (throughput * utilisation)

Utilisation is the honest part. A 400 TFLOP/s card running a
transformer at 40% efficiency delivers 160 TFLOP/s, and the
planning estimate is wrong by 2.5x if you forget it.
def wall_time(flops, tflops_peak=400.0, utilisation=0.4):
    eff = tflops_peak * utilisation * 1e12
    seconds = flops / eff
    return seconds, seconds / 86400

for label, N, D in [("1.5B / 300B tokens", 1.5e9, 3e11),
                    ("7B / 2T tokens", 7e9, 2e12),
                    ("70B / 2T tokens", 70e9, 2e12)]:
    f = 6 * N * D
    s, days = wall_time(f)
    print(f"{label:22s} {f:8.2e} FLOPs  {days:9.1f} GPU-days  "
          f"({days/30:6.2f} months on 1 GPU, {days/1024/30:.2f} months on 1024)")

print()
print("inference cost per 1M generated tokens for the 7B model:")
per_tok = 2 * 7e9
print(f"  {per_tok*1e6/1e15:.3f} PFLOP total — small, but it recurs forever")

The training run is a one-off; inference is not. For a model serving millions of users the cumulative inference cost overtakes training within weeks, which is why 6ND is a planning formula for researchers and 2N is the number that matters commercially.

6. The Numbers Worth Carrying

A table of ten key formulas with their values and meaning
Figure 48.5: Ten formulas, each derived somewhere in the previous 47 chapters.
Result Chapter
Attention compute is 2n²d per layer; projections 8nd² 41, 42
A transformer block is 12d² parameters 48
Softmax-plus-cross-entropy gradient is p − y 35, 38
Sigmoid’s derivative is bounded by ¼ 38
PCA error equals the sum of discarded eigenvalues 36
Cross-entropy = entropy + KL divergence 35
L1 zeroes, L2 shrinks — because their gradients differ in kind 34
Output size of a convolution: (n + 2p − d(k−1) − 1)/s + 1 39
RoPE makes the attention score depend only on m − n 40
Training FLOPs ≈ 6ND 46

7. Tracing a Failure Back

Symptom Where to look
Loss becomes NaN after a few hundred steps Numerical stability — log of zero, exp overflow (ch 47)
Trains but generalises badly Bias–variance, regularisation (ch 34)
Deep network will not train from a good initialisation Vanishing gradient, activation choice (ch 38)
Cannot tell word order Missing positional encoding (ch 40)
Attention maps all look identical Head collapse (ch 42)
Outputs depend on batch composition BatchNorm in eval mode (ch 43)
Loss plateaus at a high value that looks like noise Class imbalance or the wrong loss (ch 44)
Out of memory on a model that should fit Optimiser state and activations (ch 48)
Model confidently wrong, and slow to correct Saturation and loss choice (ch 38, 44)

8. Verify It in Python

import numpy as np

rng = np.random.default_rng(0)
relu = lambda z: np.maximum(z, 0.0)

def layer_norm(x, eps=1e-5):
    mu, var = x.mean(-1, keepdims=True), x.var(-1, keepdims=True)
    return (x - mu) / np.sqrt(var + eps)

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def attn(X, Wq, Wk, Wv, causal=True):
    n = X.shape[0]
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    S = Q @ K.T / np.sqrt(X.shape[1])
    if causal:
        S = S + np.triu(np.full((n, n), -np.inf), 1)
    return softmax(S) @ V

def block(X, P):
    X = X + attn(layer_norm(X), P["Wq"], P["Wk"], P["Wv"])
    h = layer_norm(X)
    return X + relu(h @ P["W1"]) @ P["W2"]

# --- 1. full forward pass --------------------------------
n, d, V, L = 4, 8, 20, 2
P = {"Wq": rng.normal(size=(d, d)), "Wk": rng.normal(size=(d, d)),
     "Wv": rng.normal(size=(d, d)), "W1": rng.normal(size=(d, 4*d)),
     "W2": rng.normal(size=(4*d, d))}
E = rng.normal(size=(V, d)) * 0.1
W_out = rng.normal(size=(d, V))

ids = np.array([1, 4, 2, 9])
X = E[ids]
for _ in range(L):
    X = block(X, P)
probs = softmax(X @ W_out)
assert probs.shape == (n, V)
assert np.allclose(probs.sum(axis=1), 1.0)
print(f"forward pass: ids {ids} -> probs {probs.shape}, rows sum to 1")

# --- 2. the block preserves shape ----------------------
assert X.shape == (n, d)
print(f"block maps {(n,d)} to {X.shape}")

# --- 3. parameter formula 12 L d^2 ---------------------
def body_params(d, L):
    return L * 12 * d * d

assert body_params(4096, 32) == 6_442_450_944
print(f"12 L d^2 at d=4096, L=32: {body_params(4096,32)/1e9:.2f}B")

# --- 4. feed-forward is twice attention ----------------
for width in (256, 768, 4096):          # do not rebind d — it is used below
    assert 8 * width * width == 2 * (4 * width * width)
print("feed-forward is always exactly 2x attention")

# --- 5. causal mask keeps the future out ---------------
A = softmax(np.zeros((4, 4)) + np.triu(np.full((4, 4), -np.inf), 1))
assert np.all(A[np.triu_indices(4, 1)] == 0)
assert np.allclose(A.sum(axis=1), 1)
print("causal mask verified")

# --- 6. change at position i cannot affect position j d -----
d_model = 4096
for n_tok in (1024, 4096, 16384):
    a, p = n_tok ** 2 * d_model, n_tok * d_model ** 2
    assert (a > p) == (n_tok > d_model)
print("attention exceeds projections exactly when n > d")

# --- 10. KV cache doubles when context doubles ---------
kv = lambda n, d, L, bpe=2: 2 * n * d * L * bpe / 1e9
assert np.isclose(kv(8192, 4096, 32), 2 * kv(4096, 4096, 32))
print(f"KV cache at 4k: {kv(4096,4096,32):.2f} GB, at 8k: {kv(8192,4096,32):.2f} GB")
print("all assertions passed")

9. Where to Go Next

  1. Implement a transformer from scratch with only NumPy, then port it to PyTorch and compare outputs.
  2. Derive the gradient of a two-layer network by hand and check it against autograd.
  3. Take a trained small model and measure head entropy, KV cache size and actual FLOPs.
  4. Re-derive flash attention’s online softmax from the stability argument in Chapter 47.
  5. Profile a training step and confirm the four memory budgets against reality.

The next course in this series, the PyTorch Engineering Masterclass, takes every formula here and implements it.

Leave a Reply

Your email address will not be published. Required fields are marked *