AI Reading
Quick summary of this article
AI Mathematics Masterclass — Chapter 48 of 48
This chapter assembles everything into one system. We trace a token from integer to prediction through every mathematical object the course has built, account for where the parameters and FLOPs actually go, budget the memory a training run needs, and collect the handful of formulas worth carrying around. Nothing here is new — the point is that it now fits together.
1. Technical Audit & Core Learning Objectives
| Component | Common Tutorial Deficiency | Engineering Standard Applied Here |
|---|---|---|
| Assembly | Chapters never composed into one object. | A token traced end to end with every operation named. |
| Budgeting | Parameter counts quoted without memory context. | All four training budgets computed for a real config. |
| Verification | No end-to-end check anywhere. | A full forward pass implemented and shape-checked. |
| Retention | 48 chapters with no summary. | The ten numbers worth memorising, derived where used. |
Core learning objectives. After this chapter you should be able to:
- Trace a token through a complete transformer.
- Account for parameters by component.
- Compute the four training memory budgets.
- Estimate training FLOPs and wall-clock time.
- Identify which chapter each practical failure traces back to.
- Reproduce the core formulas from memory.
2. One Block, End to End
id -> integer from the tokeniser
x = E[id] (d,) ch 46
x = x + pos (d,) ch 40
for each of L layers:
h = LayerNorm(x) (d,) ch 43
a = MultiHeadAttention(h) (d,) ch 41, 42
x = x + a residual ch 37
h = LayerNorm(x)
f = W2 relu(W1 h) (4d) ch 37, 38
x = x + f
logits = x W_out (V,) ch 46
p = softmax(logits / T) (V,) ch 35, 38
loss = -log p[target] ch 35, 44
import numpy as np
rng = np.random.default_rng(0)
def layer_norm(x, eps=1e-5):
mu, var = x.mean(-1, keepdims=True), x.var(-1, keepdims=True)
return (x - mu) / np.sqrt(var + eps)
def softmax(z):
z = z - z.max(axis=-1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
def attention(X, Wq, Wk, Wv, causal=True):
n, d = X.shape
Q, K, V = X @ Wq, X @ Wk, X @ Wv
S = Q @ K.T / np.sqrt(d)
if causal:
S = S + np.triu(np.full((n, n), -np.inf), k=1)
return softmax(S) @ V
def block(X, P):
h = layer_norm(X)
X = X + attention(h, P["Wq"], P["Wk"], P["Wv"])
h = layer_norm(X)
ff = np.maximum(h @ P["W1"], 0.0) @ P["W2"]
return X + ff
n, d, V, L = 5, 16, 40, 3
P = {}
for k in ("Wq", "Wk", "Wv", "W1", "W2"):
P[k] = rng.normal(size=(d, d)) / np.sqrt(d)
P["W1"] = rng.normal(size=(d, 4 * d)) / np.sqrt(d)
P["W2"] = rng.normal(size=(4 * d, d)) / np.sqrt(4 * d)
E = rng.normal(size=(V, d)) * 0.1
W_out = rng.normal(size=(d, V)) / np.sqrt(d)
ids = np.array([3, 11, 7, 2, 19])
X = E[ids]
for _ in range(L):
X = block(X, P)
logits = X @ W_out
probs = softmax(logits)
print("input ids :", ids)
print("embedding :", E[ids].shape)
print("after blocks :", X.shape)
print("logits :", logits.shape)
print("probs row sums:", np.round(probs.sum(axis=1), 12))
print("argmax per pos:", probs.argmax(axis=1))
Every shape is preserved through the block — (n, d) in, (n, d) out — which is what allows the stack to be repeated L times. The only shape changes are at the embedding (integer to vector), inside the feed-forward (d to 4d and back), and at the output projection (d to V).
3. Where the Parameters Go
Per layer, with model width d:
attention Wq, Wk, Wv, W_O 4 d^2
feed-forward W1 (d x 4d) 4 d^2
W2 (4d x d) 4 d^2
-----
total per layer 12 d^2
plus two norms 4 d
Total = 12 L d^2 + V d (embedding, if untied)
def budget(d, L, V, tie=True):
attn = 4 * d * d
ff = 8 * d * d
norms = 4 * d
body = L * (attn + ff + norms)
emb = V * d
head = 0 if tie else V * d
return dict(attn=L * attn, ff=L * ff, norms=L * norms, emb=emb, head=head,
total=body + emb + head)
for name, d, L, V in [("small", 768, 12, 50_257),
("7B-class", 4096, 32, 128_000),
("large", 8192, 80, 128_000)]:
b = budget(d, L, V)
print(f"{name:10s} d={d:5d} L={L:3d}")
print(f" attention {b['attn']/1e9:8.2f}B feed-forward {b['ff']/1e9:8.2f}B")
print(f" embedding {b['emb']/1e9:8.2f}B norms {b['norms']/1e9:8.3f}B")
print(f" total {b['total']/1e9:8.2f}B")
print(f" ff share of body: {b['ff']/(b['attn']+b['ff']):.1%}")
print()
# what 12 L d^2 predicts
d, L = 4096, 32
print(f"12 L d^2 = {12*L*d*d/1e9:.2f}B (body only, excludes norms and embedding)")
The feed-forward is always exactly twice the attention by parameter count, because 8d² against 4d². Attention dominates compute at long context but the feed-forward dominates capacity at every context.
4. The Four Memory Budgets
Training a model with N parameters:
weights N * bpe
gradients N * bpe (one per parameter)
optimiser N * 2 * 4 (Adam: m and v, in fp32)
activations n * d * L * b * bpe * k
The last term has a constant k of order 10, because every
layer stores several tensors for the backward pass.
Inference needs only the first term plus the KV cache.
def mem_gb(n_params, bpe=2, optimiser_bytes=8):
return (n_params * bpe * 2 + n_params * optimiser_bytes) / 1e9
def act_gb(n, d, L, batch, bpe=2, k=10):
return n * d * L * batch * bpe * k / 1e9
N = 7e9
print(f"7B model, bf16 weights + Adam:")
print(f" weights + grads : {N*2*2/1e9:8.2f} GB")
print(f" optimiser : {N*8/1e9:8.2f} GB")
print(f" subtotal : {mem_gb(N):8.2f} GB")
for n, batch in [(2048, 8), (8192, 8), (8192, 32)]:
a = act_gb(n, 4096, 32, batch)
print(f" activations n={n:5d} b={batch:2d}: {a:7.2f} GB -> total {mem_gb(N)+a:7.2f} GB")
print()
print("Every trick targets one line:")
print(" ZeRO / FSDP shard weights, grads and optimiser across devices")
print(" gradient checkpointing store fewer activations, recompute on backward")
print(" LoRA freeze weights and train a low-rank update")
print(" 8-bit optimiser cut the largest single line")
At 8k context with batch 32 the activation term is 215 GB on its own. This is not a tuning problem — it is why sharding and checkpointing are mandatory rather than optional above a certain size.
5. Cost and Time
Training: 6 N D FLOPs ch 46
Inference: 2 N FLOPs per generated token ch 46
Time = FLOPs / (throughput * utilisation)
Utilisation is the honest part. A 400 TFLOP/s card running a
transformer at 40% efficiency delivers 160 TFLOP/s, and the
planning estimate is wrong by 2.5x if you forget it.
def wall_time(flops, tflops_peak=400.0, utilisation=0.4):
eff = tflops_peak * utilisation * 1e12
seconds = flops / eff
return seconds, seconds / 86400
for label, N, D in [("1.5B / 300B tokens", 1.5e9, 3e11),
("7B / 2T tokens", 7e9, 2e12),
("70B / 2T tokens", 70e9, 2e12)]:
f = 6 * N * D
s, days = wall_time(f)
print(f"{label:22s} {f:8.2e} FLOPs {days:9.1f} GPU-days "
f"({days/30:6.2f} months on 1 GPU, {days/1024/30:.2f} months on 1024)")
print()
print("inference cost per 1M generated tokens for the 7B model:")
per_tok = 2 * 7e9
print(f" {per_tok*1e6/1e15:.3f} PFLOP total — small, but it recurs forever")
The training run is a one-off; inference is not. For a model serving millions of users the cumulative inference cost overtakes training within weeks, which is why 6ND is a planning formula for researchers and 2N is the number that matters commercially.
6. The Numbers Worth Carrying
| Result | Chapter |
|---|---|
| Attention compute is 2n²d per layer; projections 8nd² | 41, 42 |
| A transformer block is 12d² parameters | 48 |
| Softmax-plus-cross-entropy gradient is p − y | 35, 38 |
| Sigmoid’s derivative is bounded by ¼ | 38 |
| PCA error equals the sum of discarded eigenvalues | 36 |
| Cross-entropy = entropy + KL divergence | 35 |
| L1 zeroes, L2 shrinks — because their gradients differ in kind | 34 |
| Output size of a convolution: (n + 2p − d(k−1) − 1)/s + 1 | 39 |
| RoPE makes the attention score depend only on m − n | 40 |
| Training FLOPs ≈ 6ND | 46 |
7. Tracing a Failure Back
| Symptom | Where to look |
|---|---|
| Loss becomes NaN after a few hundred steps | Numerical stability — log of zero, exp overflow (ch 47) |
| Trains but generalises badly | Bias–variance, regularisation (ch 34) |
| Deep network will not train from a good initialisation | Vanishing gradient, activation choice (ch 38) |
| Cannot tell word order | Missing positional encoding (ch 40) |
| Attention maps all look identical | Head collapse (ch 42) |
| Outputs depend on batch composition | BatchNorm in eval mode (ch 43) |
| Loss plateaus at a high value that looks like noise | Class imbalance or the wrong loss (ch 44) |
| Out of memory on a model that should fit | Optimiser state and activations (ch 48) |
| Model confidently wrong, and slow to correct | Saturation and loss choice (ch 38, 44) |
8. Verify It in Python
import numpy as np
rng = np.random.default_rng(0)
relu = lambda z: np.maximum(z, 0.0)
def layer_norm(x, eps=1e-5):
mu, var = x.mean(-1, keepdims=True), x.var(-1, keepdims=True)
return (x - mu) / np.sqrt(var + eps)
def softmax(z):
z = z - z.max(axis=-1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
def attn(X, Wq, Wk, Wv, causal=True):
n = X.shape[0]
Q, K, V = X @ Wq, X @ Wk, X @ Wv
S = Q @ K.T / np.sqrt(X.shape[1])
if causal:
S = S + np.triu(np.full((n, n), -np.inf), 1)
return softmax(S) @ V
def block(X, P):
X = X + attn(layer_norm(X), P["Wq"], P["Wk"], P["Wv"])
h = layer_norm(X)
return X + relu(h @ P["W1"]) @ P["W2"]
# --- 1. full forward pass --------------------------------
n, d, V, L = 4, 8, 20, 2
P = {"Wq": rng.normal(size=(d, d)), "Wk": rng.normal(size=(d, d)),
"Wv": rng.normal(size=(d, d)), "W1": rng.normal(size=(d, 4*d)),
"W2": rng.normal(size=(4*d, d))}
E = rng.normal(size=(V, d)) * 0.1
W_out = rng.normal(size=(d, V))
ids = np.array([1, 4, 2, 9])
X = E[ids]
for _ in range(L):
X = block(X, P)
probs = softmax(X @ W_out)
assert probs.shape == (n, V)
assert np.allclose(probs.sum(axis=1), 1.0)
print(f"forward pass: ids {ids} -> probs {probs.shape}, rows sum to 1")
# --- 2. the block preserves shape ----------------------
assert X.shape == (n, d)
print(f"block maps {(n,d)} to {X.shape}")
# --- 3. parameter formula 12 L d^2 ---------------------
def body_params(d, L):
return L * 12 * d * d
assert body_params(4096, 32) == 6_442_450_944
print(f"12 L d^2 at d=4096, L=32: {body_params(4096,32)/1e9:.2f}B")
# --- 4. feed-forward is twice attention ----------------
for width in (256, 768, 4096): # do not rebind d — it is used below
assert 8 * width * width == 2 * (4 * width * width)
print("feed-forward is always exactly 2x attention")
# --- 5. causal mask keeps the future out ---------------
A = softmax(np.zeros((4, 4)) + np.triu(np.full((4, 4), -np.inf), 1))
assert np.all(A[np.triu_indices(4, 1)] == 0)
assert np.allclose(A.sum(axis=1), 1)
print("causal mask verified")
# --- 6. change at position i cannot affect position j d -----
d_model = 4096
for n_tok in (1024, 4096, 16384):
a, p = n_tok ** 2 * d_model, n_tok * d_model ** 2
assert (a > p) == (n_tok > d_model)
print("attention exceeds projections exactly when n > d")
# --- 10. KV cache doubles when context doubles ---------
kv = lambda n, d, L, bpe=2: 2 * n * d * L * bpe / 1e9
assert np.isclose(kv(8192, 4096, 32), 2 * kv(4096, 4096, 32))
print(f"KV cache at 4k: {kv(4096,4096,32):.2f} GB, at 8k: {kv(8192,4096,32):.2f} GB")
print("all assertions passed")
9. Where to Go Next
- Implement a transformer from scratch with only NumPy, then port it to PyTorch and compare outputs.
- Derive the gradient of a two-layer network by hand and check it against autograd.
- Take a trained small model and measure head entropy, KV cache size and actual FLOPs.
- Re-derive flash attention’s online softmax from the stability argument in Chapter 47.
- Profile a training step and confirm the four memory budgets against reality.
The next course in this series, the PyTorch Engineering Masterclass, takes every formula here and implements it.
10. Official References & Further Reading
Course Complete
- Course Index: AI Mathematics Masterclass — All 48 Chapters
- Previous Chapter: Chapter 47: Numerical Stability, Precision & Underflow Mathematics
- Next Course: PyTorch Engineering Masterclass — All 16 Modules
