AI Reading
Quick summary of this article
AI Mathematics Masterclass — Chapter 47 of 48
Most training failures are not conceptual — they are arithmetic. A softmax overflows, a gradient underflows to exactly zero in fp16, two nearly equal numbers cancel and destroy every significant digit. This chapter works through the failure modes one at a time, derives the stabilisation for each, and explains why bf16 replaced fp16 for training.
1. Technical Audit & Core Learning Objectives
| Component | Common Tutorial Deficiency | Engineering Standard Applied Here |
|---|---|---|
| Float formats | fp16 and bf16 treated as synonyms. | Bit layouts compared and the consequence derived. |
| Overflow | “Use logsumexp” with no derivation. | Max-subtraction proved exact, not approximate. |
| Cancellation | Not mentioned at all. | Demonstrated with concrete expressions that fail. |
| Loss scaling | Called a framework detail. | Shown to be algebraically a no-op that changes only range. |
Core learning objectives. After this chapter you should be able to:
- Compare fp32, fp16 and bf16 bit layouts.
- Explain why bf16 is preferred for training.
- Prove that subtracting the max from logits preserves softmax exactly.
- Implement a stable log-sum-exp.
- Recognise catastrophic cancellation and fix it.
- Explain loss scaling as an exact rescaling.
2. What Each Format Buys
sign exponent mantissa range precision
fp32 1 8 23 1e-38..3e38 ~7 digits
fp16 1 5 10 6e-8..6e4 ~3 digits
bf16 1 8 7 1e-38..3e38 ~2 digits
fp16's exponent is too narrow: gradients below ~6e-5
underflow to exactly zero, and activations above 65504
become inf.
bf16 has fp32's exponent and fewer mantissa bits. It cannot
represent small gradients precisely, but it can represent
them at all — which matters more.
import numpy as np
def bits(name, sign, exp, mant):
return dict(name=name, sign=sign, exp=exp, mant=mant,
total=sign + exp + mant, levels=2 ** mant)
for name, s, e, m in [("fp32", 1, 8, 23), ("fp16", 1, 5, 10), ("bf16", 1, 8, 7)]:
b = bits(name, s, e, m)
print(f"{b['name']:6s} {b['total']:2d} bits "
f"exponent levels 2^{e} = {2**e:4d} "
f"mantissa levels {b['levels']:>9,}")
# fp16 underflow and overflow in practice
print("\nfp16 behaviour:")
for v in (1e-4, 6e-5, 1e-7, 1e-8, 65504.0, 1e5):
try:
a = np.float16(v)
print(f" {v:9.1e} -> {a}")
except OverflowError:
print(f" {v:9.1e} -> overflow")
# 8-bit optimiser state is a separate axis
print("\nmemory for 7e9 parameters:")
for name, nbytes in [("fp32", 4), ("fp16/bf16", 2), ("8-bit", 1)]:
print(f" {name:10s} {7e9*nbytes/1e9:6.1f} GB")
Note that bf16 is strictly worse than fp16 in precision and strictly better in range. For deep learning the range is what breaks first, which is why the trade is worth making.
3. Softmax Overflow
naive: exp(1000) = inf -> inf / inf = nan
stable: let m = max(z)
exp(z_i - m) = exp(z_i) / exp(m)
softmax_i = exp(z_i - m) / sum_j exp(z_j - m)
= exp(z_i)/exp(m) / (sum_j exp(z_j)/exp(m))
= exp(z_i) / sum_j exp(z_j)
The exp(m) cancels exactly. So the shift is mathematically
free — only the floating-point range changes.
import numpy as np
def softmax_naive(z):
e = np.exp(z)
return e / e.sum()
def softmax_stable(z):
z = z - z.max()
e = np.exp(z)
return e / e.sum()
z = np.array([990.0, 1000.0, 1010.0])
with np.errstate(over="ignore", invalid="ignore"):
naive = softmax_naive(z)
stable = softmax_stable(z)
print("naive :", naive)
print("stable:", np.round(stable, 6), " sum =", stable.sum())
# shift invariance on a well-scaled input
z2 = np.array([1.0, 2.0, 3.0])
print("\nshift invariance:", np.allclose(softmax_stable(z2), softmax_stable(z2 + 500)))
# log-sum-exp, used for log-probabilities
def logsumexp(z):
m = z.max()
return m + np.log(np.sum(np.exp(z - m)))
print("logsumexp:", round(float(logsumexp(z)), 6))
print("matches log(sum(exp(z - max))) + max analytically")
Every framework’s log_softmax and cross_entropy uses exactly this shift. Writing log(softmax(x)) by hand reintroduces the overflow and, worse, the underflow when a probability is tiny.
4. Catastrophic Cancellation
1e16 + 1 - 1e16 == 0 in float64, not 1.
Why: 1e16 needs 54 bits to represent, but the mantissa has 53.
Adding 1 is below the ulp at that magnitude, so it rounds away.
Then subtracting two nearly equal large numbers leaves only
the rounding error — the answer has no correct digits at all.
Common cases in ML: variance computed as E[x^2] - E[x]^2,
softmax denominators that are nearly equal, and any
"difference of two big numbers" formula.
import numpy as np
print("1e16 + 1 - 1e16 =", 1e16 + 1 - 1e16)
print("0.1 + 0.2 - 0.3 =", repr(0.1 + 0.2 - 0.3))
print("np.finfo(float).eps =", np.finfo(float).eps)
print("ulp at 1e16 =", np.spacing(1e16))
# the variance trap: E[x^2] - E[x]^2 loses precision
rng = np.random.default_rng(0)
x = rng.normal(loc=1e8, scale=1.0, size=100000)
naive_var = np.mean(x ** 2) - np.mean(x) ** 2
stable_var = np.mean((x - np.mean(x)) ** 2)
print(f"\nnaive E[x^2]-E[x]^2 : {naive_var:.6f}")
print(f"two-pass variance : {stable_var:.6f}")
print(f"difference : {abs(naive_var - stable_var):.6f}")
# the same trap with the textbook quadratic formula
a, b, c = 1.0, -1e8, 1.0
root_naive = (-b - np.sqrt(b ** 2 - 4 * a * c)) / (2 * a)
root_stable = (2 * c) / (-b + np.sqrt(b ** 2 - 4 * a * c))
print(f"\nquadratic small root: naive {root_naive:.12e} stable {root_stable:.12e}")
The two-pass variance is not merely more accurate — it is correct, while the naive form can even return a negative variance. Any formula that computes a small quantity as the difference of two large ones is suspect.
5. Loss Scaling
fp16's smallest NORMAL is about 6.1e-5. Below that, values
become SUBNORMAL: representable, but with progressively fewer
significant bits. Below about 6e-8 they become exactly 0.
So the failure has two stages — precision decay, then silent
zero. The parameter stops updating and nothing signals it.
Fix: multiply the loss by S before backward, divide the
gradients by S before the step.
d(LS)/dw = S * dL/dw
step: w -= lr * (S * g) / S = w - lr * g
Identical update, different exponent range.
Dynamic scaling: if a step produces inf, halve S and skip;
otherwise double S every N good steps.
import numpy as np
print("fp16 smallest normal :", np.finfo(np.float16).tiny)
print("fp16 smallest subnormal:", np.nextafter(np.float16(0), np.float16(1)))
for true_grad in (1e-4, 1e-5, 1e-7, 1e-9):
direct = np.float16(true_grad)
scaled = np.float16(true_grad * 1024)
rec = np.float32(scaled) / 1024
err = abs(rec - true_grad) / true_grad
print(f"grad {true_grad:8.1e}: fp16 direct {float(direct):10.3e} "
f"scaled->recovered {rec:10.3e} rel err {err:8.2%}")
# the update is mathematically identical
# the vanishing case, where the update is lost entirely
g_tiny = 1e-9
print(f"\ngradient {g_tiny:.0e}: fp16 direct -> {np.float16(g_tiny)}")
lr, w = 0.1, 1.0
print(f"update without scaling: {w - lr * float(np.float16(g_tiny)):.15f}")
print(f"update with scaling : {w - lr * (float(np.float32(np.float16(g_tiny*1024)))/1024):.15f}")
print(f"true update : {w - lr * g_tiny:.15f}")
# dynamic scaling decision
for g in (1e-3, 1e-7, np.inf):
S = 1024
gs = g * S
action = "halve S and skip" if not np.isfinite(gs) else "use S"
print(f"grad {g:>8.1e} * S={S} -> {gs:>10.3e} -> {action}")
Read the third row: an unscaled 1e-7 is stored as 1.192e-7 — subnormal, with about 19% relative error. Scaled, the same gradient recovers to 1.000e-7 with essentially no error. In the last row the unscaled value is exactly zero and the scaled one comes back with about 1% error, because it now sits in the normal range and is limited only by fp16’s ten mantissa bits.
So scaling does not restore precision — it moves the value into a range where any precision survives. And the final row is also why mixed-precision training keeps a master fp32 copy of the weights: an update that small cannot be accumulated in fp16 at all.
6. Failure Modes and Fixes
| Habit | Why it matters |
|---|---|
Never take log of a raw probability |
Underflow to 0 becomes -inf and then NaN |
Never exp without subtracting the max first |
Overflow to inf spreads through the batch |
| Prefer a two-pass variance | E[x²] − E[x]² can return a negative number |
| Clip gradients | One bad batch cannot destroy the weights |
| Check for finite values in the loss | NaN propagates silently and permanently |
7. Worked Examples
7.1 Mixed precision
Compute the forward and backward passes in bf16 for speed,
keep a master copy of the weights in fp32 for the update.
The master copy is what stops small updates from being lost.
7.2 Why the master copy exists
An update of 1e-6 added to a weight of 1.0 is invisible in
bf16 (7 mantissa bits gives ulp ~8e-3 there). In fp32 it is
representable. Update the fp32 copy, then cast down.
7.3 Softmax in the log domain
Cross-entropy from logits directly:
loss = logsumexp(z) - z[target]
No probability is ever formed, so nothing can underflow.
7.4 NaN hunting
NaN is contagious and silent. Assert finiteness at layer
boundaries during debugging; the first NaN is the cause, and
everything downstream is a symptom.
7.5 Determinism
Floating-point addition is not associative, so parallel
reductions produce different rounding runs to run. Bit-exact
reproducibility requires deterministic kernels and is slower.
8. Verify It in Python
import numpy as np
# --- 1. fp16 range limits -----------------------------------
assert np.float16(1e-8) == 0.0
assert np.isfinite(np.float16(65504.0))
print("fp16: 1e-8 underflows to", np.float16(1e-8), "; 65504 is the max")
# --- 2. bf16 keeps the range (emulated via bit truncation) --
def to_bf16(x):
x = np.asarray(x, dtype=np.float32)
bits = x.view(np.uint32) & 0xFFFF0000
return bits.view(np.float32)
tiny = np.array([1e-30, 1e-8, 1.0, 1e30], dtype=np.float32)
assert np.all(to_bf16(tiny) != 0)
print("bf16-emulated values:", to_bf16(tiny))
# --- 3. naive softmax overflows, stable does not ------------
z = np.array([900.0, 1000.0, 1100.0])
with np.errstate(over="ignore", invalid="ignore"):
naive = np.exp(z) / np.exp(z).sum()
assert not np.all(np.isfinite(naive))
s = np.exp(z - z.max()); stable = s / s.sum()
assert np.allclose(stable.sum(), 1.0)
assert np.all(np.isfinite(stable))
print("naive ->", naive, " stable ->", np.round(stable, 6))
# --- 4. the shift is exact ----------------------------------
rng = np.random.default_rng(0)
for _ in range(100):
zz = rng.normal(size=6) * 20
a = np.exp(zz - zz.max()); a /= a.sum()
b = np.exp(zz - zz.max() - 3.5); b /= b.sum()
assert np.allclose(a, b, atol=1e-12)
print("softmax is invariant to any constant shift")
# --- 5. logsumexp stability ---------------------------------
def logsumexp(v):
m = v.max()
return m + np.log(np.sum(np.exp(v - m)))
big = np.array([1000.0, 1000.0])
assert np.isclose(logsumexp(big), 1000.0 + np.log(2))
print("logsumexp(1000,1000) =", logsumexp(big), "(no overflow)")
# --- 6. cross-entropy from logits, no probabilities ---------
def ce_from_logits(logits, target):
return logsumexp(logits) - logits[target]
logits = np.array([2.0, 1.0, 0.5])
p = np.exp(logits - logits.max()); p /= p.sum()
assert np.isclose(ce_from_logits(logits, 0), -np.log(p[0]))
print("CE from logits matches -log(softmax)")
# --- 7. catastrophic cancellation ---------------------------
assert 1e16 + 1 - 1e16 == 0.0
assert (0.1 + 0.2 - 0.3) != 0.0
print("1e16 + 1 - 1e16 =", 1e16 + 1 - 1e16)
# --- 8. two-pass variance is correct -----------------------
x = rng.normal(loc=1e8, scale=1.0, size=200_000)
naive_v = np.mean(x ** 2) - np.mean(x) ** 2
stable_v = np.mean((x - np.mean(x)) ** 2)
assert abs(stable_v - 1.0) < 0.05
print(f"two-pass var {stable_v:.6f} vs naive {naive_v:.6f}")
# --- 9. loss scaling is algebraically a no-op ---------------
g = np.float32(3.0)
S = np.float32(1024.0)
assert np.isclose(float(g * S) / float(S), float(g))
print("(g * S) / S == g")
# --- 10. gradient clipping preserves direction -------------
def clip(grad, c):
n = np.linalg.norm(grad)
return grad if n <= c else grad * (c / n)
gv = np.array([30.0, 40.0]) # norm 50
cg = clip(gv, 5.0)
assert np.isclose(np.linalg.norm(cg), 5.0)
assert np.allclose(cg / np.linalg.norm(cg), gv / np.linalg.norm(gv))
print("clipping changes magnitude, never direction")
print("all assertions passed")
9. Practice Questions
- Compare the bit layouts of fp32, fp16 and bf16.
- Why does fp16 need loss scaling but bf16 usually does not?
- Prove that subtracting the max preserves softmax exactly.
- Implement log-sum-exp and explain each term.
- Give an expression that loses all precision in float64.
- Why is
E[x²] − E[x]²dangerous? - Show that loss scaling leaves the update unchanged.
- Why does mixed precision keep an fp32 master copy?
- Compute the loss directly from logits without a probability.
- Why is bit-exact reproducibility expensive?
10. Official References & Further Reading
Continue the Course
- Course Index: AI Mathematics Masterclass — All 48 Chapters
- Previous Chapter: Chapter 46: LLM Mathematics
- Next Chapter: Chapter 48: Capstone — The Complete Mathematics of a Transformer
- Related Series: PyTorch Engineering Masterclass — All 16 Modules
