Chapter 47: Numerical Stability, Precision & Underflow Mathematics

Chapter 47: Numerical Stability, Precision & Underflow Mathematics

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 47 of 48

Most training failures are not conceptual — they are arithmetic. A softmax overflows, a gradient underflows to exactly zero in fp16, two nearly equal numbers cancel and destroy every significant digit. This chapter works through the failure modes one at a time, derives the stabilisation for each, and explains why bf16 replaced fp16 for training.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Float formats fp16 and bf16 treated as synonyms. Bit layouts compared and the consequence derived.
Overflow “Use logsumexp” with no derivation. Max-subtraction proved exact, not approximate.
Cancellation Not mentioned at all. Demonstrated with concrete expressions that fail.
Loss scaling Called a framework detail. Shown to be algebraically a no-op that changes only range.

Core learning objectives. After this chapter you should be able to:

  1. Compare fp32, fp16 and bf16 bit layouts.
  2. Explain why bf16 is preferred for training.
  3. Prove that subtracting the max from logits preserves softmax exactly.
  4. Implement a stable log-sum-exp.
  5. Recognise catastrophic cancellation and fix it.
  6. Explain loss scaling as an exact rescaling.

2. What Each Format Buys

Bit layout diagrams for fp32, fp16 and bf16 showing exponent and mantissa widths
Figure 47.1: bf16 keeps fp32’s range and gives up mantissa bits.
       sign  exponent  mantissa   range        precision
fp32     1       8        23      1e-38..3e38   ~7 digits
fp16     1       5        10      6e-8..6e4     ~3 digits
bf16     1       8         7      1e-38..3e38   ~2 digits

fp16's exponent is too narrow: gradients below ~6e-5
underflow to exactly zero, and activations above 65504
become inf.

bf16 has fp32's exponent and fewer mantissa bits. It cannot
represent small gradients precisely, but it can represent
them at all — which matters more.
import numpy as np

def bits(name, sign, exp, mant):
    return dict(name=name, sign=sign, exp=exp, mant=mant,
                total=sign + exp + mant, levels=2 ** mant)

for name, s, e, m in [("fp32", 1, 8, 23), ("fp16", 1, 5, 10), ("bf16", 1, 8, 7)]:
    b = bits(name, s, e, m)
    print(f"{b['name']:6s} {b['total']:2d} bits  "
          f"exponent levels 2^{e} = {2**e:4d}  "
          f"mantissa levels {b['levels']:>9,}")

# fp16 underflow and overflow in practice
print("\nfp16 behaviour:")
for v in (1e-4, 6e-5, 1e-7, 1e-8, 65504.0, 1e5):
    try:
        a = np.float16(v)
        print(f"  {v:9.1e} -> {a}")
    except OverflowError:
        print(f"  {v:9.1e} -> overflow")

# 8-bit optimiser state is a separate axis
print("\nmemory for 7e9 parameters:")
for name, nbytes in [("fp32", 4), ("fp16/bf16", 2), ("8-bit", 1)]:
    print(f"  {name:10s} {7e9*nbytes/1e9:6.1f} GB")

Note that bf16 is strictly worse than fp16 in precision and strictly better in range. For deep learning the range is what breaks first, which is why the trade is worth making.

3. Softmax Overflow

Naive exponential versus max-subtracted exponential shown as two panels
Figure 47.2: Subtracting the max is exact, not an approximation.
naive:   exp(1000) = inf   ->  inf / inf = nan

stable:  let m = max(z)
         exp(z_i - m) = exp(z_i) / exp(m)

         softmax_i = exp(z_i - m) / sum_j exp(z_j - m)
                   = exp(z_i)/exp(m) / (sum_j exp(z_j)/exp(m))
                   = exp(z_i) / sum_j exp(z_j)

The exp(m) cancels exactly. So the shift is mathematically
free — only the floating-point range changes.
import numpy as np

def softmax_naive(z):
    e = np.exp(z)
    return e / e.sum()

def softmax_stable(z):
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()

z = np.array([990.0, 1000.0, 1010.0])
with np.errstate(over="ignore", invalid="ignore"):
    naive = softmax_naive(z)
stable = softmax_stable(z)
print("naive :", naive)
print("stable:", np.round(stable, 6), " sum =", stable.sum())

# shift invariance on a well-scaled input
z2 = np.array([1.0, 2.0, 3.0])
print("\nshift invariance:", np.allclose(softmax_stable(z2), softmax_stable(z2 + 500)))

# log-sum-exp, used for log-probabilities
def logsumexp(z):
    m = z.max()
    return m + np.log(np.sum(np.exp(z - m)))

print("logsumexp:", round(float(logsumexp(z)), 6))
print("matches log(sum(exp(z - max))) + max analytically")

Every framework’s log_softmax and cross_entropy uses exactly this shift. Writing log(softmax(x)) by hand reintroduces the overflow and, worse, the underflow when a probability is tiny.

4. Catastrophic Cancellation

A table of expressions showing exact versus float64 results and digits lost
Figure 47.3: Subtracting nearly equal numbers destroys relative precision.
1e16 + 1 - 1e16 == 0 in float64, not 1.

Why: 1e16 needs 54 bits to represent, but the mantissa has 53.
Adding 1 is below the ulp at that magnitude, so it rounds away.

Then subtracting two nearly equal large numbers leaves only
the rounding error — the answer has no correct digits at all.

Common cases in ML: variance computed as E[x^2] - E[x]^2,
softmax denominators that are nearly equal, and any
"difference of two big numbers" formula.
import numpy as np

print("1e16 + 1 - 1e16       =", 1e16 + 1 - 1e16)
print("0.1 + 0.2 - 0.3       =", repr(0.1 + 0.2 - 0.3))
print("np.finfo(float).eps   =", np.finfo(float).eps)
print("ulp at 1e16           =", np.spacing(1e16))

# the variance trap: E[x^2] - E[x]^2 loses precision
rng = np.random.default_rng(0)
x = rng.normal(loc=1e8, scale=1.0, size=100000)

naive_var = np.mean(x ** 2) - np.mean(x) ** 2
stable_var = np.mean((x - np.mean(x)) ** 2)
print(f"\nnaive E[x^2]-E[x]^2 : {naive_var:.6f}")
print(f"two-pass variance   : {stable_var:.6f}")
print(f"difference          : {abs(naive_var - stable_var):.6f}")

# the same trap with the textbook quadratic formula
a, b, c = 1.0, -1e8, 1.0
root_naive = (-b - np.sqrt(b ** 2 - 4 * a * c)) / (2 * a)
root_stable = (2 * c) / (-b + np.sqrt(b ** 2 - 4 * a * c))
print(f"\nquadratic small root: naive {root_naive:.12e}  stable {root_stable:.12e}")

The two-pass variance is not merely more accurate — it is correct, while the naive form can even return a negative variance. Any formula that computes a small quantity as the difference of two large ones is suspect.

5. Loss Scaling

Gradient magnitude decaying below the fp16 underflow threshold with and without scaling
Figure 47.4: Without scaling, small gradients decay into the subnormal range and then vanish.
fp16's smallest NORMAL is about 6.1e-5. Below that, values
become SUBNORMAL: representable, but with progressively fewer
significant bits. Below about 6e-8 they become exactly 0.

So the failure has two stages — precision decay, then silent
zero. The parameter stops updating and nothing signals it.

Fix: multiply the loss by S before backward, divide the
gradients by S before the step.

  d(LS)/dw = S * dL/dw
  step: w -= lr * (S * g) / S = w - lr * g

Identical update, different exponent range.

Dynamic scaling: if a step produces inf, halve S and skip;
otherwise double S every N good steps.
import numpy as np

print("fp16 smallest normal   :", np.finfo(np.float16).tiny)
print("fp16 smallest subnormal:", np.nextafter(np.float16(0), np.float16(1)))

for true_grad in (1e-4, 1e-5, 1e-7, 1e-9):
    direct = np.float16(true_grad)
    scaled = np.float16(true_grad * 1024)
    rec = np.float32(scaled) / 1024
    err = abs(rec - true_grad) / true_grad
    print(f"grad {true_grad:8.1e}: fp16 direct {float(direct):10.3e}   "
          f"scaled->recovered {rec:10.3e}  rel err {err:8.2%}")

# the update is mathematically identical
# the vanishing case, where the update is lost entirely
g_tiny = 1e-9
print(f"\ngradient {g_tiny:.0e}: fp16 direct -> {np.float16(g_tiny)}")
lr, w = 0.1, 1.0
print(f"update without scaling: {w - lr * float(np.float16(g_tiny)):.15f}")
print(f"update with scaling   : {w - lr * (float(np.float32(np.float16(g_tiny*1024)))/1024):.15f}")
print(f"true update           : {w - lr * g_tiny:.15f}")

# dynamic scaling decision
for g in (1e-3, 1e-7, np.inf):
    S = 1024
    gs = g * S
    action = "halve S and skip" if not np.isfinite(gs) else "use S"
    print(f"grad {g:>8.1e} * S={S} -> {gs:>10.3e}  -> {action}")

Read the third row: an unscaled 1e-7 is stored as 1.192e-7 — subnormal, with about 19% relative error. Scaled, the same gradient recovers to 1.000e-7 with essentially no error. In the last row the unscaled value is exactly zero and the scaled one comes back with about 1% error, because it now sits in the normal range and is limited only by fp16’s ten mantissa bits.

So scaling does not restore precision — it moves the value into a range where any precision survives. And the final row is also why mixed-precision training keeps a master fp32 copy of the weights: an update that small cannot be accumulated in fp16 at all.

6. Failure Modes and Fixes

A table of numerical symptoms, their causes and the standard fixes
Figure 47.5: None of these bugs raise an exception.
Habit Why it matters
Never take log of a raw probability Underflow to 0 becomes -inf and then NaN
Never exp without subtracting the max first Overflow to inf spreads through the batch
Prefer a two-pass variance E[x²] − E[x]² can return a negative number
Clip gradients One bad batch cannot destroy the weights
Check for finite values in the loss NaN propagates silently and permanently

7. Worked Examples

7.1 Mixed precision

Compute the forward and backward passes in bf16 for speed,
keep a master copy of the weights in fp32 for the update.
The master copy is what stops small updates from being lost.

7.2 Why the master copy exists

An update of 1e-6 added to a weight of 1.0 is invisible in
bf16 (7 mantissa bits gives ulp ~8e-3 there). In fp32 it is
representable. Update the fp32 copy, then cast down.

7.3 Softmax in the log domain

Cross-entropy from logits directly:
  loss = logsumexp(z) - z[target]
No probability is ever formed, so nothing can underflow.

7.4 NaN hunting

NaN is contagious and silent. Assert finiteness at layer
boundaries during debugging; the first NaN is the cause, and
everything downstream is a symptom.

7.5 Determinism

Floating-point addition is not associative, so parallel
reductions produce different rounding runs to run. Bit-exact
reproducibility requires deterministic kernels and is slower.

8. Verify It in Python

import numpy as np

# --- 1. fp16 range limits -----------------------------------
assert np.float16(1e-8) == 0.0
assert np.isfinite(np.float16(65504.0))
print("fp16: 1e-8 underflows to", np.float16(1e-8), "; 65504 is the max")

# --- 2. bf16 keeps the range (emulated via bit truncation) --
def to_bf16(x):
    x = np.asarray(x, dtype=np.float32)
    bits = x.view(np.uint32) & 0xFFFF0000
    return bits.view(np.float32)

tiny = np.array([1e-30, 1e-8, 1.0, 1e30], dtype=np.float32)
assert np.all(to_bf16(tiny) != 0)
print("bf16-emulated values:", to_bf16(tiny))

# --- 3. naive softmax overflows, stable does not ------------
z = np.array([900.0, 1000.0, 1100.0])
with np.errstate(over="ignore", invalid="ignore"):
    naive = np.exp(z) / np.exp(z).sum()
assert not np.all(np.isfinite(naive))
s = np.exp(z - z.max()); stable = s / s.sum()
assert np.allclose(stable.sum(), 1.0)
assert np.all(np.isfinite(stable))
print("naive ->", naive, " stable ->", np.round(stable, 6))

# --- 4. the shift is exact ----------------------------------
rng = np.random.default_rng(0)
for _ in range(100):
    zz = rng.normal(size=6) * 20
    a = np.exp(zz - zz.max()); a /= a.sum()
    b = np.exp(zz - zz.max() - 3.5); b /= b.sum()
    assert np.allclose(a, b, atol=1e-12)
print("softmax is invariant to any constant shift")

# --- 5. logsumexp stability ---------------------------------
def logsumexp(v):
    m = v.max()
    return m + np.log(np.sum(np.exp(v - m)))

big = np.array([1000.0, 1000.0])
assert np.isclose(logsumexp(big), 1000.0 + np.log(2))
print("logsumexp(1000,1000) =", logsumexp(big), "(no overflow)")

# --- 6. cross-entropy from logits, no probabilities ---------
def ce_from_logits(logits, target):
    return logsumexp(logits) - logits[target]

logits = np.array([2.0, 1.0, 0.5])
p = np.exp(logits - logits.max()); p /= p.sum()
assert np.isclose(ce_from_logits(logits, 0), -np.log(p[0]))
print("CE from logits matches -log(softmax)")

# --- 7. catastrophic cancellation ---------------------------
assert 1e16 + 1 - 1e16 == 0.0
assert (0.1 + 0.2 - 0.3) != 0.0
print("1e16 + 1 - 1e16 =", 1e16 + 1 - 1e16)

# --- 8. two-pass variance is correct -----------------------
x = rng.normal(loc=1e8, scale=1.0, size=200_000)
naive_v = np.mean(x ** 2) - np.mean(x) ** 2
stable_v = np.mean((x - np.mean(x)) ** 2)
assert abs(stable_v - 1.0) < 0.05
print(f"two-pass var {stable_v:.6f} vs naive {naive_v:.6f}")

# --- 9. loss scaling is algebraically a no-op ---------------
g = np.float32(3.0)
S = np.float32(1024.0)
assert np.isclose(float(g * S) / float(S), float(g))
print("(g * S) / S == g")

# --- 10. gradient clipping preserves direction -------------
def clip(grad, c):
    n = np.linalg.norm(grad)
    return grad if n <= c else grad * (c / n)

gv = np.array([30.0, 40.0])      # norm 50
cg = clip(gv, 5.0)
assert np.isclose(np.linalg.norm(cg), 5.0)
assert np.allclose(cg / np.linalg.norm(cg), gv / np.linalg.norm(gv))
print("clipping changes magnitude, never direction")
print("all assertions passed")

9. Practice Questions

  1. Compare the bit layouts of fp32, fp16 and bf16.
  2. Why does fp16 need loss scaling but bf16 usually does not?
  3. Prove that subtracting the max preserves softmax exactly.
  4. Implement log-sum-exp and explain each term.
  5. Give an expression that loses all precision in float64.
  6. Why is E[x²] − E[x]² dangerous?
  7. Show that loss scaling leaves the update unchanged.
  8. Why does mixed precision keep an fp32 master copy?
  9. Compute the loss directly from logits without a probability.
  10. Why is bit-exact reproducibility expensive?

Leave a Reply

Your email address will not be published. Required fields are marked *