Chapter 23: Gradient Descent, Momentum & Adaptive Optimizers

Chapter 23: Gradient Descent, Momentum & Adaptive Optimizers

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 23 of 48

Every optimiser in deep learning is a modification of one rule: step against the gradient. The modifications exist because that rule has three weaknesses — it uses one step size for every coordinate, it ignores the history of previous steps, and it treats all directions as equally trustworthy. This chapter builds momentum, RMSProp, and Adam in sequence, showing exactly which weakness each one addresses and what it costs.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Plain descent Presented as complete, with weaknesses unstated. Its three specific failures named first, so each fix has a target.
Momentum Given as a formula without the averaging interpretation. Framed as an exponential average with effective step η/(1−β).
Adam Defaults quoted, bias correction unexplained. Bias correction derived as the fix for a zero-initialised accumulator.
Batch size Treated as a hyperparameter with no analysis. Variance scaling made explicit, with the diminishing-returns caveat.

Core learning objectives. After this chapter you should be able to:

  1. Write the plain descent update and state its three weaknesses.
  2. Derive momentum’s effective step size along a consistent direction.
  3. Implement Adam including bias correction and explain why correction is needed.
  4. Predict how gradient variance scales with batch size.
  5. Explain what each optimiser state costs in memory.
  6. Choose an optimiser and justify the choice.

2. Plain Gradient Descent

Descent steps along a parabola converging to the minimum with the update rule alongside
Figure 23.1: Plain descent on a well-conditioned quadratic. On a badly scaled surface this zig-zags.
theta <- theta - eta * grad L(theta)

Three weaknesses:
  1. one global eta for every coordinate, whatever its scale
  2. no memory: each step ignores the previous ones
  3. every coordinate gets the same trust, even if its
     gradient is far noisier than its neighbour's
import numpy as np

def gd(grad, theta0, eta, steps):
    theta = np.array(theta0, dtype=float)
    for _ in range(steps):
        theta = theta - eta * grad(theta)
    return theta

# ill-conditioned quadratic: curvature 1 vs 100
A = np.array([[1.0, 0.0], [0.0, 100.0]])
grad = lambda t: A @ t

theta = gd(grad, [1.0, 1.0], eta=0.015, steps=200)
print("final:", np.round(theta, 8))
print("eta ceiling is 2/100 = 0.02; the first axis then moves at", 1 - 0.015 * 1.0, "per step")

With curvature 100 on the second axis and 1 on the first, stability forces η < 0.02, but at that rate the first axis contracts by only 1.5% per step. Reaching an error of 1e-8 on that axis alone takes roughly 1,200 steps. This factor-of-100 mismatch is what momentum and adaptive methods are for.

3. Momentum

Momentum update equations and the effective step size amplification
Figure 23.2: Momentum averages gradients over time, so consistent directions accumulate.
v <- beta * v + grad L
theta <- theta - eta * v

beta = 0 draws plain descent.
beta = 0.9 averages roughly the last 1/(1-beta) = 10 gradients.

Along a consistent direction the velocity reaches grad/(1-beta),
so the effective step is eta/(1-beta) — ten times larger at 0.9.
Across a valley the oscillations cancel, because consecutive
gradients point in opposite directions.
import numpy as np

def momentum(grad, theta0, eta, beta, steps):
    theta = np.array(theta0, dtype=float)
    v = np.zeros_like(theta)
    for _ in range(steps):
        v = beta * v + grad(theta)
        theta = theta - eta * v
    return theta

A = np.array([[1.0, 0.0], [0.0, 100.0]])
grad = lambda t: A @ t

# same eta as before, but momentum reaches the basin much faster
for beta in (0.0, 0.9, 0.99):
    t = momentum(grad, [1.0, 1.0], eta=0.015, beta=beta, steps=200)
    print(f"beta={beta:4.2f}  final norm={np.linalg.norm(t):.3e}")

Momentum does not remove the stability constraint — it lowers it slightly, requiring η < 2(1+β)/λmax. What it does is convert the slow direction into a faster one by accumulating consistent steps, while cancelling the oscillatory ones.

4. Adam

Adam update equations with first and second moments, bias correction, and the final update
Figure 23.3: Adam combines momentum with per-coordinate scaling by gradient magnitude.
m <- b1*m + (1-b1)*g           first moment  (momentum)
v <- b2*v + (1-b2)*g^2         second moment (magnitude)
m_hat = m / (1 - b1^t)         bias correction
v_hat = v / (1 - b2^t)
theta <- theta - eta * m_hat / (sqrt(v_hat) + eps)

Defaults: b1 = 0.9, b2 = 0.999, eps = 1e-8

Bias correction exists for one reason: v starts at zero and is updated by (1−β₂)g², so after one step it equals 0.001·g², not g². Dividing by 1−β₂ᵗ rescales it back. Without correction, the first few steps would be enormous — η/sqrt(0.001) ≈ 31η instead of η.

import numpy as np

def adam(grad, theta0, eta=0.01, b1=0.9, b2=0.999, eps=1e-8, steps=500):
    theta = np.array(theta0, dtype=float)
    m = np.zeros_like(theta); v = np.zeros_like(theta)
    for t in range(1, steps + 1):
        g = grad(theta)
        m = b1 * m + (1 - b1) * g
        v = b2 * v + (1 - b2) * g ** 2
        m_hat = m / (1 - b1 ** t)
        v_hat = v / (1 - b2 ** t)
        theta = theta - eta * m_hat / (np.sqrt(v_hat) + eps)
    return theta

A = np.array([[1.0, 0.0], [0.0, 100.0]])
result = adam(lambda t: A @ t, [1.0, 1.0])
print("adam final:", np.round(result, 10), " norm:", f"{np.linalg.norm(result):.2e}")

# the same eta works on a much worse-conditioned problem
B = np.diag([0.01, 1000.0])
result2 = adam(lambda t: B @ t, [1.0, 1.0], eta=0.01, steps=2000)
print("kappa=100000 case:", np.round(result2, 8))

5. Batch Size and Gradient Noise

Full batch, mini-batch and single sample compared by cost, gradient quality and trade-off
Figure 23.4: The three regimes, and the variance/throughput trade-off between them.
Three optimizer trajectories through a narrow valley: plain descent zig-zags, momentum tracks, Adam converges directly
Figure 23.5: The same valley under three optimisers.
import numpy as np

rng = np.random.default_rng(0)
true_grad = np.array([1.0, -2.0, 0.5])
noise = 1.0

print(f"{'batch':>6} {'std of mean grad':>18}")
for k in (1, 4, 16, 64, 256, 1024):
    samples = true_grad + rng.normal(0, noise, size=(4000, k, 3))
    means = samples.mean(axis=1)
    print(f"{k:6d} {means.std(axis=0).mean():18.4f}")
print("\nvariance falls as 1/k, so std falls as 1/sqrt(k)")

Standard deviation halves when batch size quadruples, which is the diminishing return: going from 256 to 1024 costs 4× the compute for 2× the gradient precision. Beyond a few hundred samples, the wall-clock cost dominates the benefit, which is why mini-batches in the tens-to-hundreds range are standard.

6. Optimiser State and Memory

Optimiser Extra State × Model Size
SGD none 0
Momentum velocity v 1×
Adam m and v 2×
AdamW m and v, decoupled decay 2×
params = 1_000_000_000
for name, mult in (("SGD", 0), ("Momentum", 1), ("Adam", 2)):
    weights = params * 4 / 1e9          # float32
    opt = params * 4 * mult / 1e9
    print(f"{name:10s} weights {weights:5.1f} GB  optimiser {opt:5.1f} GB  total {weights+opt:5.1f} GB")

A billion-parameter model in float32 needs 4 GB for weights and 8 GB more for Adam's state — before activations. This is why mixed precision and 8-bit optimisers exist: the optimiser state, not the weights, is often the dominant memory cost.

7. Worked Examples

7.1 Effective momentum step

beta = 0.9, eta = 0.01, constant gradient g
Steady-state velocity = g/(1-0.9) = 10g
Effective step = 0.01 * 10g = 0.1g    ten times the plain step

7.2 Bias correction at t = 1

v_1 = 0.001 * g^2
1 - b2^1 = 0.001
v_hat = 0.001 g^2 / 0.001 = g^2       exactly right

7.3 Why Adam handles scale

Coordinate A: gradient 100.   m/sqrt(v) ~ 1
Coordinate B: gradient 0.01.  m/sqrt(v) ~ 1
Both get a step of about eta. The ratio of 10,000 disappears.

7.4 AdamW versus Adam

Adam:  theta <- theta - eta*(m_hat/(sqrt(v_hat)+eps) + wd*theta)
AdamW: theta <- theta - eta*m_hat/(sqrt(v_hat)+eps) - eta*wd*theta
In Adam the decay is scaled by the adaptive denominator, so it is
not a uniform pull toward zero. Decoupling fixes that.

7.5 Batch size and steps

Doubling the batch halves the number of steps per epoch but roughly
halves the gradient variance. Practitioners often scale eta with
sqrt(batch) or linearly, depending on the regime.

8. Verify It in Python

import numpy as np

A = np.array([[1.0, 0.0], [0.0, 100.0]])

def gd(theta0, eta, steps=2000):
    t = np.array(theta0, float)
    for _ in range(steps):
        t = t - eta * (A @ t)
    return t

def mom(theta0, eta, beta, steps=500):
    t = np.array(theta0, float); v = np.zeros_like(t)
    for _ in range(steps):
        v = beta * v + A @ t
        t = t - eta * v
    return t

def adam(theta0, eta=0.01, b1=0.9, b2=0.999, eps=1e-8, steps=2000):
    t = np.array(theta0, float)
    m = np.zeros_like(t); v = np.zeros_like(t)
    for k in range(1, steps + 1):
        g = A @ t
        m = b1 * m + (1 - b1) * g
        v = b2 * v + (1 - b2) * g ** 2
        t = t - eta * (m / (1 - b1 ** k)) / (np.sqrt(v / (1 - b2 ** k)) + eps)
    return t

# --- 1. plain descent is limited by the steep axis ---------------
assert np.linalg.norm(gd([1., 1.], 0.015)) < 1e-8
with np.errstate(over="ignore", invalid="ignore"):
    bad = gd([1., 1.], 0.03)               # eta*lambda_max = 3 > 2 -> diverges
diverged = (not np.all(np.isfinite(bad))) or np.linalg.norm(bad) > 1.0
assert diverged
print("plain descent: stable at 0.015, diverges at 0.03")

# --- 2. momentum accelerates the flat direction ------------------
n_plain = np.linalg.norm(gd([1., 1.], 0.015, steps=60))
n_mom = np.linalg.norm(mom([1., 1.], 0.015, 0.9, steps=60))
assert n_mom < n_plain
print(f"after 60 steps: plain={n_plain:.3e}  momentum={n_mom:.3e}")

# --- 3. Adam reaches machine precision --------------------------
adm = adam([1., 1.])
assert np.linalg.norm(adm) < 1e-6
print("adam final norm:", f"{np.linalg.norm(adm):.2e}")

# --- 4. Adam handles extreme conditioning -----------------------
B = np.diag([0.01, 1000.0])
def adam_B(theta0, eta=0.01, steps=3000):
    t = np.array(theta0, float)
    m = np.zeros_like(t); v = np.zeros_like(t)
    b1, b2, eps = 0.9, 0.999, 1e-8
    for k in range(1, steps + 1):
        g = B @ t
        m = b1 * m + (1 - b1) * g
        v = b2 * v + (1 - b2) * g ** 2
        t = t - eta * (m / (1 - b1 ** k)) / (np.sqrt(v / (1 - b2 ** k)) + eps)
    return t
assert np.linalg.norm(adam_B([1., 1.])) < 1e-4
print("adam on kappa=1e5:", np.round(adam_B([1., 1.]), 8))

# --- 5. bias correction is exact at t = 1 -----------------------
g = 3.0
b2 = 0.999
v = (1 - b2) * g ** 2
assert np.isclose(v / (1 - b2), g ** 2)
print("bias correction recovers g^2 exactly at t=1")

# --- 6. gradient noise falls as 1/sqrt(k) -----------------------
rng = np.random.default_rng(0)
stds = {}
for k in (1, 16, 256):
    means = (1.0 + rng.normal(0, 1.0, size=(4000, k))).mean(axis=1)
    stds[k] = means.std()
assert stds[256] < stds[16] < stds[1]
assert np.isclose(stds[256] / stds[1], 1 / np.sqrt(256), rtol=0.15)
print("noise scaling:", {k: round(v, 4) for k, v in stds.items()})

# --- 7. optimizer memory arithmetic ----------------------------
params = 1_000_000_000
weights_gb = params * 4 / 1e9
adam_extra_gb = params * 8 / 1e9
assert np.isclose(weights_gb, 4.0) and np.isclose(adam_extra_gb, 8.0)
print(f"1B params: {weights_gb:.0f} GB weights + {adam_extra_gb:.0f} GB Adam state")
print("all assertions passed")

9. Practice Questions

  1. State the three weaknesses of plain gradient descent.
  2. With β = 0.9, how many past gradients does momentum effectively average?
  3. Show that steady-state momentum step size is η/(1−β) times the gradient.
  4. Why is bias correction necessary in Adam, and when does it stop mattering?
  5. Explain how Adam makes a 10,000× scale difference between coordinates irrelevant.
  6. Compute Adam's memory overhead for a 7-billion-parameter model in float32.
  7. Why does gradient standard deviation halve when batch size quadruples?
  8. What is the practical limit on increasing batch size?
  9. Contrast Adam and AdamW weight decay.
  10. Given λmax = 50, state the largest stable plain-descent learning rate.

Leave a Reply

Your email address will not be published. Required fields are marked *