Chapter 21: The Chain Rule & Backpropagation Mathematics

Chapter 21: The Chain Rule & Backpropagation Mathematics

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 21 of 48

Backpropagation is the chain rule applied to a composition graph, evaluated from the output backwards. That is the whole idea, and it fits in one identity: the gradient passed to a node is its local derivative multiplied by the gradient coming from above. This chapter derives that identity, applies it to a real network layer by layer, implements a miniature autograd engine, and then explains the two failure modes — vanishing and exploding gradients — as arithmetic consequences of multiplying many factors together.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Chain rule Stated for two functions and never extended to a graph. Extended to arbitrary depth, then to a branching graph.
Backprop Described as a “special algorithm”. Shown to be one identity applied at each node, in reverse.
Implementation Deferred entirely to the framework. A working 40-line autograd engine is built and checked.
Pathologies Vanishing gradients asserted. Measured across depth, with the fix for each.

Core learning objectives. After this chapter you should be able to:

  1. Apply the chain rule through a chain of four functions.
  2. State the local-times-upstream identity for gradient flow.
  3. Derive gradients for a two-layer network by hand.
  4. Implement a minimal reverse-mode autograd engine.
  5. Verify analytic gradients against finite differences.
  6. Quantify gradient decay across depth and name the remedies.

2. The Chain Rule Along a Path

A chain x to u to y to loss with the product of derivatives beneath
Figure 21.1: One factor per edge. Depth converts a product into a magnitude problem.
L = l(f(g(x)))

dL/dx = (dL/dl)(dl/df)(df/dg)(dg/dx)

Read right to left: the derivative of the innermost function
comes last. Read as a product: any factor near zero kills
the whole product; any factor above one inflates it.
import numpy as np

g = lambda x: 2 * x + 1
f = lambda u: u ** 2
l = lambda y: np.log(y)

x0 = 1.0
# dL/dx = (1/y) * (2u) * 2   with u = g(x0), y = f(u)
u0, y0 = g(x0), f(g(x0))
dL_dx = (1 / y0) * (2 * u0) * 2
print("analytic:", dL_dx)

composed = lambda x: l(f(g(x)))
h = 1e-7
numeric = (composed(x0 + h) - composed(x0 - h)) / (2 * h)
print("numeric :", round(numeric, 8))
assert np.isclose(dL_dx, numeric, atol=1e-5)

3. Backpropagation Is Reverse Traversal

Forward chain of three hidden layers and a loss, with dashed backward arrows and a memory note
Figure 21.2: The forward pass computes activations; the backward pass walks them in reverse.
A node receiving a local derivative and an upstream gradient, producing the product
Figure 21.3: The entire algorithm. Everything else is bookkeeping.
For a node computing z from its inputs:

  gradient passed to input = (dL/dz) * (dz/dinput)
                             ^upstream   ^local

At the loss, dL/dz = 1 (seed).
At each node, multiply. Where paths merge, add.
import numpy as np

def numerical_grad(fn, p, h=1e-6):
    g = np.zeros_like(p, dtype=float)
    for i in range(p.size):
        e = np.zeros_like(p, dtype=float); e[i] = h
        g[i] = (fn(p + e) - fn(p - e)) / (2 * h)
    return g

def relu(t):
    return np.maximum(0.0, t)

def net(params, x):
    W1, b1, W2, b2 = params
    h = relu(x @ W1 + b1)
    return np.sum(h @ W2 + b2)

rng = np.random.default_rng(0)
W1 = rng.normal(size=(3, 4)) * 0.5
b1 = np.zeros(4)
W2 = rng.normal(size=(4, 1)) * 0.5
b2 = np.zeros(1)
x = rng.normal(size=(2, 3))

def flat(W1, b1, W2, b2):
    return np.concatenate([W1.ravel(), b1, W2.ravel(), b2])

def unflat(p):
    W1 = p[0:12].reshape(3, 4); b1 = p[12:16]
    W2 = p[16:20].reshape(4, 1); b2 = p[20:21]
    return W1, b1, W2, b2

p0 = flat(W1, b1, W2, b2)
g = numerical_grad(lambda p: net(unflat(p), x), p0)
print("finite-difference gradient norm:", round(np.linalg.norm(g), 6))
print("nonzero entries:", int(np.sum(np.abs(g) > 1e-8)))

4. A Minimal Autograd Engine

Reverse-mode differentiation is not complicated machinery. It is a recording of the forward operations, each storing its local derivative, plus a reverse sweep that multiplies. This is a complete implementation:

import numpy as np

class Val:
    """A scalar with a gradient, recording the op that produced it."""
    def __init__(self, data, parents=(), local=()):
        self.data = float(data)
        self.grad = 0.0
        self.parents = parents
        self.local = local          # d(self)/d(parent) for each parent

    def __add__(self, other):
        other = other if isinstance(other, Val) else Val(other)
        return Val(self.data + other.data, (self, other), (1.0, 1.0))

    def __mul__(self, other):
        other = other if isinstance(other, Val) else Val(other)
        return Val(self.data * other.data, (self, other), (other.data, self.data))

    def __pow__(self, n):
        return Val(self.data ** n, (self,), (n * self.data ** (n - 1),))

    def tanh(self):
        t = np.tanh(self.data)
        return Val(t, (self,), (1 - t ** 2,))

    def backward(self):
        order, seen = [], set()
        def topo(v):
            if id(v) in seen:
                return
            seen.add(id(v))
            for p_ in v.parents:
                topo(p_)
            order.append(v)
        topo(self)
        self.grad = 1.0
        for v in reversed(order):
            for p_, d in zip(v.parents, v.local):
                p_.grad += v.grad * d

a, b = Val(2.0), Val(-3.0)
out = a * b + b ** 2 + (a * 3.0).tanh()
out.backward()
print(f"out={out.data:.6f}  da={a.grad:.6f}  db={b.grad:.6f}")

# check against finite differences
def f(a_, b_):
    return a_ * b_ + b_ ** 2 + np.tanh(a_ * 3.0)
h = 1e-6
print("numeric da:", round((f(2+h, -3) - f(2-h, -3)) / (2*h), 6))
print("numeric db:", round((f(2, -3+h) - f(2, -3-h)) / (2*h), 6))

Forty lines, and it reproduces what a framework does. The only additions in production are tensor data instead of scalars, and the topological sort applied to a much larger graph. The identity it implements is exactly the one in Figure 21.3.

5. Gradient Flow Across Depth

Plot of relative gradient magnitude versus depth showing geometric decay and growth
Figure 21.4: Multiplying many factors below one decays the gradient; above one, it explodes.
import numpy as np

for factor, name in ((0.25, "sigmoid"), (0.9, "mild"), (1.1, "slightly >1")):
    print(f"{name:12s} 10 layers={factor**10:.3e}  30 layers={factor**30:.3e}")

# a deep chain with sigmoid saturating
def chain_gradient(depth, sigma):
    return np.prod([sigma] * depth)

print("\nsigmoid at saturation (derivative 0.01):")
for d in (5, 20, 50):
    print(f"  depth {d:2d}: {chain_gradient(d, 0.01):.2e}")
Failure Cause Remedy
Vanishing Each factor < 1 (saturated sigmoid, small weights) Residual paths, ReLU, LayerNorm, ρ ≈ 1 init
Exploding Each factor > 1 (large weights, unstable recurrence) Gradient clipping, spectral normalisation, smaller init
Dead units ReLU stuck at zero → local derivative exactly 0 Leaky ReLU, GELU, lower learning rate

6. Worked Examples

The explicit backward pass for a linear layer with the three gradient formulas and shape requirements
Figure 21.5: The complete backward pass for a linear layer: two matrix products and one reduction. Note that each gradient has the same shape as the parameter it updates.

6.1 Three-link chain

y = (3x + 2)^2,  x = 1
dy/dx = 2(3x+2)·3 = 6(3x+2)
At x=1: 6·5 = 30

6.2 Two-layer network gradient

h = xW1 + b1
y = hW2
L = (y - t)^2

dL/dy = 2(y - t)
dL/dW2 = h^T (dL/dy)
dL/dh = (dL/dy) W2^T
dL/dW1 = x^T (dL/dh)

6.3 Gradient through softmax plus cross-entropy

For one-hot targets, the two Jacobians cancel to:
  dL/dz = p - onehot(y)
A surprisingly clean result, and a common interview question.

6.4 Why a shared weight gets a summed gradient

y1 = w·x1,  y2 = w·x2,  L = y1 + y2
dL/dw = x1 + x2
Two paths through w, so two contributions added.

6.5 Residual gradient

y = x + f(x)
dy/dx = 1 + f'(x)
The identity term guarantees a gradient of at least 1 reaches
earlier layers, regardless of how small f' becomes.

7. Verify It in Python

import numpy as np

def nd(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)

# --- 1. chain rule through four functions -------------------------
g = lambda x: 2 * x + 1
f = lambda u: u ** 2
ell = lambda y: np.log(y)
composed = lambda x: ell(f(g(x)))
x0 = 1.5
analytic = (1 / f(g(x0))) * (2 * g(x0)) * 2
assert np.isclose(analytic, nd(composed, x0), atol=1e-5)
print("chain rule verified:", round(analytic, 6))

# --- 2. linear layer gradients match finite differences -----------
rng = np.random.default_rng(1)
X = rng.normal(size=(4, 3))
W = rng.normal(size=(3, 2))
b = rng.normal(size=2)
t = rng.normal(size=(4, 2))

def loss(W_, b_):
    return np.sum((X @ W_ + b_ - t) ** 2)

dL_dy = 2 * (X @ W + b - t)
dW = X.T @ dL_dy
db = dL_dy.sum(axis=0)

# numeric check on a few entries
for i, j in [(0, 0), (2, 1)]:
    Wp, Wm = W.copy(), W.copy()
    Wp[i, j] += 1e-6; Wm[i, j] -= 1e-6
    num = (loss(Wp, b) - loss(Wm, b)) / (2e-6)
    assert np.isclose(dW[i, j], num, atol=1e-4)
print("dW entries verified numerically")

for j in range(2):
    bp, bm = b.copy(), b.copy()
    bp[j] += 1e-6; bm[j] -= 1e-6
    num = (loss(W, bp) - loss(W, bm)) / (2e-6)
    assert np.isclose(db[j], num, atol=1e-4)
print("db verified numerically")

# --- 3. gradient of softmax + cross-entropy -----------------------
def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

z = rng.normal(size=(3, 5))
labels = np.array([0, 2, 4])
onehot = np.zeros_like(z)
onehot[np.arange(3), labels] = 1.0
grad = softmax(z) - onehot                     # p - y
assert grad.shape == z.shape
assert np.allclose(grad.sum(axis=-1), 0.0)     # each row sums to zero
print("softmax+CE gradient is p - onehot, rows sum to 0")

# --- 4. shared weight gradient is summed -------------------------
def two_path(w):
    w = float(np.asarray(w).ravel()[0])
    return w * 2.0 + w * 5.0
assert np.isclose(float(nd(two_path, np.array([1.0]))), 7.0)
print("shared weight gradient sums both paths: 2 + 5 = 7")

# --- 5. residual preserves gradient magnitude --------------------
seq = lambda x: 0.001 * x            # almost-zero local derivative
res = lambda x: x + 0.001 * x        # with identity path
assert nd(seq, 1.0) < 0.01
assert np.isclose(nd(res, 1.0), 1.001)
print("residual keeps total gradient near 1")

# --- 6. vanishing gradient magnitude across depth ----------------
factors = [0.25] * 20
assert np.prod(factors) < 1e-12
print("20 sigmoid factors:", f"{np.prod(factors):.2e}")

# --- 7. gradient clipping bounds the update ---------------------
gradient = np.array([30.0, 40.0])
norm = np.linalg.norm(gradient)
clipped = gradient * min(1.0, 1.0 / norm)
assert np.isclose(np.linalg.norm(clipped), 1.0)
print("clipped gradient norm:", np.linalg.norm(clipped))
print("all assertions passed")

8. Practice Questions

  1. Apply the chain rule to y = (3x + 2)² and evaluate at x = 1.
  2. State the local-times-upstream identity in your own words.
  3. Why does the backward pass start with dL/dL = 1?
  4. Derive dL/dW1 for a two-layer network with a squared loss.
  5. Show that dL/dz = p − onehot(y) for softmax plus cross-entropy.
  6. Why is the gradient of a twice-used weight a sum of two terms?
  7. Explain why y = x + f(x) bounds gradient decay from below.
  8. Compute 0.25²⁰ and state its implication for a 20-layer sigmoid network.
  9. Name three remedies for exploding gradients.
  10. Why does the topological sort matter in a general graph?

Leave a Reply

Your email address will not be published. Required fields are marked *