AI Reading
Quick summary of this article
AI Mathematics Masterclass — Chapter 38 of 48
The activation function decides whether gradients survive a deep stack. This chapter derives the saturation problem from the derivatives of sigmoid and tanh, shows exactly how many layers a vanishing gradient can cross, explains the dying-ReLU failure as a zero-gradient trap, and derives the softmax Jacobian that makes attention and classification tractable.
1. Technical Audit & Core Learning Objectives
| Component | Common Tutorial Deficiency | Engineering Standard Applied Here |
|---|---|---|
| Saturation | “Sigmoid is bad” as folklore. | Derivative bounded by 1/4 and the product computed. |
| Depth limit | Never quantified. | Gradient after 10 layers computed explicitly. |
| Dying ReLU | Mentioned, not demonstrated. | A unit shown permanently dead with zero gradient. |
| Softmax Jacobian | Skipped. | Derived as diag(p) – p pᵀ and checked numerically. |
Core learning objectives. After this chapter you should be able to:
- Compute sigmoid and tanh derivatives and their bounds.
- Explain saturation as a vanishing-gradient mechanism.
- Quantify gradient decay across many layers.
- Diagnose a dying ReLU from its update history.
- Compare ReLU, leaky ReLU, ELU and GELU.
- Derive the softmax Jacobian.
2. Saturation
sigmoid(z) = 1 / (1 + e^-z)
sigma'(z) = sigma(z) (1 - sigma(z)) max 1/4 at z = 0
tanh(z) = 2 sigma(2z) - 1
tanh'(z) = 1 - tanh^2(z) max 1 at z = 0
At z = 6, sigmoid' is about 0.0025 — a factor of 100
below its peak. The unit has stopped learning.
import numpy as np
sig = lambda z: 1 / (1 + np.exp(-z))
d_sig = lambda z: sig(z) * (1 - sig(z))
print(f"{'z':>5} {'sigma':>10} {'deriv':>12}")
for z in (-6, -3, 0, 3, 6):
print(f"{z:5.1f} {sig(z):10.6f} {d_sig(z):12.8f}")
print()
print("max sigmoid derivative:", d_sig(0.0))
print("derivative at z=6 relative to peak:", f"{d_sig(6)/d_sig(0):.2%}")
print("tanh max derivative:", 1 - np.tanh(0.0) ** 2)
The derivative at z = 6 is 0.25% of its peak, and at z = 10 it is 0.0045%. Saturation is not gradual degradation — it is a hard ceiling on how much signal reaches the layer below.
3. Gradient Decay Across Depth
Backprop multiplies one Jacobian per layer:
dL/dh_0 = prod_l J_l
If every J_l has gain g, the gradient carries g^n.
g = 0.25, n = 10 -> 9.5e-7 (vanished)
g = 1.00, n = 10 -> 1.0 (healthy)
g = 1.15, n = 10 -> 4.05 (growing)
Saturation sets g well below 1, which is the whole
vanishing-gradient story in one line.
import numpy as np
for g in (0.25, 0.5, 0.9, 1.0, 1.15):
vals = [g ** n for n in (1, 5, 10, 20, 50)]
print(f"gain {g:4.2f}: " + " ".join(f"n={n}:{v:10.3e}" for n, v in zip((1,5,10,20,50), vals)))
print()
# a relu network with unit-scale weights keeps gain near 1
rng = np.random.default_rng(0)
for depth in (10, 50, 100):
z = rng.normal(size=8)
for _ in range(depth):
z = np.maximum(z, 0) * 1.0
live = int(np.sum(z > 0))
print(f"depth {depth:3d}: {live} of 8 paths still alive")
The second part is the practical lesson: with gain 1, ReLU paths survive arbitrarily deep. With gain 0.25, ten layers already destroy the signal. Everything called “normalisation” in Chapter 43 exists to pin this gain near 1.
4. The ReLU Family and Its Failure Mode
relu(z) = max(0, z) slope 0 or 1
leaky(z) = max(alpha z, z) slope alpha or 1
elu(z) = z if z>0 else alpha(e^z - 1)
gelu(z) = z * Phi(z) smooth, non-monotone near 0
For z < 0: relu' = 0 exactly.
If a unit's pre-activation is negative for every sample,
its gradient is permanently zero and it never recovers.
import numpy as np
relu = lambda z: np.maximum(z, 0)
leaky = lambda z, a=0.1: np.where(z > 0, z, a * z)
elu = lambda z, a=1.0: np.where(z > 0, z, a * (np.exp(np.minimum(z, 0)) - 1))
z = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
for name, f in [("relu", relu), ("leaky", leaky), ("elu", elu)]:
print(f"{name:6s} {np.round(f(z), 5)}")
print()
# dying relu: force a unit deeply negative, watch the weight never move
w, b, lr = 1.0, -10.0, 0.1
X = np.array([0.5, 1.0, 1.5])
for step in range(6):
z_pre = w * X + b
h = relu(z_pre)
grad_w = np.mean((h - 1.0) * (z_pre > 0) * X)
w -= lr * grad_w
if step in (0, 5):
print(f"step {step}: pre-activations {np.round(z_pre,3)} grad_w={grad_w:.4f} w={w:.4f}")
print("weight never changed — the unit is dead")
The weight does not move because (z_pre > 0) is False for every sample, so the gradient is exactly zero, not small. This is why leakage below zero matters: it keeps the gate from latching shut.
5. The Softmax Jacobian
p_i = e^{z_i} / sum_j e^{z_j}
dp_i/dz_j = p_i (delta_ij - p_j)
In matrix form: J = diag(p) - p p^T
Every row sums to zero, because the p's must sum to 1 —
raising one probability must lower another.
import numpy as np
def softmax(z):
e = np.exp(z - z.max())
return e / e.sum()
z = np.array([2.0, 1.0, 0.1, -0.5])
p = softmax(z)
J = np.diag(p) - np.outer(p, p)
print("softmax:", np.round(p, 5))
print("jacobian:\n", np.round(J, 5))
print("row sums:", np.round(J.sum(axis=1), 12))
# numeric check
eps = 1e-6
Jn = np.zeros((4, 4))
for j in range(4):
zp, zm = z.copy(), z.copy()
zp[j] += eps; zm[j] -= eps
Jn[:, j] = (softmax(zp) - softmax(zm)) / (2 * eps)
print("max analytic-vs-numeric error:", f"{np.abs(J - Jn).max():.2e}")
Each row of the Jacobian sums to zero, which is the algebraic statement that probability mass is conserved. It also means the softmax Jacobian is singular — one direction always carries no information, which is why softmax is never invertible.
6. Choosing an Activation
| Rule | Reason |
|---|---|
| Do not use sigmoid in hidden layers of a deep net | Derivative capped at 0.25 compounds away |
| Use sigmoid for binary output and for gates | Output range matches a probability; gates need 0–1 |
| Use softmax only when classes are exclusive | It forces the outputs to sum to 1 |
| Use ReLU or GELU by default | No upper saturation, cheap derivative |
| Use leaky ReLU if units are dying | A nonzero negative slope keeps the gradient alive |
7. Worked Examples
7.1 Derivative identity for sigmoid
sigma' = sigma (1 - sigma)
Cheap: reuse the forward value, never recompute exp.
7.2 Numerical stability
Compute sigmoid as 1/(1+exp(-z)) for z>=0 and
exp(z)/(1+exp(z)) for z<0, to avoid exp overflow.
7.3 Softmax temperature
T -> 0 approaches argmax (one-hot).
T -> infinity approaches uniform.
7.4 GELU versus ReLU
GELU is smooth and slightly non-monotone near zero,
which gives a small negative response where ReLU
gives exactly zero.
7.5 What the activation does to the loss surface
Piecewise-linear activations give piecewise-linear
outputs, hence piecewise-quadratic losses and
piecewise-constant Hessians. Smooth activations
give smoother but not necessarily better-conditioned
surfaces.
8. Verify It in Python
import numpy as np
sig = lambda z: 1 / (1 + np.exp(-z))
relu = lambda z: np.maximum(z, 0.0)
# --- 1. sigmoid derivative bound -------------------------------
zs = np.linspace(-20, 20, 40001)
d = sig(zs) * (1 - sig(zs))
assert d.max() <= 0.25 + 1e-12
assert np.isclose(d.max(), 0.25, atol=1e-9)
print(f"max sigmoid derivative = {d.max():.6f} (bound 0.25)")
# --- 2. tanh derivative bound ----------------------------------
dt = 1 - np.tanh(zs) ** 2
assert dt.max() <= 1.0 + 1e-12
print(f"max tanh derivative = {dt.max():.6f} (bound 1.0)")
# --- 3. saturation is severe -----------------------------------
assert (sig(6) * (1 - sig(6))) < 0.003
assert (sig(10) * (1 - sig(10))) < 0.00005
print("sigmoid derivative at z=10 is", f"{(sig(10)*(1-sig(10))):.2e}")
# --- 4. gradient product across depth -------------------------
for n in (10, 30, 50):
g = 0.25 ** n
assert g < 1e-6 or n < 10
print(f"gain 0.25 over {n:2d} layers -> {g:.3e}")
# --- 5. relu derivative is exactly 0 or 1 ---------------------
z = np.array([-3.0, -0.001, 0.0, 0.001, 3.0])
slope = (relu(z + 1e-7) - relu(z - 1e-7)) / 2e-7
print("relu slope:", np.round(slope, 3))
# exact equality fails at z=3 because 3 +/- 1e-7 is not representable
assert slope[0] == 0.0
assert np.isclose(slope[-1], 1.0, atol=1e-6)
assert np.isclose(slope[2], 0.5) # central difference straddles the kink
# --- 6. leaky relu never has zero slope -----------------------
leaky = lambda z, a=0.1: np.where(z > 0, z, a * z)
sl = (leaky(z + 1e-7) - leaky(z - 1e-7)) / 2e-7
assert np.all(sl > 0)
print("leaky relu slope is never zero:", np.round(sl, 3))
# --- 7. softmax jacobian rows sum to zero ---------------------
def softmax(v):
e = np.exp(v - v.max())
return e / e.sum()
v = np.array([1.0, 2.0, -0.5, 0.0])
p = softmax(v)
J = np.diag(p) - np.outer(p, p)
assert np.allclose(J.sum(axis=1), 0, atol=1e-15)
assert np.isclose(p.sum(), 1.0)
print("softmax jacobian row sums =", np.round(J.sum(axis=1), 15))
# --- 8. softmax jacobian matches numeric ---------------------
Jn = np.zeros((4, 4))
for j in range(4):
vp, vm = v.copy(), v.copy()
vp[j] += 1e-6; vm[j] -= 1e-6
Jn[:, j] = (softmax(vp) - softmax(vm)) / 2e-6
assert np.allclose(J, Jn, atol=1e-7)
print("softmax jacobian verified numerically")
# --- 9. softmax is shift invariant -----------------------------
assert np.allclose(softmax(v + 100.0), softmax(v), atol=1e-12)
print("softmax(z + 100) == softmax(z)")
# --- 10. temperature behaviour --------------------------------
cold = softmax(v / 0.01)
hot = softmax(v / 100.0)
assert cold.max() > 0.99
assert hot.max() < 0.30
print(f"T=0.01 max prob {cold.max():.4f}, T=100 max prob {hot.max():.4f}")
print("all assertions passed")
9. Practice Questions
- Derive the derivative of the sigmoid function.
- State the maximum derivative of sigmoid and of tanh.
- Compute the gradient scale after 20 sigmoid layers at gain 0.25.
- Why does a ReLU unit die permanently rather than slowly recover?
- How does leaky ReLU prevent death?
- Derive the softmax Jacobian.
- Why is the softmax Jacobian singular?
- Explain the effect of temperature on a softmax distribution.
- When is softmax the wrong output layer?
- Why does GELU keep a small negative response?
10. Official References & Further Reading
Continue the Course
- Course Index: AI Mathematics Masterclass — All 48 Chapters
- Previous Chapter: Chapter 37: Neural Network Forward Pass Mathematics
- Next Chapter: Chapter 39: Convolution & Pooling Mathematics
- Related Series: PyTorch Engineering Masterclass — All 16 Modules
