Chapter 38: Activation Functions & Nonlinearity

Chapter 38: Activation Functions & Nonlinearity

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 38 of 48

The activation function decides whether gradients survive a deep stack. This chapter derives the saturation problem from the derivatives of sigmoid and tanh, shows exactly how many layers a vanishing gradient can cross, explains the dying-ReLU failure as a zero-gradient trap, and derives the softmax Jacobian that makes attention and classification tractable.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Saturation “Sigmoid is bad” as folklore. Derivative bounded by 1/4 and the product computed.
Depth limit Never quantified. Gradient after 10 layers computed explicitly.
Dying ReLU Mentioned, not demonstrated. A unit shown permanently dead with zero gradient.
Softmax Jacobian Skipped. Derived as diag(p) – p pᵀ and checked numerically.

Core learning objectives. After this chapter you should be able to:

  1. Compute sigmoid and tanh derivatives and their bounds.
  2. Explain saturation as a vanishing-gradient mechanism.
  3. Quantify gradient decay across many layers.
  4. Diagnose a dying ReLU from its update history.
  5. Compare ReLU, leaky ReLU, ELU and GELU.
  6. Derive the softmax Jacobian.

2. Saturation

Sigmoid and tanh curves flattening at both ends, with derivatives peaking then collapsing
Figure 38.1: Both functions flatten outside |z| < 2, and their derivatives collapse with them.
sigmoid(z) = 1 / (1 + e^-z)
sigma'(z)  = sigma(z) (1 - sigma(z))        max 1/4 at z = 0

tanh(z)    = 2 sigma(2z) - 1
tanh'(z)   = 1 - tanh^2(z)                  max 1 at z = 0

At z = 6, sigmoid' is about 0.0025 — a factor of 100
below its peak. The unit has stopped learning.
import numpy as np

sig = lambda z: 1 / (1 + np.exp(-z))
d_sig = lambda z: sig(z) * (1 - sig(z))

print(f"{'z':>5} {'sigma':>10} {'deriv':>12}")
for z in (-6, -3, 0, 3, 6):
    print(f"{z:5.1f} {sig(z):10.6f} {d_sig(z):12.8f}")

print()
print("max sigmoid derivative:", d_sig(0.0))
print("derivative at z=6 relative to peak:", f"{d_sig(6)/d_sig(0):.2%}")
print("tanh max derivative:", 1 - np.tanh(0.0) ** 2)

The derivative at z = 6 is 0.25% of its peak, and at z = 10 it is 0.0045%. Saturation is not gradual degradation — it is a hard ceiling on how much signal reaches the layer below.

3. Gradient Decay Across Depth

Gradient magnitude falling geometrically with sigmoid and holding steady with ReLU
Figure 38.2: Per-layer gain compounds multiplicatively — near 1 keeps depth usable.
Backprop multiplies one Jacobian per layer:

dL/dh_0 = prod_l J_l

If every J_l has gain g, the gradient carries g^n.
g = 0.25, n = 10  -> 9.5e-7   (vanished)
g = 1.00, n = 10  -> 1.0      (healthy)
g = 1.15, n = 10  -> 4.05     (growing)

Saturation sets g well below 1, which is the whole
vanishing-gradient story in one line.
import numpy as np

for g in (0.25, 0.5, 0.9, 1.0, 1.15):
    vals = [g ** n for n in (1, 5, 10, 20, 50)]
    print(f"gain {g:4.2f}: " + "  ".join(f"n={n}:{v:10.3e}" for n, v in zip((1,5,10,20,50), vals)))

print()
# a relu network with unit-scale weights keeps gain near 1
rng = np.random.default_rng(0)
for depth in (10, 50, 100):
    z = rng.normal(size=8)
    for _ in range(depth):
        z = np.maximum(z, 0) * 1.0
    live = int(np.sum(z > 0))
    print(f"depth {depth:3d}: {live} of 8 paths still alive")

The second part is the practical lesson: with gain 1, ReLU paths survive arbitrarily deep. With gain 0.25, ten layers already destroy the signal. Everything called “normalisation” in Chapter 43 exists to pin this gain near 1.

4. The ReLU Family and Its Failure Mode

ReLU, leaky ReLU and ELU curves compared on the negative side
Figure 38.3: ReLU has no upper saturation, but it has a flat zero region below the origin.
relu(z)      = max(0, z)          slope 0 or 1
leaky(z)     = max(alpha z, z)    slope alpha or 1
elu(z)       = z if z>0 else alpha(e^z - 1)
gelu(z)      = z * Phi(z)         smooth, non-monotone near 0

For z < 0: relu' = 0 exactly.
If a unit's pre-activation is negative for every sample,
its gradient is permanently zero and it never recovers.
import numpy as np

relu = lambda z: np.maximum(z, 0)
leaky = lambda z, a=0.1: np.where(z > 0, z, a * z)
elu = lambda z, a=1.0: np.where(z > 0, z, a * (np.exp(np.minimum(z, 0)) - 1))

z = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
for name, f in [("relu", relu), ("leaky", leaky), ("elu", elu)]:
    print(f"{name:6s} {np.round(f(z), 5)}")

print()
# dying relu: force a unit deeply negative, watch the weight never move
w, b, lr = 1.0, -10.0, 0.1
X = np.array([0.5, 1.0, 1.5])
for step in range(6):
    z_pre = w * X + b
    h = relu(z_pre)
    grad_w = np.mean((h - 1.0) * (z_pre > 0) * X)
    w -= lr * grad_w
    if step in (0, 5):
        print(f"step {step}: pre-activations {np.round(z_pre,3)} grad_w={grad_w:.4f} w={w:.4f}")
print("weight never changed — the unit is dead")

The weight does not move because (z_pre > 0) is False for every sample, so the gradient is exactly zero, not small. This is why leakage below zero matters: it keeps the gate from latching shut.

5. The Softmax Jacobian

Five logits converted into a normalised probability distribution across bars
Figure 38.4: Softmax maps arbitrary reals onto the simplex; the bars are the distribution.
p_i = e^{z_i} / sum_j e^{z_j}

dp_i/dz_j = p_i (delta_ij - p_j)

In matrix form:  J = diag(p) - p p^T

Every row sums to zero, because the p's must sum to 1 —
raising one probability must lower another.
import numpy as np

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

z = np.array([2.0, 1.0, 0.1, -0.5])
p = softmax(z)

J = np.diag(p) - np.outer(p, p)
print("softmax:", np.round(p, 5))
print("jacobian:\n", np.round(J, 5))
print("row sums:", np.round(J.sum(axis=1), 12))

# numeric check
eps = 1e-6
Jn = np.zeros((4, 4))
for j in range(4):
    zp, zm = z.copy(), z.copy()
    zp[j] += eps; zm[j] -= eps
    Jn[:, j] = (softmax(zp) - softmax(zm)) / (2 * eps)
print("max analytic-vs-numeric error:", f"{np.abs(J - Jn).max():.2e}")

Each row of the Jacobian sums to zero, which is the algebraic statement that probability mass is conserved. It also means the softmax Jacobian is singular — one direction always carries no information, which is why softmax is never invertible.

6. Choosing an Activation

A comparison table of activation functions with range, saturation, typical use and pitfalls
Figure 38.5: The practical decision table, with the pitfall for each choice.
Rule Reason
Do not use sigmoid in hidden layers of a deep net Derivative capped at 0.25 compounds away
Use sigmoid for binary output and for gates Output range matches a probability; gates need 0–1
Use softmax only when classes are exclusive It forces the outputs to sum to 1
Use ReLU or GELU by default No upper saturation, cheap derivative
Use leaky ReLU if units are dying A nonzero negative slope keeps the gradient alive

7. Worked Examples

7.1 Derivative identity for sigmoid

sigma' = sigma (1 - sigma)
Cheap: reuse the forward value, never recompute exp.

7.2 Numerical stability

Compute sigmoid as 1/(1+exp(-z)) for z>=0 and
exp(z)/(1+exp(z)) for z<0, to avoid exp overflow.

7.3 Softmax temperature

T -> 0 approaches argmax (one-hot).
T -> infinity approaches uniform.

7.4 GELU versus ReLU

GELU is smooth and slightly non-monotone near zero,
which gives a small negative response where ReLU
gives exactly zero.

7.5 What the activation does to the loss surface

Piecewise-linear activations give piecewise-linear
outputs, hence piecewise-quadratic losses and
piecewise-constant Hessians. Smooth activations
give smoother but not necessarily better-conditioned
surfaces.

8. Verify It in Python

import numpy as np

sig = lambda z: 1 / (1 + np.exp(-z))
relu = lambda z: np.maximum(z, 0.0)

# --- 1. sigmoid derivative bound -------------------------------
zs = np.linspace(-20, 20, 40001)
d = sig(zs) * (1 - sig(zs))
assert d.max() <= 0.25 + 1e-12
assert np.isclose(d.max(), 0.25, atol=1e-9)
print(f"max sigmoid derivative = {d.max():.6f} (bound 0.25)")

# --- 2. tanh derivative bound ----------------------------------
dt = 1 - np.tanh(zs) ** 2
assert dt.max() <= 1.0 + 1e-12
print(f"max tanh derivative = {dt.max():.6f} (bound 1.0)")

# --- 3. saturation is severe -----------------------------------
assert (sig(6) * (1 - sig(6))) < 0.003
assert (sig(10) * (1 - sig(10))) < 0.00005
print("sigmoid derivative at z=10 is", f"{(sig(10)*(1-sig(10))):.2e}")

# --- 4. gradient product across depth -------------------------
for n in (10, 30, 50):
    g = 0.25 ** n
    assert g < 1e-6 or n < 10
    print(f"gain 0.25 over {n:2d} layers -> {g:.3e}")

# --- 5. relu derivative is exactly 0 or 1 ---------------------
z = np.array([-3.0, -0.001, 0.0, 0.001, 3.0])
slope = (relu(z + 1e-7) - relu(z - 1e-7)) / 2e-7
print("relu slope:", np.round(slope, 3))
# exact equality fails at z=3 because 3 +/- 1e-7 is not representable
assert slope[0] == 0.0
assert np.isclose(slope[-1], 1.0, atol=1e-6)
assert np.isclose(slope[2], 0.5)      # central difference straddles the kink

# --- 6. leaky relu never has zero slope -----------------------
leaky = lambda z, a=0.1: np.where(z > 0, z, a * z)
sl = (leaky(z + 1e-7) - leaky(z - 1e-7)) / 2e-7
assert np.all(sl > 0)
print("leaky relu slope is never zero:", np.round(sl, 3))

# --- 7. softmax jacobian rows sum to zero ---------------------
def softmax(v):
    e = np.exp(v - v.max())
    return e / e.sum()

v = np.array([1.0, 2.0, -0.5, 0.0])
p = softmax(v)
J = np.diag(p) - np.outer(p, p)
assert np.allclose(J.sum(axis=1), 0, atol=1e-15)
assert np.isclose(p.sum(), 1.0)
print("softmax jacobian row sums =", np.round(J.sum(axis=1), 15))

# --- 8. softmax jacobian matches numeric ---------------------
Jn = np.zeros((4, 4))
for j in range(4):
    vp, vm = v.copy(), v.copy()
    vp[j] += 1e-6; vm[j] -= 1e-6
    Jn[:, j] = (softmax(vp) - softmax(vm)) / 2e-6
assert np.allclose(J, Jn, atol=1e-7)
print("softmax jacobian verified numerically")

# --- 9. softmax is shift invariant -----------------------------
assert np.allclose(softmax(v + 100.0), softmax(v), atol=1e-12)
print("softmax(z + 100) == softmax(z)")

# --- 10. temperature behaviour --------------------------------
cold = softmax(v / 0.01)
hot = softmax(v / 100.0)
assert cold.max() > 0.99
assert hot.max() < 0.30
print(f"T=0.01 max prob {cold.max():.4f}, T=100 max prob {hot.max():.4f}")
print("all assertions passed")

9. Practice Questions

  1. Derive the derivative of the sigmoid function.
  2. State the maximum derivative of sigmoid and of tanh.
  3. Compute the gradient scale after 20 sigmoid layers at gain 0.25.
  4. Why does a ReLU unit die permanently rather than slowly recover?
  5. How does leaky ReLU prevent death?
  6. Derive the softmax Jacobian.
  7. Why is the softmax Jacobian singular?
  8. Explain the effect of temperature on a softmax distribution.
  9. When is softmax the wrong output layer?
  10. Why does GELU keep a small negative response?

Leave a Reply

Your email address will not be published. Required fields are marked *