AI Reading
Quick summary of this article
AI Mathematics Masterclass — Chapter 3 of 48
A neural network is a function. Not metaphorically — literally: a rule that maps every input vector to exactly one output vector, built by composing simpler functions. This chapter takes the function concept apart: what domain and codomain actually constrain, how to read a graph instead of plotting one, which four functions do almost all the work in deep learning, how transformations inside and outside the function differ, and why composition is the structural reason backpropagation looks the way it does.
1. Technical Audit & Core Learning Objectives
Function chapters usually spend their budget on plotting conic sections and never explain why a sigmoid saturates or why an inverse cannot exist for ReLU. This chapter keeps the vocabulary but aims every concept at a modelling decision you will actually make.
| Component | Common Tutorial Deficiency | Engineering Standard Applied Here |
|---|---|---|
| Definition | Domain and codomain listed as vocabulary, with no consequence drawn. | Codomain vs image shown to determine surjectivity, which determines whether a decoder can reach every output. |
| Transformation | Shift/scale rules memorised as “inside is opposite”, with no model attached. | Inside vs outside mapped directly to residual connections and normalisation placement. |
| Activation shapes | Plots shown decoratively; saturation and non-differentiability not connected to training. | Saturation traced to vanishing gradients; the ReLU kink traced to subgradient handling. |
| Composition | Treated as an algebra exercise, disconnected from network architecture. | Composition presented as the architecture itself, forward and backward. |
Core learning objectives. After this chapter you should be able to:
- State the domain, codomain, and image of a given function and decide injectivity and surjectivity.
- Predict the effect of
f(x)+c,f(x+c),cf(x),f(cx), and−f(x)on the graph. - Explain why sigmoid and tanh saturate and what that does to a gradient.
- Read ReLU, Leaky ReLU, GELU, and softmax as piecewise or composed functions.
- Expand a four-layer network as an explicit composition and state the shape at each stage.
- Decide when a function is invertible, and why most activations are not.
2. A Function Is a Rule With a Domain
A function f : X → Y assigns to each element of the domain X exactly one element of the codomain Y. Three sets are in play, and conflating them causes most confusion:
f(x)=x² the codomain is all of ℝ but the image is only [0, ∞). That gap is exactly what “not surjective” means.| Set | Meaning | Neural Network Example |
|---|---|---|
| Domain | Every legal input | ℝ^784 for a flattened 28×28 image |
| Codomain | The declared output type | ℝ^10 for ten class logits |
| Image | Outputs actually produced | After softmax: the probability simplex, not all of ℝ^10 |
The distinction has real consequences. A decoder whose image covers only a submanifold of its codomain cannot produce every declared output; that is a modelling fact, not an implementation bug. Likewise, a function that maps distinct inputs to the same output cannot be inverted — which is exactly the situation for every activation function with a flat region.
3. Reading a Graph
A graph is not a decoration. Six features carry essentially all the information a practitioner needs:
- Where it crosses zero — the roots. For a loss surface, these are global minima.
- Slope — the derivative, and therefore the gradient that drives training.
- Flat regions — zero gradient, which means no learning signal.
- Asymptotes — saturation limits, as in sigmoid approaching 1.
- Symmetry — even, odd, or neither; tanh is odd, sigmoid is not.
- Corners — points where the derivative does not exist.
You rarely plot these deliberately. You recognise them from the equation, which is faster and works in dimensions you cannot draw.
4. The Four Functions That Run Deep Learning
Almost every activation you will meet is a variation on one of four shapes.
| Function | Formula | Derivative | Failure Mode |
|---|---|---|---|
| Sigmoid | 1/(1+e⁻ˣ) |
σ(1−σ), max 0.25 |
Saturates: gradient → 0 at large |x| |
| Tanh | (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ) |
1−tanh², max 1.0 |
Also saturates, but zero-centred |
| ReLU | max(0,x) |
1 if x>0 else 0 |
Dying units: zero gradient for x<0 |
| Softmax | eᶻⁱ/Σeᶻʲ |
Jacobian diag(p)−ppᵀ |
Overflow without max-subtraction |
The saturated gradient deserves emphasis. Since σ′(x) ≤ 0.25, each sigmoid layer multiplies the backward signal by at most a quarter. Stack ten of them and the gradient reaching the first layer is scaled by 0.25¹⁰ ≈ 10⁻⁶. That is the mathematical content of the vanishing-gradient problem, and it is why deep networks moved to ReLU and then to residual connections.
5. Transformations: Inside Versus Outside
The most useful rule in this chapter: modifications inside the function act horizontally and inverted; modifications outside act vertically and directly.
f(2x) compresses while 2f(x) stretches — the inside/outside asymmetry.f(x) + c vertical shift up by c
f(x) − c vertical shift down by c
c · f(x) vertical stretch by c (c > 1) or compression (0 < c < 1)
−f(x) reflection across the x-axis
f(x + c) horizontal shift LEFT by c ← opposite of intuition
f(cx) horizontal compression by 1/c ← opposite of intuition
f(−x) reflection across the y-axis
Why does f(x + c) move left? Because to get the old value at position p, you now need input p − c. The whole graph arrives earlier. This "inside is inverted" rule is precisely why batch normalisation placed before the activation changes the effective function more subtly than one placed after.
In network terms: a residual connection adds x to f(x) — outside, vertical, and critically, it makes the total function x + f(x) approximately identity when f is small. That is what allows gradients to flow through very deep stacks, and it is a transformation trick, not a new kind of layer.
6. Piecewise Functions: ReLU in Detail
ReLU is defined by cases, and every one of its properties follows from that definition.
import numpy as np
def relu(x):
return np.maximum(0.0, x)
def relu_grad(x):
# subgradient: conventionally 0 at exactly 0
return (x > 0).astype(float)
x = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
print("relu :", relu(x)) # [0. 0. 0. 0.5 2.]
print("relu' :", relu_grad(x)) # [0. 0. 0. 1. 1.]
print("continuous at 0:", np.isclose(relu(-1e-9), relu(1e-9), atol=1e-8))
Two consequences follow directly. First, a unit whose pre-activation is always negative receives gradient 0 forever — the dying ReLU. Second, ReLU is not injective: everything on (−∞, 0] maps to 0, so no inverse exists. Leaky ReLU and GELU exist largely to soften these two consequences while keeping the cheap gradient on the positive side.
7. Composition Is the Architecture
Composition applies one function to the output of another. A multi-layer network is nothing more than that, repeated.
import numpy as np
rng = np.random.default_rng(0)
def affine(x, W, b):
return x @ W + b
def relu(x):
return np.maximum(0.0, x)
def softmax(z):
z = z - z.max(axis=-1, keepdims=True) # numeric safety
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
# f3 ∘ f2 ∘ f1 assembled as an explicit composition
W1 = rng.normal(0, 0.05, (784, 256)); b1 = np.zeros(256)
W2 = rng.normal(0, 0.05, (256, 64)); b2 = np.zeros(64)
W3 = rng.normal(0, 0.05, (64, 10)); b3 = np.zeros(10)
def network(x):
h1 = relu(affine(x, W1, b1)) # (B,784) -> (B,256)
h2 = relu(affine(h1, W2, b2)) # (B,256) -> (B,64)
return softmax(affine(h2, W3, b3)) # (B,64) -> (B,10)
X = rng.normal(size=(8, 784))
out = network(X)
print(out.shape) # (8, 10)
print(out.sum(axis=1)) # [1. 1. 1. ...] each row is a distribution
Because the function is a composition, its derivative is a product of derivatives — the chain rule. Backpropagation simply evaluates that product from right to left, reusing intermediate results. Chapter 21 proves it; here the point is only structural: composition is why a network can express a complicated function while remaining differentiable.
8. Inverses and Invertibility
A function has an inverse only if it is bijective — both injective and surjective onto its codomain. Restricting the domain can create bijectivity from a non-injective function: x² on [0, ∞) is invertible, and its inverse is √x.
| Function | Invertible? | Reason |
|---|---|---|
y = 3x + 2 |
Yes | Strictly monotonic, bijective on ℝ |
y = x² |
No (on ℝ) | Not injective: f(−2)=f(2) |
y = eˣ |
Yes | Inverse ln x on (0, ∞) |
y = ReLU(x) |
No | Infinitely many inputs map to 0 |
This is why normalising flows must use strictly monotonic activations (typically a coupling of exp or a spline), and why reversible architectures such as RevNets must preserve information through additive coupling rather than multiplicative gating.
9. Worked Examples
9.1 Identify domain and image
For f(x) = ln(x − 3): the logarithm requires a positive argument, so x − 3 > 0, giving domain (3, ∞). The logarithm's image is all of ℝ, so the image is ℝ. The function is injective and surjective onto ℝ — invertible, with f⁻¹(y) = eʸ + 3.
9.2 Compose and evaluate
With f(x) = 2x + 1 and g(x) = x², compute f(g(3)) and g(f(3)).
f(g(3)) = f(9) = 2(9) + 1 = 19
g(f(3)) = g(7) = 49 ≠ 19 → composition is not commutative
9.3 Track shapes through composition
A network takes x ∈ ℝ^(32×768) through W₁ ∈ ℝ^(768×256), then W₂ ∈ ℝ^(256×10). State the final shape.
(32, 768) @ (768, 256) → (32, 256)
(32, 256) @ (256, 10) → (32, 10)
Batch dimension 32 unchanged throughout; only the feature axis moves.
9.4 Estimate a saturated gradient
Six stacked sigmoid layers each contribute at most 0.25 to the backward product. What is the upper bound on the gradient scale reaching the first layer?
0.25⁶ = 2.44 × 10⁻⁴
Even in the best case, a signal is attenuated by four orders of magnitude.
In saturation (|x| large) the derivative is nearer 10⁻² each, so 10⁻¹² overall.
9.5 Decide invertibility from a graph
A curve that rises, plateaus, then rises again is not injective — two distinct inputs share an output on the plateau. Pass the horizontal line test: a function is injective on an interval exactly when every horizontal line meets the graph at most once.
10. Verify It in Python
import numpy as np
# --- 1. Composition is not commutative ----------------------------
f = lambda x: 2 * x + 1
g = lambda x: x ** 2
assert f(g(3)) == 19 and g(f(3)) == 49 and f(g(3)) != g(f(3))
print("f(g(3)) =", f(g(3)), " g(f(3)) =", g(f(3)))
# --- 2. ReLU is not injective -------------------------------------
relu = lambda x: np.maximum(0.0, x)
assert relu(-5.0) == relu(-0.1) == 0.0
print("relu(-5) == relu(-0.1) == 0 -> not injective")
# --- 3. Sigmoid saturation and gradient bound ---------------------
sig = lambda x: 1.0 / (1.0 + np.exp(-x))
sig_p = lambda x: sig(x) * (1 - sig(x))
for z in (0.0, 5.0, 20.0):
print(f"z={z:5.1f} sigma={sig(z):.6f} sigma'={sig_p(z):.3e}")
assert sig_p(0.0) == 0.25 # maximum slope exactly 0.25
assert sig_p(20.0) < 1e-8 # effectively zero when saturated
# --- 4. Vanishing gradient across stacked sigmoids -----------------
scale = 0.25 ** 6
print("6-layer sigmoid upper bound on backward scale:", scale)
assert scale < 1e-3
# --- 5. Shape propagation through the composition ------------------
rng = np.random.default_rng(0)
X = rng.normal(size=(32, 768))
W1 = rng.normal(0, 0.05, (768, 256))
W2 = rng.normal(0, 0.05, (256, 10))
h = relu(X @ W1)
out = h @ W2
assert h.shape == (32, 256) and out.shape == (32, 10)
print("shapes:", h.shape, "->", out.shape)
# --- 6. Inverse of a bijection round-trips -------------------------
fwd = lambda x: 3 * x + 2
inv = lambda y: (y - 2) / 3
x0 = np.array([-4.0, 0.0, 7.5])
assert np.allclose(inv(fwd(x0)), x0)
print("inverse round-trip exact:", inv(fwd(x0)))
print("all assertions passed")
11. Practice Questions
- Give the domain and image of
f(x) = √(4 − x)and state whether it is injective. - Sketch-by-description: how does the graph of
f(x − 2) + 3relate tof(x)? - Why is
f(2x)a compression while2f(x)is a stretch? Answer in one sentence. - Compute
σ′(0)and explain why no sigmoid layer can ever pass a gradient larger than 0.25×. - Show that tanh is odd while sigmoid is not, and explain one practical consequence for training.
- Write Leaky ReLU with slope
0.01and state its derivative on each branch. - Why can a residual connection
x + f(x)be described as "outside the function"? - A 4-layer network has shapes
784 → 512 → 128 → 10. State all four weight matrix shapes, with biases. - Prove by counterexample that ReLU has no inverse on ℝ.
- Given
κ-style thinking, what happens to a gradient multiplied by 0.25 in each of 20 layers?
12. Official References & Further Reading
Continue the Course
- Course Index: AI Mathematics Masterclass — All 48 Chapters
- Previous Chapter: Chapter 2: Linear Equations, Systems & Solving for Unknowns
- Next Chapter: Chapter 4: Exponents, Logarithms & the Information-Theoretic View
- Related Series: PyTorch Engineering Masterclass — All 16 Modules
