Chapter 36: Dimensionality Reduction Mathematics & PCA

Chapter 36: Dimensionality Reduction Mathematics & PCA

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 36 of 48

Principal Component Analysis is eigen-decomposition applied to a covariance matrix, and that single fact explains every one of its properties: why components are orthogonal, why variance equals the eigenvalues, why the reconstruction error equals the discarded eigenvalues, and why standardisation changes the answer. This chapter derives all of it and connects PCA to the SVD that actually computes it.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Derivation Components handed down as eigenvectors. Derived by maximising projected variance under a unit norm.
Reconstruction Claimed optimal, never proved. Error shown to equal the sum of discarded eigenvalues.
Standardisation Treated as an optional one-liner. Demonstrated to change the result entirely.
Computation Eigen-decomposition used directly. SVD explained as the numerically stable route.

Core learning objectives. After this chapter you should be able to:

  1. Derive PCA as variance maximisation under a unit-norm constraint.
  2. Explain why the eigenvalues are the component variances.
  3. Prove the reconstruction error equals the discarded variance.
  4. Compute explained variance ratios and choose a rank.
  5. Explain why PCA must be done on centred data.
  6. Show why SVD is preferred over eigen-decomposition.

2. Variance Maximisation

A tilted point cloud with the principal axis along its long direction and the second axis perpendicular
Figure 36.1: PCA finds the direction along which the data varies most.
Maximise   Var(Xw) = w^T C w
subject to ||w|| = 1

Lagrangian:  w^T C w - lambda (w^T w - 1)
d/dw:        2 C w - 2 lambda w = 0
Hence:       C w = lambda w

The maximisers are the eigenvectors of C, and the
achieved variance is the eigenvalue.
import numpy as np

rng = np.random.default_rng(0)
n = 300
base = rng.normal(size=(n, 2))
transform = np.array([[2.0, 1.2], [0.0, 0.4]])
X = base @ transform.T

Xc = X - X.mean(axis=0)
C = Xc.T @ Xc / n          # population convention throughout this chapter
w, V = np.linalg.eigh(C)
order = np.argsort(w)[::-1]
w, V = w[order], V[:, order]

print("eigenvalues (variances):", np.round(w, 4))
print("eigenvectors:\n", np.round(V, 4))
print("orthonormal:", np.allclose(V.T @ V, np.eye(2)))

ratio = w / w.sum()
print("explained variance ratio:", np.round(ratio, 4))

# brute-force search confirms the first eigenvector maximises variance
best, best_v = -np.inf, None
for t in np.linspace(0, np.pi, 20001):
    u = np.array([np.cos(t), np.sin(t)])
    v = u @ C @ u
    if v > best:
        best, best_v = v, u
print(f"grid max variance {best:.4f} at {np.round(best_v, 4)}")

The grid search over 20,001 directions lands on the leading eigenvector, confirming the derivation rather than merely asserting it.

3. Reconstruction Error

A point projected onto the principal axis with the residual perpendicular to it
Figure 36.2: Cumulative explained variance against the number of components — the curve that chooses k.
Project onto top k eigenvectors:  Z = Xc V_k
Reconstruct:                       X_hat = Z V_k^T

Mean squared error = sum of the DISCARDED eigenvalues
                    = sum_{j>k} lambda_j

So keeping k components and minimising squared
reconstruction error are the same objective.
import numpy as np

rng = np.random.default_rng(1)
n, d = 400, 6
X = rng.normal(size=(n, d)) @ np.diag([5.0, 3.0, 1.0, 0.4, 0.1, 0.05])
Xc = X - X.mean(axis=0)
C = Xc.T @ Xc / n          # population convention throughout this chapter
w, V = np.linalg.eigh(C)
order = np.argsort(w)[::-1]
w, V = w[order], V[:, order]

for k in range(1, d + 1):
    Vk = V[:, :k]
    Xr = (Xc @ Vk) @ Vk.T
    mse = np.mean(np.sum((Xc - Xr) ** 2, axis=1))
    discarded = w[k:].sum()
    print(f"k={k}  mse={mse:8.4f}  discarded eigen={discarded:8.4f}")

Each row’s mse equals its discarded eigen to four decimals. This identity is what licenses the “explained variance ratio” heuristic: it is not a rule of thumb, it is the exact fraction of squared error removed.

One caveat worth knowing: the equality is exact only when the covariance and the error use the same normalisation. With the population convention (divide by n) both sides are Σj>kλj. If you compute eigenvalues with n−1 but measure error as a mean over n samples, every entry here shifts by exactly (n−1)/n — a 0.5% discrepancy at n = 200 that silently fails an equality assertion. This chapter uses n throughout for that reason.

4. Why Centre — and Why Scale, and Which Method

A comparison table of reduction methods, what each captures, and when to use it
Figure 36.3: Preprocessing and method choice. PCA assumes a linear, globally-varying structure; the alternatives relax that.
Without centring, the second-moment matrix E[xx^T] includes
the mean: E[xx^T] = Cov + mu mu^T.

The dominant term becomes mu mu^T, a rank-one matrix
pointing along the mean. Your first "principal component"
then measures where the data sits, not how it varies.
import numpy as np

rng = np.random.default_rng(2)
pts = rng.normal(scale=0.3, size=(200, 2)) + np.array([10.0, 0.0])

def lead(mat):
    w, V = np.linalg.eigh(mat)
    return V[:, np.argmax(w)]

uncentred = lead(pts.T @ pts / len(pts))
centred = lead((pts - pts.mean(0)).T @ (pts - pts.mean(0)) / len(pts))
mean_dir = pts.mean(0) / np.linalg.norm(pts.mean(0))

print("uncentred leading direction:", np.round(uncentred, 3))
print("mean direction            :", np.round(mean_dir, 3))
print("centred leading direction :", np.round(centred, 3))
print("uncentred aligns with mean:", np.allclose(np.abs(uncentred), np.abs(mean_dir), atol=0.05))

The uncentred leading direction coincides with the mean vector. Any pipeline that forgets to centre will report a component that simply encodes where the data lives — a mistake that survives every downstream visualisation check.

Preprocessing is only half the decision. Figure 36.3 also places PCA against the alternatives: truncated SVD for sparse inputs, t-SNE and UMAP for visualisation, autoencoders for genuinely nonlinear manifolds. PCA is the right default because it is fast, deterministic, and gives an explicit transform you can apply to unseen data — properties that t-SNE in particular does not offer.

5. Standardisation Changes Everything

Two loadings plots showing the dominant feature switching when features are standardised
Figure 36.4: Rescaling a feature reorders the components, because variance is unit-dependent.
Variance carries the units of the feature squared.
A column measured in millimetres has 1e6 times the
variance of the same column measured in metres.

Therefore: standardise when features have different units
or arbitrary scales. Skip it only when all features are
commensurable and you deliberately want the loudest one.
import numpy as np

rng = np.random.default_rng(3)
n = 500
age = rng.normal(40, 8, n)          # years
income = rng.normal(50000, 9000, n) # currency
X = np.column_stack([age, income])

def lead_loading(M):
    Mc = M - M.mean(0)
    C = Mc.T @ Mc / len(M)
    w, V = np.linalg.eigh(C)
    return V[:, np.argmax(w)] / np.sign(V[0, np.argmax(w)])

raw = lead_loading(X)
std = lead_loading(X / X.std(0))
print("raw loadings      :", np.round(raw, 4))
print("standardised      :", np.round(std, 4))
print("ratio of variances:", f"{X[:,1].var()/X[:,0].var():.0f}x")

On raw data the income column dominates purely because its variance is ~100,000× larger. Standardising reduces both to unit variance and suddenly both features contribute. Neither answer is wrong — but you must choose deliberately rather than by accident.

6. PCA via SVD

PCA drawn as a linear autoencoder with a narrow bottleneck between encoder and decoder
Figure 36.5: PCA is the linear autoencoder — encode to k numbers, decode to the same dimension.
Centre X, then take its SVD:   Xc = U S V^T

Right singular vectors V  = principal directions
Singular values squared S^2 / (n-1) = eigenvalues of C
Scores               U S  = projected coordinates

SVD never forms X^T X, so it never squares the condition
number. That is why every serious library uses it.
import numpy as np

rng = np.random.default_rng(4)
X = rng.normal(size=(120, 5)) @ np.diag([4., 2., 1., 0.5, 0.1])
Xc = X - X.mean(0)
n = len(X)

C = Xc.T @ Xc / n
w_eig, V_eig = np.linalg.eigh(C)
order = np.argsort(w_eig)[::-1]
w_eig, V_eig = w_eig[order], V_eig[:, order]

U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
w_svd = S ** 2 / n

print("eigen :", np.round(w_eig, 5))
print("svd   :", np.round(w_svd, 5))
print("max difference:", f"{np.abs(w_eig - w_svd).max():.2e}")

# sign of eigenvectors is arbitrary; compare directions up to sign
dots = np.abs(np.sum(V_eig * Vt.T, axis=0))
print("directions match (up to sign):", np.round(dots, 6))

Both routes give identical eigenvalues; only the arbitrary sign of each eigenvector can differ. In production, use the SVD route: forming XᵀX squares the condition number, so a matrix with κ = 10⁸ becomes one with κ = 10¹⁶ and precision is lost exactly where PCA is supposed to be most informative.

7. Worked Examples

7.1 Explained variance ratio

ratio_j = lambda_j / sum_i lambda_i
Cumulative ratio tells you the rank needed for a target fidelity.

7.2 Whitening

Z = Xc V S^-1 makes the covariance the identity.
All directions then have equal variance by construction.

7.3 PCA is not feature selection

Components are linear combinations, not columns.
To keep original features, use variance thresholding instead.

7.4 PCA is linear and unsupervised

It cannot capture a nonlinear manifold, and it ignores labels
entirely — a high-variance direction may be pure nuisance.

7.5 Cost

Full SVD is O(min(n,d) n d). For d >> n use the n x n
Gram matrix instead, or a randomised truncated SVD.

8. Verify It in Python

import numpy as np

rng = np.random.default_rng(0)
n, d = 200, 5
X = rng.normal(size=(n, d)) @ np.diag([3., 2., 1., 0.3, 0.05])
Xc = X - X.mean(0)
C = Xc.T @ Xc / n          # population convention throughout this chapter
w, V = np.linalg.eigh(C)
order = np.argsort(w)[::-1]
w, V = w[order], V[:, order]

# --- 1. eigenvectors are orthonormal ---------------------------
assert np.allclose(V.T @ V, np.eye(d), atol=1e-10)
print("eigenvectors orthonormal")

# --- 2. eigenvalues are decreasing and non-negative ------------
assert np.all(w >= 0)
assert np.all(np.diff(w) <= 0)
print("eigenvalues descending, non-negative")

# --- 3. eigenvalue equals projected variance -------------------
for j in range(d):
    proj = Xc @ V[:, j]
    assert np.isclose(proj.var(ddof=0), w[j], rtol=1e-9)
print("eigenvalue == projected variance for every component")

# --- 4. first component maximises variance ---------------------
best = max(np.linspace(0, 2*np.pi, 72001), key=lambda t: np.array([np.cos(t), np.sin(t)]) @ C[:2, :2] @ np.array([np.cos(t), np.sin(t)]))
grid_best = np.array([np.cos(best), np.sin(best)]) @ C[:2, :2] @ np.array([np.cos(best), np.sin(best)])
assert grid_best <= w[0] + 1e-6
print("no direction exceeds the leading eigenvalue")

# --- 5. explained variance sums to one -------------------------
ratio = w / w.sum()
assert np.isclose(ratio.sum(), 1.0)
print("explained variance ratios sum to 1:", np.round(ratio, 4))

# --- 6. reconstruction error equals discarded eigenvalues ------
for k in (1, 2, 3, 4):
    Vk = V[:, :k]
    Xr = (Xc @ Vk) @ Vk.T
    mse = np.mean(np.sum((Xc - Xr) ** 2, axis=1))
    assert np.isclose(mse, w[k:].sum(), rtol=1e-9)
print("reconstruction error == sum of discarded eigenvalues")

# --- 7. SVD agrees ---------------------------------------------
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
assert np.allclose(S ** 2 / n, w, rtol=1e-9)
print("SVD singular values squared match eigenvalues")

# --- 8. centring is required -----------------------------------
raw_lead = np.linalg.eigh(X.T @ X / n)[1][:, -1]
print("centred and uncentred leading directions differ as expected:",
      not np.allclose(np.abs(raw_lead), np.abs(V[:, 0]), atol=1e-3))

# --- 9. whitenng yields identity covariance --------------------
W = V / np.sqrt(w)
Zw = Xc @ W
pop_cov = Zw.T @ Zw / n           # ddof=0, matching the convention above
assert np.allclose(pop_cov, np.eye(d), atol=1e-10)
print("whitened covariance is the identity")
print("all assertions passed")

9. Practice Questions

  1. Derive PCA by maximising projected variance under a unit-norm constraint.
  2. Why are the component variances the eigenvalues of the covariance?
  3. Prove that reconstruction error equals the discarded eigenvalues.
  4. Why must data be centred before PCA?
  5. When must features be standardised?
  6. Explain why SVD is more numerically stable than eigen-decomposition of XᵀX.
  7. Define whitening and state its covariance.
  8. Why is PCA not a feature-selection method?
  9. Give an example where the highest-variance direction is useless for prediction.
  10. What is the cost of a full SVD for an n × d matrix?

Leave a Reply

Your email address will not be published. Required fields are marked *