Chapter 33: Correlation & Regression Mathematics

Chapter 33: Correlation & Regression Mathematics

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 33 of 48

Correlation and regression are the two oldest tools in applied statistics, and the two most frequently misread. Correlation measures only linear association and says nothing about causation; regression fits a conditional mean and says nothing about the individual case. This chapter derives both from covariance, shows why the least-squares line has the form it does, and explains how residual plots reveal what the fit missed.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Correlation Presented as “how related two things are”. Restricted explicitly to linear association, with a zero-correlation curve counterexample.
Regression Formula given without deriving slope from covariance. Slope derived as Cov/Var, matching the projection argument of Chapter 12.
Diagnostics R² reported alone. Residual plots shown for the three standard failure patterns.
Aggregation Simpson’s paradox never mentioned. Shown as a case where pooling reverses a within-group trend.

Core learning objectives. After this chapter you should be able to:

  1. Compute Pearson correlation and state its range and limits.
  2. Give an example of zero correlation with a strong relationship.
  3. Derive the least-squares slope as Cov(x,y)/Var(x).
  4. Show that the fitted line passes through the point of means.
  5. Read a residual plot and name the misspecification it reveals.
  6. Explain Simpson’s paradox and why stratification changes conclusions.

2. Correlation Is Linear Only

Three scatter plots: strong positive, strong negative, and a perfect curve with zero correlation
Figure 33.1: The third panel has r ≈ 0 and a perfectly deterministic relationship.
Pearson r = Cov(x, y) / (SD(x) SD(y))        range [-1, 1]

r = 0 means no LINEAR association.
It does not mean independence — see y = x^2.

Spearman rank correlation catches monotone non-linear
relationships that Pearson misses.
import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(-3, 3, 20000)
y_lin = 2 * x + rng.normal(0, 0.5, x.size)
y_quad = x ** 2 + rng.normal(0, 0.3, x.size)

print("linear  : r =", round(np.corrcoef(x, y_lin)[0, 1], 4))
print("quadratic: r =", round(np.corrcoef(x, y_quad)[0, 1], 4), "  <- near zero")

# a rank correlation would be needed to detect the monotone half;
# Pearson cannot, by construction

The quadratic case is the standard demonstration that r = 0 does not mean "unrelated". It means "no straight line summarises this". Reporting correlation without plotting is how that error survives.

3. Least Squares From Covariance

Scatter with a fitted line and vertical residuals, plus the slope and intercept formulas
Figure 33.2: The slope is covariance divided by variance; the intercept makes the line pass through the means.
minimise  sum (y_i - b0 - b1 x_i)^2

b1 = Cov(x, y) / Var(x)
b0 = mean(y) - b1 mean(x)

Consequences:
  the line passes through (mean x, mean y)
  the residuals sum to zero
  r^2 = b1^2 Var(x) / Var(y)
import numpy as np

rng = np.random.default_rng(1)
x = rng.normal(0, 1, 500)
y = 3.0 * x + 2.0 + rng.normal(0, 0.5, 500)

b1 = np.cov(x, y, ddof=0)[0, 1] / x.var()
b0 = y.mean() - b1 * x.mean()
print(f"fitted: y = {b1:.3f} x + {b0:.3f}   (true 3.0, 2.0)")

resid = y - (b0 + b1 * x)
print("sum of residuals:", round(resid.sum(), 10))
print("mean point on line:", np.isclose(b0 + b1 * x.mean(), y.mean()))

# cross-check against lstsq
X = np.column_stack([x, np.ones_like(x)])
w, *_ = np.linalg.lstsq(X, y, rcond=None)
assert np.allclose(w, [b1, b0], atol=1e-9)
print("matches lstsq")

This is the same solution derived in Chapter 12 by projection. The normal equations XᵀX w = Xᵀy reduce to the covariance formulas in the single-predictor case — a useful consistency check that the geometry and the algebra agree.

4. Residual Plots

Three residual plots: random scatter, a U-shaped curve, and a fan shape widening with x
Figure 33.3: Structure in residuals is the model telling you what it could not fit.
Pattern Diagnosis Fix
Random scatter Model is adequate None
Curvature (U or arch) Relationship is non-linear Add a polynomial or transform
Fan shape Heteroscedasticity — variance depends on x Weighted least squares or log-transform
import numpy as np

rng = np.random.default_rng(2)
x = np.linspace(0, 10, 400)

# heteroscedastic: noise grows with x
y = 2 * x + rng.normal(0, 1, x.size) * (0.2 + x * 0.3)
X = np.column_stack([x, np.ones_like(x)])
w, *_ = np.linalg.lstsq(X, y, rcond=None)
resid = y - X @ w

# bucket the residuals and compare their spread
buckets = np.array_split(resid, 5)
print("residual std by bucket:", [round(b.std(), 3) for b in buckets])
print("spread grows left to right -> heteroscedasticity")

5. R² and Its Limits

The R-squared formula with notes on its inflation under added features and the need for cross-validation
Figure 33.4: R² measures fit, not usefulness, and rises mechanically with more features.
import numpy as np

rng = np.random.default_rng(3)
n = 100
x = rng.normal(size=(n, 1))
y = rng.normal(size=n)               # pure noise, no relationship

def r2(X, y):
    X1 = np.column_stack([X, np.ones(len(y))])
    w, *_ = np.linalg.lstsq(X1, y, rcond=None)
    pred = X1 @ w
    ss_res = ((y - pred) ** 2).sum()
    ss_tot = ((y - y.mean()) ** 2).sum()
    return 1 - ss_res / ss_tot

print(f"R^2 with 1 noise feature : {r2(x, y):.4f}")
X_many = rng.normal(size=(n, 30))
print(f"R^2 with 30 noise features: {r2(X_many, y):.4f}")
print("R^2 rises with irrelevant features — it always does")

With 30 random predictors and 100 samples, R² climbs to roughly 0.3 despite there being no relationship whatsoever. Adjusted R² and cross-validation penalise this; a raw in-sample R² on a model with many parameters is close to meaningless.

6. Simpson's Paradox

Two separated clusters each with a positive trend but a shallower pooled trend
Figure 33.5: Within each group the slope is positive; pooled it reverses because of the offset.
import numpy as np

# each group rises internally, but the groups sit at opposite heights
g1x = np.array([1., 2., 3., 4.]); g1y = np.array([10.0, 10.9, 12.1, 12.8])
g2x = np.array([6., 7., 8., 9.]); g2y = np.array([2.0, 2.9, 4.1, 4.8])

def slope(x, y):
    return np.cov(x, y, ddof=0)[0, 1] / x.var()

print("group 1 slope:", round(slope(g1x, g1y), 3))
print("group 2 slope:", round(slope(g2x, g2y), 3))
print("pooled slope :", round(slope(np.r_[g1x, g2x], np.r_[g1y, g2y]), 3))

Both groups rise. Pooled, the trend reverses, because the between-group offset dominates the within-group relationship. Any analysis that ignores a group label correlated with both variables can report the opposite sign to the truth — which is why stratifying before concluding is not optional.

7. Worked Examples

7.1 Correlation from covariance

Cov(x,y) = 6,  SD(x) = 3,  SD(y) = 4
r = 6 / 12 = 0.5

7.2 Slope from correlation

b1 = r * SD(y) / SD(x) = 0.5 * 4 / 3 = 0.667

7.3 R² from correlation

For simple linear regression, R^2 = r^2.
r = 0.5 -> R^2 = 0.25, so 25% of variance explained.

7.4 Intercept

mean(x) = 5, mean(y) = 10, b1 = 0.667
b0 = 10 - 0.667*5 = 6.67

7.5 Why not regress x on y

Regressing x on y gives a different slope: r SD(x)/SD(y).
Ordinary least squares minimises vertical error only,
so the direction of regression is a choice, not a formality.

8. Verify It in Python

import numpy as np

rng = np.random.default_rng(0)

# --- 1. correlation bounds ---------------------------------------
for _ in range(50):
    a = rng.normal(size=200); b = rng.normal(size=200)
    r = np.corrcoef(a, b)[0, 1]
    assert -1.0001 <= r <= 1.0001
print("correlation stays in [-1, 1]")

# --- 2. zero correlation with dependence -------------------------
x = rng.uniform(-3, 3, 30000)
assert abs(np.corrcoef(x, x ** 2)[0, 1]) < 0.05
print("corr(x, x^2) ~ 0 despite exact dependence")

# --- 3. slope equals Cov/Var ------------------------------------
xv = rng.normal(0, 2, 800)
yv = -1.5 * xv + 4 + rng.normal(0, 0.4, 800)
b1 = np.cov(xv, yv, ddof=0)[0, 1] / xv.var()
assert abs(b1 + 1.5) < 0.05
print("slope recovered:", round(b1, 3))

# --- 4. line passes through the means ---------------------------
b0 = yv.mean() - b1 * xv.mean()
assert np.isclose(b0 + b1 * xv.mean(), yv.mean())
print("line passes through the mean point")

# --- 5. residuals sum to zero -----------------------------------
resid = yv - (b0 + b1 * xv)
assert abs(resid.sum()) < 1e-9
print("residuals sum to zero")

# --- 6. R^2 == r^2 for simple regression ------------------------
r = np.corrcoef(xv, yv)[0, 1]
ss_res = (resid ** 2).sum()
ss_tot = ((yv - yv.mean()) ** 2).sum()
assert np.isclose(1 - ss_res / ss_tot, r ** 2, atol=1e-9)
print("R^2 equals r^2:", round(r**2, 4))

# --- 7. R^2 inflates with noise features ------------------------
n = 120
noise_y = rng.normal(size=n)

def r2(X, y):
    X1 = np.column_stack([X, np.ones(len(y))])
    w, *_ = np.linalg.lstsq(X1, y, rcond=None)
    pred = X1 @ w
    return 1 - ((y - pred) ** 2).sum() / ((y - y.mean()) ** 2).sum()

r2_one = r2(rng.normal(size=(n, 1)), noise_y)
r2_many = r2(rng.normal(size=(n, 30)), noise_y)
assert r2_many > r2_one + 0.1
print(f"R^2 with 1 vs 30 noise features: {r2_one:.3f} -> {r2_many:.3f}")

# --- 8. Simpson's paradox is reproducible -----------------------
g1x = np.array([1., 2., 3., 4.]); g1y = np.array([10., 10.9, 12.1, 12.8])
g2x = np.array([6., 7., 8., 9.]); g2y = np.array([2., 2.9, 4.1, 4.8])
slope = lambda a, b: np.cov(a, b, ddof=0)[0, 1] / a.var()
assert slope(g1x, g1y) > 0 and slope(g2x, g2y) > 0   # both rise within group
assert slope(np.r_[g1x, g2x], np.r_[g1y, g2y]) < 0   # pooled trend reverses
print("within-group slopes:", round(slope(g1x,g1y),2), round(slope(g2x,g2y),2),
      " pooled:", round(slope(np.r_[g1x,g2x], np.r_[g1y,g2y]),2))
print("all assertions passed")

9. Practice Questions

  1. Compute the Pearson correlation given Cov=6, SD(x)=3, SD(y)=4.
  2. Give an example of zero correlation with a strong relationship.
  3. Derive the least-squares slope in terms of covariance and variance.
  4. Show that the fitted line passes through the mean point.
  5. Why do the residuals of a fitted line sum to zero?
  6. State the relation between R² and r for simple regression.
  7. Interpret a fan-shaped residual plot.
  8. Why does R² rise when noise features are added?
  9. Explain Simpson's paradox and its practical warning.
  10. Why is regressing x on y different from y on x?

Leave a Reply

Your email address will not be published. Required fields are marked *