Chapter 29: Maximum Likelihood & Maximum A Posteriori Estimation

Chapter 29: Maximum Likelihood & Maximum A Posteriori Estimation

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 29 of 48

Every loss function you have trained with is a negative log-likelihood in disguise. Squared error is the log-likelihood of Gaussian noise; binary cross-entropy is the log-likelihood of a Bernoulli label; softmax cross-entropy is the log-likelihood of a categorical label. This chapter derives those equivalences, then adds a prior to turn maximum likelihood into maximum a posteriori estimation — which is exactly weight decay, and explains where the regularisation coefficient comes from.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
MLE Presented as a separate statistical topic. Shown to be what every training loop already does.
Log-likelihood The switch from product to sum asserted. Three concrete reasons given, including underflow.
MAP Never connected to regularisation. Gaussian prior derived as L2 decay; Laplace as L1.
Small samples MLE’s degenerate behaviour unmentioned. Zero-count failure shown, with Laplace smoothing as the fix.

Core learning objectives. After this chapter you should be able to:

  1. Write the likelihood and log-likelihood for a dataset.
  2. Derive squared error from a Gaussian likelihood.
  3. Derive cross-entropy from Bernoulli and categorical likelihoods.
  4. Show that a Gaussian prior yields L2 regularisation.
  5. Explain why the log form is used rather than the product.
  6. Identify the failure mode of MLE with zero counts and its remedy.

2. Maximum Likelihood

A likelihood curve over theta peaking at one third with the argmax formulation
Figure 29.1: Maximum likelihood picks the parameters under which the observed data were most probable.
likelihood      L(theta) = product_i p(x_i | theta)
log-likelihood  l(theta) = sum_i  log p(x_i | theta)

theta_hat = argmax L = argmax l      (log is monotone)

Note: argmax, not the value. We care about the location of the
peak, not its height.
import numpy as np

# 3 coin flips: 1 success, 2 failures -> MLE theta = 1/3
def loglik(theta, k, n):
    if not 0 < theta < 1:
        return -np.inf
    return k * np.log(theta) + (n - k) * np.log(1 - theta)

k, n = 1, 3
grid = np.linspace(1e-6, 1 - 1e-6, 200001)
vals = np.array([loglik(t, k, n) for t in grid])
best = grid[int(np.argmax(vals))]
print(f"MLE on a grid: {best:.6f}  (exact {k/n:.6f})")
assert abs(best - k / n) < 1e-4

3. Every Loss Is a Likelihood

Table mapping Gaussian, Bernoulli, categorical and Laplace noise models to their implied loss functions
Figure 29.2: Choosing a loss is choosing a noise model.
Noise Model Negative Log-Likelihood Standard Name
Gaussian (y − ŷ)² / 2σ² + const squared error / MSE
Bernoulli −[y log ŷ + (1−y) log(1−ŷ)] binary cross-entropy
Categorical −log ŷ_class softmax cross-entropy
Laplace |y − ŷ| / b + const MAE — robust to outliers
import numpy as np

y = np.array([1.0, 0.0, 1.0, 1.0])
p = np.array([0.9, 0.2, 0.6, 0.4])

bce = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
print("binary cross-entropy:", round(bce, 4))

# same quantity as a negative log-likelihood
nll = -np.sum(y * np.log(p) + (1 - y) * np.log(1 - p))
print("negative log-likelihood:", round(nll, 4), " (bce * n)")
assert np.isclose(bce * len(y), nll)

# Gaussian NLL reduces to MSE up to constants
target = np.array([1.0, 2.0, 3.0])
pred = np.array([1.1, 1.9, 3.2])
mse = np.mean((target - pred) ** 2)
sigma = 1.0
gauss_nll = np.mean(0.5 * (target - pred) ** 2 / sigma ** 2 + 0.5 * np.log(2 * np.pi * sigma ** 2))
print("MSE:", round(mse, 4), " Gaussian NLL minus const:", round(gauss_nll - 0.5 * np.log(2 * np.pi), 4))

4. From MLE to MAP

MLE and MAP objectives, with Gaussian prior giving L2 decay and Laplace prior giving L1
Figure 29.3: Adding a prior term turns likelihood maximisation into regularised training.
MAP:  argmax [ log p(D | theta) + log p(theta) ]
                 ^data term           ^prior term

Gaussian prior N(0, s^2):
  log p(theta) = -||theta||^2 / (2 s^2) + const
  => objective gains  -lambda ||theta||^2      L2 / weight decay

Laplace prior:
  log p(theta) = -||theta||_1 / b + const
  => objective gains  -lambda ||theta||_1      L1 / sparsity
import numpy as np

rng = np.random.default_rng(0)
X = rng.normal(size=(60, 5))
w_true = np.array([2.0, -1.0, 0.0, 0.0, 1.5])
y = X @ w_true + rng.normal(0, 0.3, 60)

XtX = X.T @ X
Xty = X.T @ y

print(f"{'lambda':>8} {'||w||':>8} {'train MSE':>10}")
for lam in (0.0, 0.1, 1.0, 10.0, 100.0):
    w = np.linalg.solve(XtX + lam * np.eye(5), Xty)
    mse = np.mean((X @ w - y) ** 2)
    print(f"{lam:8.1f} {np.linalg.norm(w):8.4f} {mse:10.4f}")

The prior's variance and the regularisation coefficient are inversely related: a tight prior (small s) means strong regularisation. Tuning weight decay is therefore tuning a belief about how large the weights plausibly are — which is a more useful framing than treating λ as an arbitrary knob.

5. Why the Log Form

Product likelihood turning into a sum of logs with three practical advantages listed
Figure 29.4: Same maximum, better numerics, easier derivatives, parallelisable.

import numpy as np

# products of many small probabilities underflow
probs = np.full(2000, 0.5)
product = np.prod(probs)
logsum = np.sum(np.log(probs))
print("product of 2000 halves :", product)          # 0.0
print("sum of logs            :", logsum)           # -1386.29
print("exp of the log-sum     :", np.exp(logsum))  # underflows in float64
print("...which is exactly why losses are kept in log space")
Reason Detail
Underflow A product of thousands of probabilities flushes to zero; a sum of logs does not
Derivatives The derivative of a sum is a sum of derivatives — no product rule chain explosion
Batching Terms decompose across samples, so a batch loss is a mean of per-sample losses

6. MLE With Few Samples

A likelihood concentrated near one after a single success, with the smoothing remedy noted
Figure 29.5: One observation drives MLE to certainty. A prior prevents the degenerate estimate.
import numpy as np

def mle(k, n):
    return k / n

def smoothed(k, n, alpha=1.0):
    return (k + alpha) / (n + 2 * alpha)

print(f"{'k/n':>8} {'MLE':>8} {'smoothed':>10}")
for k, n in ((1, 1), (1, 3), (10, 10), (0, 5), (95, 100)):
    print(f"{k}/{n:<6} {mle(k, n):8.3f} {smoothed(k, n):10.3f}")
print("\nMLE hits exactly 0 or 1; smoothing never does.")

This is Laplace smoothing, and it is why n-gram language models add one to every count. Without it a single unseen bigram makes the whole sequence probability zero, and one zero anywhere collapses the product — the exact failure that log-space arithmetic cannot rescue, because log 0 is negative infinity.

7. Worked Examples

7.1 MLE for a Bernoulli

k successes in n trials
l(theta) = k log theta + (n-k) log(1-theta)
dl/dtheta = k/theta - (n-k)/(1-theta) = 0
=> theta_hat = k/n

7.2 MLE for a Gaussian mean

Maximising sum log N(x_i; mu, sigma^2) over mu
gives mu_hat = (1/n) sum x_i
The sample mean IS the maximum-likelihood estimate.

7.3 Gaussian prior is L2

log p(w) = -sum w_j^2 / (2s^2) + const
Adding it to the log-likelihood is exactly adding
lambda * sum w_j^2, with lambda = 1/(2s^2).

7.4 Laplace prior is L1

log p(w) = -sum |w_j| / b + const
A prior peaked at zero favours exactly-zero weights —
which is why L1 produces sparse solutions.

7.5 Cross-entropy from a categorical

For one-hot y with true class c:
  -log p(y | theta) = -log q_c
Exactly the softmax cross-entropy of Chapter 4.

8. Verify It in Python

import numpy as np

# --- 1. MLE equals the observed frequency ------------------------
for k, n in ((3, 10), (7, 20), (1, 4)):
    assert np.isclose(k / n, k / n)
print("MLE for Bernoulli is k/n")

# --- 2. grid search recovers it ----------------------------------
def loglik(th, k, n):
    return k * np.log(th) + (n - k) * np.log(1 - th)

grid = np.linspace(1e-6, 1 - 1e-6, 100001)
best = grid[np.argmax([loglik(t, 7, 20) for t in grid])]
assert abs(best - 0.35) < 1e-4
print("grid MLE:", round(best, 5))

# --- 3. Gaussian NLL is MSE up to a constant --------------------
y = np.array([1.0, 2.0, 3.0, 4.0])
p = np.array([1.2, 1.8, 3.1, 3.9])
mse = np.mean((y - p) ** 2)
nll = np.mean(0.5 * (y - p) ** 2)
assert np.isclose(nll, 0.5 * mse)
print("Gaussian NLL = 0.5 * MSE")

# --- 4. BCE is the Bernoulli NLL --------------------------------
yl = np.array([1., 0., 1., 1.]); pl = np.array([.9, .2, .6, .4])
bce = -np.mean(yl * np.log(pl) + (1 - yl) * np.log(1 - pl))
nll2 = -np.sum(yl * np.log(pl) + (1 - yl) * np.log(1 - pl))
assert np.isclose(bce * len(yl), nll2)
print("BCE * n == Bernoulli NLL")

# --- 5. log form avoids underflow ------------------------------
probs = np.full(3000, 0.5)
assert np.prod(probs) == 0.0                    # underflows
assert np.isfinite(np.sum(np.log(probs)))       # does not
print("product underflows, log-sum does not")

# --- 6. MAP shrinks the estimate --------------------------------
X = np.random.default_rng(0).normal(size=(50, 4))
XtX = X.T @ X
Xty = XtX @ np.array([3.0, 0.0, 0.0, -2.0])
w_ml = np.linalg.solve(XtX, Xty)
w_map = np.linalg.solve(XtX + 10.0 * np.eye(4), Xty)
assert np.linalg.norm(w_map) < np.linalg.norm(w_ml)
print(f"MLE norm {np.linalg.norm(w_ml):.3f} -> MAP norm {np.linalg.norm(w_map):.3f}")

# --- 7. smoothing removes the degenerate estimate ---------------
def sm(k, n, a=1.0):
    return (k + a) / (n + 2 * a)
assert (1 / 1) == 1.0 and sm(1, 1) < 1.0
assert (0 / 5) == 0.0 and sm(0, 5) > 0.0
print(f"1/1: MLE 1.000 -> smoothed {sm(1,1):.3f}; 0/5: MLE 0.000 -> {sm(0,5):.3f}")

# --- 8. L1 penalty yields exact zeros (soft threshold) ----------
def soft_threshold(z, lam):
    return np.sign(z) * np.maximum(np.abs(z) - lam, 0.0)
assert soft_threshold(0.3, 1.0) == 0.0
assert soft_threshold(2.0, 1.0) == 1.0
print("soft thresholding zeroes small weights exactly")
print("all assertions passed")

9. Practice Questions

  1. Write the log-likelihood for n Bernoulli trials with k successes.
  2. Derive θ̂ = k/n by differentiating and setting to zero.
  3. Show that a Gaussian likelihood yields squared error.
  4. Show that a Bernoulli likelihood yields binary cross-entropy.
  5. State three reasons to use the log-likelihood.
  6. Derive that a Gaussian prior gives L2 regularisation.
  7. Relate the prior's variance to the weight-decay coefficient.
  8. Why does MLE give p = 1 after one success?
  9. What is Laplace smoothing and why does it help?
  10. Which loss corresponds to a Laplace noise model?

Leave a Reply

Your email address will not be published. Required fields are marked *