Chapter 26: Random Variables & Probability Distributions

Chapter 26: Random Variables & Probability Distributions

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 26 of 48

A random variable turns an outcome into a number, which is what lets probability be computed. This chapter defines discrete and continuous random variables, distinguishes the mass/density function from the cumulative distribution, introduces the three distributions that cover almost all of machine learning, and explains why the Gaussian appears in settings that have nothing to do with Gaussian data.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Definitions Discrete and continuous treated as interchangeable. Separated, with the density-can-exceed-1 point made explicitly.
PDF vs CDF Conflated, leading to “the probability at a point”. Both plotted, with the integral relationship stated.
Distributions A long catalogue with no selection guidance. Three families that cover classification, regression, and counting.
Why Gaussian Asserted as an empirical fact. Derived from the central limit theorem with a demonstration.

Core learning objectives. After this chapter you should be able to:

  1. Map outcomes to values and state the support of a random variable.
  2. Distinguish a pmf from a pdf and explain why a density can exceed 1.
  3. Compute probabilities from a CDF as differences.
  4. Recognise Bernoulli, categorical, and Gaussian settings in a model.
  5. Explain the central limit theorem and its consequence for noise modelling.
  6. Simulate each distribution and check the empirical moments.

2. From Outcomes to Numbers

Outcomes heads and tails mapped to values 1 and 0 with pmf and pdf notes
Figure 26.1: A random variable is a function from outcomes to numbers. Probability is then defined on the numbers.
Discrete:     P(X = x) = p(x)          pmf, sums to 1
Continuous:   P(a <= X <= b) = integral f  from a to b

f(x) is a density: it may exceed 1.
Only the integral over an interval is a probability.
P(X = exactly x) = 0 for any continuous X.
import numpy as np
from math import erf, sqrt, pi

# Gaussian pdf and cdf
pdf = lambda x: np.exp(-x ** 2 / 2) / sqrt(2 * pi)
cdf = lambda x: 0.5 * (1 + erf(x / sqrt(2)))

print("pdf at 0     :", round(pdf(0.0), 6))        # 0.3989
print("pdf at 0.1   :", round(pdf(0.1), 6))        # 0.3970
print("CDF(1) - CDF(-1):", round(cdf(1) - cdf(-1), 6))   # ~0.6827

# a pdf can exceed 1: N(0, 0.01) at its peak
narrow = lambda x: np.exp(-x ** 2 / (2 * 0.01)) / sqrt(2 * pi * 0.01)
print("narrow pdf at 0:", round(narrow(0.0), 4), " (> 1, which is fine)")

3. Three Distributions

Bernoulli bars, categorical bars over four outcomes, and a Gaussian density curve
Figure 26.2: Bernoulli for binary labels, categorical for class labels, Gaussian for continuous targets.
Distribution Support Where It Appears
Bernoulli(p) {0, 1} Binary classification, dropout mask
Categorical(p₁…pₖ) {1, …, k} Multi-class output, next-token prediction
Gaussian(μ, σ²) ℝ Regression residuals, weight init, VAE latent
import numpy as np

rng = np.random.default_rng(0)
n = 200_000

bern = rng.random(n) < 0.3
print(f"Bernoulli mean {bern.mean():.4f} (expect 0.3)")

cat = rng.choice(4, size=n, p=[0.1, 0.5, 0.25, 0.15])
freq = np.bincount(cat, minlength=4) / n
print("categorical frequencies:", np.round(freq, 3))

gauss = rng.normal(0, 1, n)
print(f"Gaussian mean {gauss.mean():+.4f} std {gauss.std():.4f}")

4. Density and Cumulative Distribution

Gaussian pdf bell curve beside its monotonically increasing CDF
Figure 26.3: The pdf is the derivative of the CDF. The CDF is monotone and ends at 1.
import numpy as np
from math import erf, sqrt, pi

cdf = lambda x: 0.5 * (1 + erf(x / sqrt(2)))
pdf = lambda x: np.exp(-x ** 2 / 2) / sqrt(2 * pi)

# P(-2 <= X <= 2) two ways
interval = cdf(2) - cdf(-2)
print("from CDF difference:", round(interval, 6))          # 0.9545

# numerically integrate the pdf over the same range
xs = np.linspace(-2, 2, 200001)
integral = np.trapezoid(pdf(xs), xs) if hasattr(np, "trapezoid") else np.trapz(pdf(xs), xs)
print("from pdf integration:", round(integral, 6))
assert abs(interval - integral) < 1e-4

The two are related by calculus: F(x) = ∫f and f = F′. Reading a pdf at a point gives a density, not a probability; reading a CDF gives an accumulated probability. Confusing them produces the classic error of "the probability of exactly 0.5 is 0.35", which is dimensionally meaningless for a continuous variable.

5. Why the Gaussian Appears Everywhere

A running sample mean converging on the true mean as the sample size grows
Figure 26.4: The law of large numbers in action: the running mean converges to the true mean, and the noise shrinks like 1/√n.
A uniform density becoming triangular then bell-shaped as independent variables are summed
Figure 26.5: Summing independent variables produces a Gaussian regardless of the original shape.
import numpy as np

rng = np.random.default_rng(0)
for k in (1, 2, 8, 50):
    sums = rng.uniform(0, 1, size=(100_000, k)).sum(axis=1)
    z = (sums - sums.mean()) / sums.std()
    excess = (z ** 4).mean() - 3.0        # 0 for a Gaussian
    print(f"k={k:2d}  mean={sums.mean():.3f}  std={sums.std():.4f}  excess kurtosis={excess:+.4f}")

The central limit theorem is why measurement noise, aggregate gradient noise, and the distribution of a sum of many small effects are all modelled as Gaussian. It does not say the data are Gaussian — it says the sum of many independent contributions is.

Assumption What It Implies for the Loss
Gaussian noise Squared error (Chapter 29 derives this)
Laplace noise Absolute error — robust to outliers
Bernoulli labels Binary cross-entropy

6. Worked Examples

6.1 Bernoulli moments

X ~ Bernoulli(0.3)
E[X] = 0.3
Var(X) = p(1-p) = 0.21

6.2 Gaussian interval probability

P(|X| <= 1) = 0.6827     the 68% rule
P(|X| <= 2) = 0.9545     the 95% rule
P(|X| <= 3) = 0.9973     the 99.7% rule

6.3 Standardising

If X ~ N(mu, sigma^2), then Z = (X - mu)/sigma ~ N(0, 1)
This is why one z-table serves every Gaussian.

6.4 Categorical from logits

logits [2, 1, 0] -> softmax -> [0.665, 0.245, 0.090]
That vector IS the categorical pmf over tokens.

6.5 Sampling a categorical

Draw u ~ Uniform(0,1); walk the CDF until it exceeds u.
That is exactly what rng.choice with probabilities does.

7. Verify It in Python

import numpy as np
from math import erf, sqrt, pi

rng = np.random.default_rng(0)
n = 400_000

# --- 1. Bernoulli moments -----------------------------------------
p = 0.3
b = rng.random(n) < p
assert abs(b.mean() - p) < 0.005
assert abs(b.var() - p * (1 - p)) < 0.005
print(f"Bernoulli: mean {b.mean():.4f}, var {b.var():.4f} (expect 0.21)")

# --- 2. categorical sums to one -----------------------------------
pr = np.array([0.1, 0.5, 0.25, 0.15])
assert np.isclose(pr.sum(), 1.0)
cat = rng.choice(4, size=n, p=pr)
freq = np.bincount(cat, minlength=4) / n
assert np.allclose(freq, pr, atol=0.005)
print("categorical frequencies match the pmf")

# --- 3. pdf is the derivative of the cdf --------------------------
cdf = lambda x: 0.5 * (1 + erf(x / sqrt(2)))
pdf = lambda x: np.exp(-x ** 2 / 2) / sqrt(2 * pi)
h = 1e-5
assert abs((cdf(0.5 + h) - cdf(0.5 - h)) / (2 * h) - pdf(0.5)) < 1e-3
print("numerical d(cdf)/dx equals pdf")

# --- 4. density can exceed one ------------------------------------
narrow = lambda x: np.exp(-x ** 2 / (2 * 0.01)) / sqrt(2 * pi * 0.01)
assert narrow(0.0) > 1.0
print("narrow Gaussian density at 0:", round(narrow(0.0), 3))

# --- 5. probability of an exact continuous value is zero ----------
xs = np.linspace(-5, 5, 400_001)
width = 1e-9
tiny = np.trapezoid(pdf(np.linspace(0.5, 0.5 + width, 3)), dx=width) if hasattr(np, "trapezoid") else 0.0
assert tiny < 1e-8
print("P(X = exactly 0.5) is effectively zero")

# --- 6. empirical interval matches the 68-95-99.7 rule -----------
g = rng.normal(0, 1, n)
for k, expect in ((1, 0.6827), (2, 0.9545), (3, 0.9973)):
    got = (np.abs(g) <= k).mean()
    assert abs(got - expect) < 0.005
print("Gaussian interval coverage matches the rule")

# --- 7. central limit theorem: kurtosis -> 0 ---------------------
for k in (1, 20):
    z = rng.uniform(0, 1, size=(60_000, k)).sum(axis=1)
    z = (z - z.mean()) / z.std()
    excess = (z ** 4).mean() - 3.0
    if k == 20:
        assert abs(excess) < 0.15
print("sum of 20 uniforms has near-zero excess kurtosis")

# --- 8. standardisation preserves shape --------------------------
raw = rng.normal(10, 4, n)
z = (raw - raw.mean()) / raw.std()
assert abs(z.mean()) < 0.01 and abs(z.std() - 1) < 0.01
print("standardised: mean", round(z.mean(), 4), " std", round(z.std(), 4))
print("all assertions passed")

8. Practice Questions

  1. Distinguish a pmf from a pdf and state which can exceed 1.
  2. Why is P(X = x) = 0 for any continuous random variable?
  3. Compute P(−1 ≤ X ≤ 1) for a standard Gaussian.
  4. Give the mean and variance of Bernoulli(p).
  5. Standardise an observation of 7 from N(10, 4²).
  6. Explain why a softmax output is a categorical pmf.
  7. State the central limit theorem in one sentence.
  8. Why does Gaussian noise lead to squared-error loss?
  9. What does an excess kurtosis of 0 indicate?
  10. Why is the exact value of a density at a point not a probability?

Leave a Reply

Your email address will not be published. Required fields are marked *