Chapter 32: Hypothesis Testing & Confidence Intervals

Chapter 32: Hypothesis Testing & Confidence Intervals

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 32 of 48

Hypothesis testing is the machinery for deciding whether an observed difference is real or noise. Its logic is indirect — you assume no effect and ask how surprising the data would be — and that indirection is the source of nearly every misuse. This chapter covers the null hypothesis, the two error types, the two-sided rejection region, and why a confidence interval communicates more than a p-value.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
p-value Described as “the probability H0 is true”. Corrected: it is P(data | H0), not P(H0 | data).
Error types Both listed with no trade-off. Shown that only more data improves both simultaneously.
Multiplicity Omitted entirely. Bonferroni and the expected false-positive count given.
Intervals Confidence interval explained as “95% chance θ is inside”. Corrected to a statement about the procedure.

Core learning objectives. After this chapter you should be able to:

  1. State H₀ and H₁ and compute a p-value for a simple test.
  2. Explain the difference between Type I and Type II error.
  3. Show the expected number of false positives under multiplicity.
  4. Construct and interpret a confidence interval correctly.
  5. Distinguish statistical from practical significance.
  6. Report an effect size alongside a p-value.

2. The Logic of the Null

Null and alternative hypotheses with the p-value definition and rejection rule
Figure 32.1: Testing is indirect. You assume no effect and measure the surprise.
H0: no effect        H1: some effect

p-value = P(data at least as extreme | H0)

Reject H0 if p < alpha (conventionally 0.05).
Fail to reject  !=  prove H0.
import numpy as np
from math import erf, sqrt

rng = np.random.default_rng(0)

# does a new feature improve accuracy?
baseline = rng.binomial(1, 0.70, 1000)
new      = rng.binomial(1, 0.73, 1000)

diff = new.mean() - baseline.mean()
se = np.sqrt(baseline.var() / 1000 + new.var() / 1000)
z = diff / se

normal_cdf = lambda x: 0.5 * (1 + erf(x / sqrt(2)))
p = 2 * (1 - normal_cdf(abs(z)))          # two-sided

print(f"baseline {baseline.mean():.3f}  new {new.mean():.3f}  diff {diff:+.3f}")
print(f"z = {z:.3f}   p-value = {p:.4f}")
print("significant at 0.05:", p < 0.05)

The p-value answers: if the new feature did nothing, how often would a difference this large appear by chance? It does not answer whether the feature works, nor how large the effect is.

3. Two Ways to Be Wrong

A two by two table of decisions against reality showing Type I, Type II, and the correct outcomes
Figure 32.2: Confidence and power are on the diagonal; the two errors are off it.
Term Symbol Given By
Significance level α You choose it
Power 1 − β Effect size, variance, and n
Minimum detectable effect δ Inverse function of n
import numpy as np
from math import erf, sqrt

normal_cdf = lambda x: 0.5 * (1 + erf(x / sqrt(2)))
rng = np.random.default_rng(1)

def power(effect, n, sigma=1.0, alpha=0.05):
    """Approximate two-sided power for a mean difference."""
    se = sigma * np.sqrt(2 / n)
    crit = 1.959963985 * se
    return 1 - normal_cdf((crit - effect) / se) + normal_cdf((-crit - effect) / se)

for n in (100, 1000, 10000):
    print(f"n={n:6d}  power to detect effect 0.1: {power(0.1, n):.3f}")

Power is the number that makes an experiment worth running. A study with 20% power has an 80% chance of missing a real effect, and a non-significant result from such a study carries almost no information.

4. Multiplicity

A null distribution with shaded rejection tails and a note on multiple comparisons
Figure 32.3: The rejection region is set by α. Testing many hypotheses inflates false positives.
import numpy as np

rng = np.random.default_rng(2)

# 20 A/B tests on a metric with no real effect anywhere
n_tests = 20
n = 500
false_positives = 0
for _ in range(2000):
    a = rng.binomial(1, 0.5, n)
    b = rng.binomial(1, 0.5, n)
    se = np.sqrt(a.var()/n + b.var()/n)
    z = (b.mean() - a.mean()) / se
    if abs(z) > 1.959963985:
        false_positives += 1
print(f"tests with >=1 false positive: {false_positives/2000:.3f}")
print("theory: 1 - 0.95^20 =", round(1 - 0.95 ** 20, 3))

# Bonferroni: divide alpha by the number of tests
print("Bonferroni threshold:", 0.05 / n_tests)

Twenty independent tests at α = 0.05 give a 64% chance of at least one false positive — computed as 1 − 0.95²⁰. This is why tuning many hyperparameters and reporting the best is a form of overfitting: you are selecting on noise.

5. Confidence Intervals

Twelve horizontal confidence intervals, most crossing the true value and one missing it
Figure 32.4: Each bar is one interval from one sample. About 1 in 20 misses.
CI = estimate +/- z * SE

95% interval means: if you repeated the experiment many times,
95% of the intervals built this way would contain the true value.
It does NOT mean "95% chance the true value is in this interval"
once the interval is computed — the value is either in it or not.
import numpy as np

rng = np.random.default_rng(3)
true_mu = 5.0
cover = 0
trials = 5000
for _ in range(trials):
    s = rng.normal(true_mu, 1.0, 100)
    lo = s.mean() - 1.959963985 * 1.0 / np.sqrt(100)
    hi = s.mean() + 1.959963985 * 1.0 / np.sqrt(100)
    cover += (lo <= true_mu <= hi)
print(f"coverage across {trials} experiments: {cover/trials:.3f}")

The interval carries more information than the p-value because it shows the magnitude and its uncertainty together. A CI that spans both trivially small and usefully large effects is an inconclusive experiment, and no p-value threshold repairs that.

6. Statistical Versus Practical Significance

Statistical significance and practical significance shown as distinct conditions
Figure 32.5: An effect can be detected without being worth acting on.
Report Tells You
p-value only Whether the effect is distinguishable from zero
Effect size + CI How large, and how precisely known
n and power Whether a null result means anything

7. Worked Examples

7.1 One-sample z-test

sample mean 52, mu0 = 50, sigma = 10, n = 100
z = (52 - 50) / (10/10) = 2.0
two-sided p = 0.0455  ->  just significant at 0.05

7.2 Required sample size

To detect effect d with sigma = 1 at 80% power, alpha 0.05:
n ≈ 16 / d^2
d = 0.1 -> n ≈ 1600
d = 0.5 -> n ≈ 64

7.3 Bonferroni

20 tests, family-wise alpha 0.05
per-test threshold = 0.05/20 = 0.0025

7.4 Interval width

CI half-width = 1.96 sigma / sqrt(n)
To halve it, quadruple n.

7.5 Interpreting a null result

p = 0.4 with n = 30 and power 0.2
This is not evidence of no effect — it is an underpowered test.

8. Verify It in Python

import numpy as np
from math import erf, sqrt

ncdf = lambda x: 0.5 * (1 + erf(x / sqrt(2)))
rng = np.random.default_rng(0)

# --- 1. z-test and p-value ---------------------------------------
a = rng.binomial(1, 0.5, 500)
b = rng.binomial(1, 0.5, 500)
se = np.sqrt(a.var()/len(a) + b.var()/len(b))
z = (b.mean() - a.mean()) / se
p = 2 * (1 - ncdf(abs(z)))
assert 0 <= p <= 1
print(f"z={z:.3f}  p={p:.4f}")

# --- 2. Type I rate under the null -------------------------------
fp = 0
for _ in range(2000):
    x = rng.binomial(1, .5, 300); y = rng.binomial(1, .5, 300)
    s = np.sqrt(x.var()/300 + y.var()/300)
    if abs((y.mean()-x.mean())/s) > 1.959963985:
        fp += 1
rate = fp / 2000
assert 0.02 < rate < 0.08
print(f"Type I rate at alpha=0.05: {rate:.4f}")

# --- 3. multiplicity inflates it --------------------------------
assert 1 - 0.95 ** 20 > 0.6
print("family-wise error for 20 tests:", round(1 - 0.95**20, 3))

# --- 4. power grows with n --------------------------------------
def power(effect, n, sigma=1.0):
    se = sigma * np.sqrt(2 / n)
    crit = 1.959963985 * se
    return 1 - ncdf((crit - effect) / se) + ncdf((-crit - effect) / se)

assert power(0.3, 100) < power(0.3, 1000)
print("power at n=100, 1000:", round(power(0.3,100),3), round(power(0.3,1000),3))

# --- 5. CI coverage is 95% --------------------------------------
cover = 0
for _ in range(4000):
    s = rng.normal(5.0, 1.0, 100)
    lo = s.mean() - 1.959963985 / 10
    hi = s.mean() + 1.959963985 / 10
    cover += (lo <= 5.0 <= hi)
assert abs(cover/4000 - 0.95) < 0.02
print("empirical CI coverage:", round(cover/4000, 4))

# --- 6. CI width scales as 1/sqrt(n) ---------------------------
w100 = 2 * 1.959963985 / np.sqrt(100)
w400 = 2 * 1.959963985 / np.sqrt(400)
assert np.isclose(w100 / w400, 2.0)
print("quadrupling n halves the CI width")

# --- 7. Bonferroni threshold -----------------------------------
assert np.isclose(0.05 / 20, 0.0025)
print("Bonferroni threshold for 20 tests:", 0.05/20)
print("all assertions passed")

9. Practice Questions

  1. State H₀ and H₁ for a test of whether accuracy improved.
  2. Define the p-value precisely and state what it is not.
  3. Distinguish Type I and Type II error with an example each.
  4. Explain why lowering α raises β.
  5. Compute the family-wise error rate for 10 tests at α = 0.05.
  6. State the Bonferroni threshold for 25 tests.
  7. Interpret a 95% confidence interval correctly in one sentence.
  8. What does a non-significant result from an underpowered study tell you?
  9. Give an example where an effect is statistically but not practically significant.
  10. Why does quadrupling n halve a confidence interval's width?

Leave a Reply

Your email address will not be published. Required fields are marked *