Chapter 31: Descriptive Statistics & Data Summarization

Chapter 31: Descriptive Statistics & Data Summarization

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 31 of 48

Before any model is trained, the data must be summarised, and the choices made in that summary determine what the model can learn. This chapter covers measures of centre and spread, the difference between standard deviation and standard error, why quantiles are the honest summary for skewed data, and how a single outlier can move a mean while leaving a median untouched.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Centre Mean presented as “the” average. Mean, median and mode contrasted on the same distribution, with skew drawn.
Spread SD and SE used interchangeably. Separated, with the √n factor made explicit.
Quantiles Mentioned only for box plots. Positioned as the summary that makes no shape assumption.
Robustness Never quantified. Measured: how far one outlier moves each statistic.

Core learning objectives. After this chapter you should be able to:

  1. Compute mean, median, mode, and explain when they disagree.
  2. Compute variance, standard deviation, IQR, and MAD.
  3. Distinguish standard deviation from standard error.
  4. Quantify how much a single outlier shifts each statistic.
  5. Choose a summary appropriate to the shape of the data.
  6. Read a summary table and identify which entries are modelling assumptions.

2. Measures of Centre

A histogram with mean, median and mode marked side by side plus a note on robustness
Figure 31.1: In a symmetric distribution the three coincide. Skew separates them.
mean    = sum x / n          sensitive to every value
median  = middle value       only position matters
mode    = most frequent      only for discrete or binned data

symmetric   mean = median
right-skew  mean > median    income, latency, loss spikes
left-skew   mean < median
import numpy as np

latency = np.array([10, 11, 12, 12, 13, 14, 15, 16, 18, 92])

print("mean  :", round(latency.mean(), 3))
print("median:", np.median(latency))
print("mode  :", np.bincount(latency).argmax(), "(value 12)")
print("p90   :", np.percentile(latency, 90))
print("p99   :", np.percentile(latency, 99))

The mean is 21.3 and the median is 13.5. Both are correct; they answer different questions. The mean answers “what is the total divided by the count”, the median answers “what does a typical request experience”. For latency monitoring the second is almost always the one users care about.

3. Quantiles and Robust Spread

A cumulative distribution with Q1, median and Q3 marked and notes on p90 and IQR
Figure 31.2: Quantiles describe the distribution without assuming any shape.
Statistic Formula Robust?
Standard deviation √(Σ(x−x̄)²/n) No
IQR Q3 − Q1 Yes
MAD median(|x − median|) Yes
import numpy as np

x = np.array([10, 11, 12, 12, 13, 14, 15, 16, 18, 92])
q1, q3 = np.percentile(x, [25, 75])
iqr = q3 - q1
mad = np.median(np.abs(x - np.median(x)))

print(f"IQR = {iqr:.2f}   MAD = {mad:.2f}   SD = {x.std():.2f}")

# add one extreme value and watch what moves
y = np.append(x, 1000)
print(f"\nSD  {x.std():8.2f} -> {y.std():8.2f}")
print(f"IQR {iqr:8.2f} -> {np.percentile(y,75)-np.percentile(y,25):8.2f}")
print(f"MAD {mad:8.2f} -> {np.median(np.abs(y-np.median(y))):8.2f}")

The standard deviation roughly doubles while the IQR and MAD barely move. That is what robustness means quantitatively: the estimator’s sensitivity to a small fraction of arbitrarily bad points.

4. Standard Deviation Versus Standard Error

Standard deviation of observations contrasted with standard error of the mean and the root n factor
Figure 31.3: One describes the data, the other describes your estimate of a mean.
SD = sigma                 spread of individual observations
SE = sigma / sqrt(n)       uncertainty of the sample mean

SE < SD always (for n > 1), by a factor of sqrt(n)
Quoting SE where SD is meant makes results look
far more consistent than they are.
import numpy as np

rng = np.random.default_rng(0)
data = rng.normal(100, 15, 400)

sd = data.std(ddof=1)
se = sd / np.sqrt(len(data))
print(f"SD = {sd:.3f}   SE = {se:.3f}   ratio = {sd/se:.2f} (= sqrt(n) = {np.sqrt(len(data)):.2f})")

# the SE is confirmed by resampling
means = [rng.choice(data, size=len(data), replace=True).mean() for _ in range(3000)]
print(f"bootstrap SD of the mean = {np.std(means):.3f}")

5. Outliers and the Training Analogue

A cluster of points with one extreme outlier and notes on robust statistics and gradient clipping
Figure 31.4: One point moves the mean more than the median. Gradient clipping is the same defence.
import numpy as np

clean = np.array([1.0, 1.1, 1.2, 1.3, 1.4, 1.5])
noisy = np.append(clean, 50.0)

print(f"mean   {clean.mean():7.3f} -> {noisy.mean():7.3f}  (shift {noisy.mean()-clean.mean():+.3f})")
print(f"median {np.median(clean):7.3f} -> {np.median(noisy):7.3f}  (shift {np.median(noisy)-np.median(clean):+.3f})")

# trimmed mean: drop the extreme 10% on each side
def trimmed(a, frac=0.1):
    a = np.sort(a)
    k = max(1, int(len(a) * frac))
    return a[k:-k].mean() if len(a) > 2 * k else a.mean()

print(f"trimmed mean of noisy: {trimmed(noisy):.3f}")

The training-loop analogue is gradient clipping. A single sample producing a gradient two orders of magnitude larger than the rest will dominate the update exactly as a single outlier dominates a mean. Clipping to a norm bound is the robust-estimator fix applied to a vector.

6. A Summary Table Is a Set of Assumptions

Six summary statistics with what each reveals and its caveat
Figure 31.5: Every row encodes a decision about what counts as a fair summary.
import numpy as np
import pandas as pd

rng = np.random.default_rng(1)
df = pd.DataFrame({
    "age": rng.integers(18, 80, 500).astype(float),
    "income": rng.lognormal(10, 1, 500),
    "score": rng.normal(70, 12, 500),
})
df.loc[3, "income"] = np.nan

summary = pd.DataFrame({
    "count": df.count(),
    "missing_%": (df.isna().mean() * 100).round(2),
    "mean": df.mean().round(2),
    "median": df.median().round(2),
    "std": df.std().round(2),
    "min": df.min().round(2),
    "max": df.max().round(2),
})
print(summary)

Note how far income‘s mean sits from its median. A log-normal distribution has a heavy right tail, so the mean is pulled upward by a small number of large values while the median describes the typical case. Reporting only the mean is technically correct and practically misleading.

7. Worked Examples

7.1 Five-number summary

data: 3, 7, 8, 12, 15, 18, 22
min 3   Q1 7.5   median 12   Q3 16.5   max 22
IQR = 16.5 - 7.5 = 9

7.2 Detecting an outlier by IQR

fence = Q1 - 1.5 IQR  and  Q3 + 1.5 IQR
      = 7.5 - 13.5 = -6   and  16.5 + 13.5 = 30
Anything outside [-6, 30] is flagged.

7.3 SD versus SE

sigma = 20, n = 100
SD = 20, SE = 2
Reporting SE=2 for individual variation understates spread tenfold.

7.4 Effect of one outlier on the mean

n = 10, mean 10, one value replaced by 1000
new mean = (10*10 - 10 + 1000)/10 = 109
Shift of +99 from a single point in ten.

7.5 Why median is the 50th percentile

Percentile p = value below which p% of observations fall.
Median = 50th percentile by definition.

8. Verify It in Python

import numpy as np

# --- 1. mean and median agree only when symmetric ----------------
sym = np.array([1., 2., 3., 4., 5.])
assert np.isclose(sym.mean(), np.median(sym))
skewed = np.array([1., 2., 3., 4., 100.])
assert skewed.mean() > np.median(skewed)
print("skewed: mean", skewed.mean(), " median", np.median(skewed))

# --- 2. quantiles are order statistics ---------------------------
q1, med, q3 = np.percentile(sym, [25, 50, 75])
assert q1 <= med <= q3
print("Q1, median, Q3:", q1, med, q3)

# --- 3. IQR and MAD resist an outlier ---------------------------
rng0 = np.random.default_rng(7)
clean = rng0.normal(50, 5, 1000)
dirty = np.append(clean, 1e6)                 # one absurd value in 1001

def rel_change(a, b):
    return abs(a - b) / abs(a)

iqr_c = np.subtract(*np.percentile(clean, [75, 25]))
iqr_d = np.subtract(*np.percentile(dirty, [75, 25]))
mad_c = np.median(np.abs(clean - np.median(clean)))
mad_d = np.median(np.abs(dirty - np.median(dirty)))

print(f"IQR relative change: {rel_change(iqr_c, iqr_d):.2e}")
print(f"MAD relative change: {rel_change(mad_c, mad_d):.2e}")
print(f"SD  relative change: {rel_change(clean.std(), dirty.std()):.2e}")

assert rel_change(iqr_c, iqr_d) < 1e-2        # IQR essentially unmoved
assert rel_change(mad_c, mad_d) < 1e-2        # MAD essentially unmoved
assert rel_change(clean.std(), dirty.std()) > 1.0   # SD more than doubles
print("robust statistics barely moved; standard deviation blew up")

# --- 4. SE = SD / sqrt(n) ---------------------------------------
rng = np.random.default_rng(0)
d = rng.normal(0, 1, 900)
assert np.isclose(d.std(ddof=1) / np.sqrt(len(d)), d.std(ddof=1) / 30)
print("SE factor:", round(np.sqrt(len(d)), 2))

# --- 5. trimmed mean is robust ----------------------------------
def trimmed(a, frac=0.1):
    a = np.sort(a)
    k = max(1, int(len(a) * frac))
    return a[k:-k].mean()
assert abs(trimmed(dirty) - np.median(clean)) < 3.0
print("trimmed mean stays near the bulk:", round(trimmed(dirty), 2))

# --- 6. IQR fence flags an injected outlier ----------------------
sample = np.append(rng0.normal(50, 5, 200), 200.0)
q1s, q3s = np.percentile(sample, [25, 75])
iqr_s = q3s - q1s
lo, hi = q1s - 1.5 * iqr_s, q3s + 1.5 * iqr_s
flagged = sample[(sample < lo) | (sample > hi)]
assert len(flagged) == 1 and flagged[0] == 200.0
print("fence flagged:", flagged)

# --- 7. variance identity ---------------------------------------
assert np.isclose(np.mean((clean - clean.mean()) ** 2),
                  np.mean(clean ** 2) - clean.mean() ** 2)
print("Var = E[X^2] - E[X]^2 verified")
print("all assertions passed")

9. Practice Questions

  1. Compute mean, median and mode for [2,3,3,3,5,7,9].
  2. For which shape of distribution does mean exceed median?
  3. Compute the five-number summary and IQR for [4,8,15,16,23,42].
  4. Apply the 1.5×IQR fence and identify any outliers.
  5. Distinguish SD from SE in one sentence each.
  6. With σ = 12 and n = 144, compute the standard error.
  7. Why is MAD more robust than SD?
  8. Quantify the shift in the mean caused by replacing one value of 10 with 1000 in ten observations of 10.
  9. Explain the training-loop analogue of robust statistics.
  10. Why is reporting only the mean of a log-normal variable misleading?

Leave a Reply

Your email address will not be published. Required fields are marked *