Chapter 4: Exponents, Logarithms & the Information-Theoretic View

Chapter 4: Exponents, Logarithms & the Information-Theoretic View

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 4 of 48

Exponentiation and its inverse, the logarithm, are the two operations that connect probability to arithmetic in machine learning. Softmax exponentiates logits so they become positive; cross-entropy takes logarithms so that multiplication becomes addition; perplexity exponentiates back so the number is interpretable. This chapter builds the algebra, then uses it to explain two things that break in practice: why a log scale is mandatory for loss curves, and why naive exp silently returns inf or 0.0 on perfectly ordinary inputs.

1. Technical Audit & Core Learning Objectives

Most expositions present exponent rules as algebra to memorise and logarithms as a calculator button. Neither explains why every language model reports its quality in perplexity, or why a softmax implementation without max-subtraction fails on inputs that are mathematically unremarkable.

Component Common Tutorial Deficiency Engineering Standard Applied Here
Exponent laws Rules stated, then abandoned; never used again. Every rule connected to a transformation actually performed in softmax and log-space arithmetic.
Log scale Treated as a plotting preference. Shown to be the correct scale whenever the quantity is a ratio — loss, perplexity, learning rate.
Softmax Presented as a formula, with no numerical discussion. Derived, then hardened with max-subtraction; the failure it prevents is demonstrated.
Information Cross-entropy appears as a given formula. Built from −log p as the cost of encoding an event, then averaged.

Core learning objectives. After this chapter you should be able to:

  1. Apply all six exponent laws and the corresponding logarithm laws without lookup.
  2. Convert between log, ln, and log₂ and state which base each convention implies.
  3. Explain why loss and learning rate are plotted on a log scale and what a straight line on that scale means.
  4. Derive softmax from the requirement of a positive vector summing to one.
  5. Compute information content −log₂p and connect it to cross-entropy loss.
  6. State the representable range of exp in float64 and write overflow-safe code.

2. Three Ways to Say the Same Thing

Three questions are equivalent for b > 0, b ≠ 1:

bⁿ = x          "what is b raised to n?"          exponentiation
log_b(x) = n    "what power of b gives x?"        logarithm
b^(log_b x) = x these are inverse operations

The base convention matters in practice. Machine learning uses three, and mixing them produces factor-of-ln 2 errors:

Notation Base Used For
ln x e ≈ 2.718 Calculus, cross-entropy, gradient derivation
log₂ x 2 Information theory, bits, entropy
log x 10 or e (ambiguous!) Avoid it in ML writing; state the base

The three bases differ only by a constant factor: log₂ x = ln x / ln 2. Since −log is used as a loss, the choice of base merely rescales the loss by 1/ln 2 ≈ 1.4427 — harmless for optimisation, but crucial when reporting an absolute number in bits versus nats.

3. Exponent Laws and the Log Laws They Imply

Six exponent laws with plain-language translations and notes on floating point overflow
Figure 4.1: The exponent laws and their consequence — multiplication in linear space is addition in log space.
Exponent Law Logarithm Law Why It Matters
bᵐ·bⁿ = bᵐ⁺ⁿ log(xy) = log x + log y Joint log-probability of independent events is a sum
bᵐ/bⁿ = bᵐ⁻ⁿ log(x/y) = log x − log y Log-odds are a difference of log-probabilities
(bᵐ)ⁿ = bᵐⁿ log(xⁿ) = n·log x Perplexity = exp(mean NLL)
b⁰ = 1 log 1 = 0 A perfectly predicted event costs nothing
b⁻ᵐ = 1/bᵐ log(1/x) = −log x Negative log-likelihood is a positive cost

The first row is the load-bearing one. A sequence of n independent token probabilities multiplies to the sequence probability; taking logs turns that product into a sum, which is numerically stable, differentiable, and trivially parallelisable across a batch. This single identity is why language models train in log space.

4. Log Scale: When a Straight Line Means “Constant Ratio”

On a linear axis, 2ˣ is invisible near zero and off the chart by x = 10. On a log axis it becomes a straight line, because log₂(2ˣ) = x.

Exponential curve on linear axis compared with the same data on a log axis where it becomes a straight line
Figure 4.2: The same exponential, two axes. On a log axis, equal vertical distances mean equal multiplicative factors.

This is the correct axis whenever the quantity is a ratio rather than a magnitude:

  • Loss curves. A training loss falling from 2.0 to 0.2 is a 10× improvement whether it happens at epoch 1 or epoch 50. On a linear axis the early progress dominates and the interesting tail is flat.
  • Perplexity. Defined as exp(NLL), so it is inherently multiplicative.
  • Learning rate schedules. Cosine versus exponential decay differ by orders of magnitude at the tail; only a log axis shows it.
  • Parameter magnitudes. Weight distributions spanning 10⁻⁶ to 10⁰ are unreadable linearly.

The practical reading rule: on a log-y plot, a straight line means exponential change; a concave curve that flattens means the change is slowing in ratio terms, which is what a well-behaved training run looks like.

5. Softmax: From Logits to a Distribution

A classifier produces raw scores called logits. They can be any real number, negative or positive, unbounded. We need a probability vector: positive entries summing to one. Two requirements, two steps.

Softmax pipeline showing logits transformed by exp then normalised to probabilities summing to one
Figure 4.3: Logits → exponential (now strictly positive) → divide by the sum (now sums to 1.0).
softmax(z)ᵢ = e^zᵢ / Σⱼ e^zⱼ

Why exp and not something else?
  exp is strictly positive  →  every class gets non-zero probability
  exp is strictly increasing →  order of logits is preserved
  exp turns differences into ratios:
        e^(a−b) = e^a / e^b
  so a logit gap of 1.0 always means the same odds ratio, regardless of scale.
import numpy as np

def softmax(z):
    z = np.asarray(z, dtype=float)
    z = z - z.max()            # shift: identical result, no overflow
    e = np.exp(z)
    return e / e.sum()

z = np.array([2.0, 1.0, 0.1, -1.0])
p = softmax(z)
print("probs:", np.round(p, 4))     # [0.6381 0.2347 0.0954 0.0318]
print("sum  :", p.sum())            # exactly 1.0
print("gap 2 vs 1 -> ratio", np.round(p[0] / p[1], 4), "= e^1 =", np.round(np.e, 4))

The ratio property is the reason softmax is used rather than simple normalisation. Dividing by the sum would also produce a distribution, but the numbers would depend on the arbitrary origin of the logits. With softmax, only differences matter: adding a constant to every logit leaves the output unchanged, which is why the max-subtraction trick is mathematically free.

6. Information Content and Cross-Entropy

Information theory asks: how surprising is an event, and how many bits does it cost to encode? The answer for an event of probability p is −log₂ p.

Plot of information content minus log base 2 of p showing one bit at p equals 0.5 and unbounded cost as p approaches zero
Figure 4.4: Information content. Certain events cost nothing; rare events cost a great deal and the function is unbounded at zero.
Quantity Formula Interpretation
Information I(p) = −log p Surprise of one outcome
Entropy H(p) = −Σ pᵢ log pᵢ Average surprise of a distribution
Cross-entropy H(p,q) = −Σ pᵢ log qᵢ Cost of encoding true distribution p with model q
Perplexity exp(H(p,q)) Effective number of equally-likely choices

For one-hot labels — the standard classification case — cross-entropy collapses to −log q_y where y is the true class. That is the negative log-probability the model assigned to the correct answer. The unbounded behaviour as q_y → 0 is precisely why a confidently wrong prediction is punished so heavily.

import numpy as np

def cross_entropy_one_hot(q, y):
    q = np.asarray(q, dtype=float)
    return -np.log(q[y])

q = np.array([0.7, 0.2, 0.1])
print("correct, confident  :", round(cross_entropy_one_hot(q, 0), 4))   # 0.3567
print("correct, unsure     :", round(cross_entropy_one_hot(q, 2), 4))   # 2.3026
print("confidently wrong   :", round(cross_entropy_one_hot(np.array([0.001, 0.001, 0.998]), 2), 4))

print("perplexity from NLL :", round(np.exp(0.6931), 4))   # ~2.0 classes

7. Where Floating Point Breaks

exp grows fast enough to leave the representable range of a 64-bit float on inputs that arise routinely in a transformer.

Number line showing float64 exp underflow below minus 745, safe range, and overflow above 709
Figure 4.5: The float64 danger zone for exp, and the two standard escapes: max-subtraction and log-sum-exp.
import numpy as np

# Naive softmax on large logits
z = np.array([1000.0, 1001.0, 1002.0])
with np.errstate(over="ignore"):
    naive = np.exp(z) / np.exp(z).sum()
print("naive:", naive)          # [nan nan nan]  -- inf/inf

# Max-subtracted version: identical maths, stable
zs = z - z.max()
safe = np.exp(zs) / np.exp(zs).sum()
print("safe :", np.round(safe, 6))    # [0.09 0.2447 0.6652]

# log-sum-exp: compute log(Σ exp(x)) without exponentiating the max
def logsumexp(x):
    m = np.max(x)
    return m + np.log(np.exp(x - m).sum())

print("logsumexp:", round(logsumexp(np.array([1000.0, 1001.0, 1002.0])), 4))
# 1002.4076 -- finite, although exp(1002) is inf

Both tricks rest on the same identity: subtracting a constant from every exponent is equivalent to dividing the whole expression by eᶜ, and a common factor cancels in a ratio. Choosing c = max(z) guarantees every exponent is ≤ 0, so no exp can overflow.

8. Worked Examples

8.1 Collapsing a product to a sum

A model assigns probabilities 0.5, 0.25, 0.125 to three independent tokens. Compute the sequence probability and its negative log.

P = 0.5 × 0.25 × 0.125 = 0.015625
−ln P = −(ln 0.5 + ln 0.25 + ln 0.125) = 0.693 + 1.386 + 2.079 = 4.159

8.2 Converting bits to nats

An entropy of 3 bits per token is how many nats?

1 bit  = ln 2 ≈ 0.6931 nats
3 bits = 3 × 0.6931 = 2.079 nats

8.3 Reading a perplexity

A language model reports perplexity 20. What average negative log-likelihood does that correspond to?

PPL = exp(NLL)  →  NLL = ln(20) = 2.996 nats/token
In bits: 2.996 / 0.6931 ≈ 4.32 bits per token
Interpretation: the model is as uncertain as choosing uniformly among 20 options.

8.4 Why the max-shift is exact

Show that softmax(z) = softmax(z − c) for any constant c.

e^(zᵢ−c) / Σⱼ e^(zⱼ−c)
  = (e^zᵢ · e^(−c)) / (Σⱼ e^zⱼ · e^(−c))
  = e^zᵢ / Σⱼ e^zⱼ        the factor e^(−c) cancels

8.5 Information content of a rare event

A token with probability 10⁻⁵ carries how many bits?

−log₂(10⁻⁵) = 5 × log₂ 10 = 5 × 3.3219 ≈ 16.6 bits

9. Verify It in Python

import numpy as np

# --- 1. Log laws hold ----------------------------------------------
x, y = 8.0, 32.0
assert np.isclose(np.log(x * y), np.log(x) + np.log(y))
assert np.isclose(np.log(x / y), np.log(x) - np.log(y))
assert np.isclose(np.log(x ** 3), 3 * np.log(x))
print("log laws verified")

# --- 2. Base conversion --------------------------------------------
bits = -np.log2(1e-3)
nats = -np.log(1e-3)
assert np.isclose(bits, nats / np.log(2))
print(f"p=1e-3 -> {bits:.4f} bits = {nats:.4f} nats")

# --- 3. Softmax invariants -----------------------------------------
def softmax(z):
    z = np.asarray(z, float)
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()

z = np.array([2.0, 1.0, 0.1, -1.0])
p = softmax(z)
assert np.isclose(p.sum(), 1.0)
assert np.all(p > 0)
assert np.isclose(p[0] / p[1], np.e)          # gap of 1.0 -> odds e
assert np.allclose(p, softmax(z + 100.0))     # shift invariance
print("softmax:", np.round(p, 4), "sum =", p.sum())

# --- 4. Naive softmax fails, shifted version survives --------------
big = np.array([1000.0, 1001.0, 1002.0])
with np.errstate(over="ignore", invalid="ignore"):
    naive = np.exp(big) / np.exp(big).sum()
assert not np.all(np.isfinite(naive))         # nan
print("naive on big logits:", naive)
print("safe  on big logits:", np.round(softmax(big), 6))

# --- 5. log-sum-exp stays finite -----------------------------------
def logsumexp(a):
    m = np.max(a)
    return m + np.log(np.exp(a - m).sum())

val = logsumexp(big)
assert np.isfinite(val)
assert np.isclose(val, np.log(np.exp(big - 1002.0).sum()) + 1002.0)
print("logsumexp:", round(val, 4))

# --- 6. Cross-entropy and perplexity -------------------------------
q = np.array([0.7, 0.2, 0.1])
nll = -np.log(q[0])
print("NLL for true class 0:", round(nll, 4))
print("perplexity of uniform-3:", round(np.exp(-np.log(1/3)), 4))   # 3.0
print("all assertions passed")

10. Practice Questions

  1. Simplify (2³)² · 2⁻⁴ and verify the result with logarithms.
  2. Express log₂(64), ln(e³), and log₁₀(0.01) without a calculator.
  3. Why must the base of exp in softmax match the base used in the cross-entropy that follows it?
  4. Prove that softmax is invariant to adding a constant to all logits.
  5. A model outputs logits [10, 9]. Compute the probability of the first class; then recompute after adding 1000 to both.
  6. Explain why −log p is unbounded as p → 0, and what that implies for label smoothing.
  7. Convert a perplexity of 8 into bits per token.
  8. State the smallest and largest x for which np.exp(x) is finite in float64.
  9. Why does log-sum-exp avoid overflow even when every input exceeds 700?
  10. A loss curve is a straight line on a log-y plot. Describe the underlying decay.

Leave a Reply

Your email address will not be published. Required fields are marked *