AI Reading
Quick summary of this article
AI Mathematics Masterclass — Chapter 16 of 48
A tensor is just an array with a rank, and rank is just a count of axes. The difficulty is never the definition — it is the bookkeeping. Which axis is the batch? Which is the feature? What does axis=1 actually remove? This chapter makes axes and broadcasting fully explicit, then shows that every shape annotation in a transformer — (B, T, d) splitting into (B, H, T, d/H) and back — is one of a small set of split and merge operations you can name precisely.
1. Technical Audit & Core Learning Objectives
| Component | Common Tutorial Deficiency | Engineering Standard Applied Here |
|---|---|---|
| Rank and axes | Shapes given as tuples with no named meaning. | Every axis named and its role stated. |
| Broadcasting | Called “numpy magic” or explained only by example. | Right-alignment rule stated, then verified against real shapes. |
| Axis reductions | axis= described as “the direction” — ambiguous. |
Defined as the axis that is removed, with keepdims explained. |
| Transformer shapes | Presented as a wall of unfamiliar tuples. | Reduced to explicit split/merge steps. |
Core learning objectives. After this chapter you should be able to:
- State the rank, shape, and axis meaning of any tensor in a model.
- Predict the result of broadcasting two shapes, or identify them as incompatible.
- Compute
sum,mean, andmaxalong any axis and predict the output shape. - Use
keepdims=Trueto preserve broadcasting alignment. - Perform reshape, transpose, and axis insertion deliberately.
- Decompose an attention shape into its split and merge steps.
2. Rank Is a Count of Axes
import numpy as np
scalar = np.array(3.0)
vector = np.zeros(4)
matrix = np.zeros((3, 4))
batch = np.zeros((32, 3, 224, 224))
for name, t in [("scalar", scalar), ("vector", vector), ("matrix", matrix), ("batch", batch)]:
print(f"{name:8s} shape={str(t.shape):22s} ndim={t.ndim} size={t.size}")
| Shape | Axes Named | Convention |
|---|---|---|
(B, T, d) |
batch, sequence length, model width | Transformer activations |
(B, H, T, d/H) |
batch, heads, sequence, per-head width | Inside attention |
(N, C, H, W) |
batch, channels, height, width | PyTorch convolution (NCHW) |
(N, H, W, C) |
batch, height, width, channels | TensorFlow / channels-last |
The NCHW versus NHWC difference is a pure axis-ordering convention, and mixing them across libraries is a common source of silently wrong results — the shapes are compatible enough to run, but every convolution sees the wrong axis as channels.
3. Broadcasting: Align From the Right
Two axes are compatible if:
they are equal, or
one of them is 1 → that axis is stretched
Compare from the RIGHT. Missing leading axes count as 1.
A: (3, 1, 4)
B: (2, 1)
aligned: (3,1,4)
(1,2,1) ← padded on the left with 1s
result: (3,2,4)
No data is copied; the stride on a stretched axis is 0.
import numpy as np
A = np.zeros((3, 1, 4))
B = np.zeros((2, 1))
print("A + B shape:", (A + B).shape) # (3, 2, 4)
C = np.zeros((3, 4))
D = np.zeros((4,))
print("C + D shape:", (C + D).shape) # (3, 4) -- D broadcasts across rows
try:
np.zeros((3, 4)) + np.zeros((3,))
except ValueError as exc:
print("incompatible:", str(exc)[:70])
The failure case is worth internalising: (3,4) + (3,) aligns 3 against 4, which is neither equal nor 1, so it raises. To add a per-row vector you need (3,1), which aligns as (1,3) against (3,4) — compatible. That is what keepdims=True gives you for free.
4. Axis Reductions
axis=k removes axis k. keepdims keeps a size-1 placeholder instead.import numpy as np
X = np.array([[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0]]) # (2, 3)
print("axis=0 ->", X.sum(axis=0), X.sum(axis=0).shape) # per column, (3,)
print("axis=1 ->", X.sum(axis=1), X.sum(axis=1).shape) # per row, (2,)
print("axis=-1->", X.sum(axis=-1).shape) # last axis, (2,)
print("all ->", X.sum(), X.sum().shape) # scalar, ()
keep = X.sum(axis=1, keepdims=True)
print("keepdims:", keep.shape) # (2, 1)
print("broadcast back:", (X / keep).shape) # (2, 3)
axis=-1 is the idiomatic way to say “the last axis” without hard-coding its index, and it is what almost every transformer implementation uses — because it means the same thing regardless of how many leading batch axes exist.
5. Splitting and Merging Axes
import numpy as np
B, T, d, H = 2, 8, 64, 4
x = np.random.rand(B, T, d)
# split the model axis d into (H, d/H) -- this INSERTS an axis
q = x.reshape(B, T, H, d // H).transpose(0, 2, 1, 3)
print("after split :", q.shape) # (B, H, T, d/H)
# scores: contract the last axis only
scores = q @ q.swapaxes(-1, -2)
print("scores :", scores.shape) # (B, H, T, T)
# merge heads back: transpose then reshape
merged = q.transpose(0, 2, 1, 3).reshape(B, T, d)
print("after merge :", merged.shape) # (B, T, d)
assert np.allclose(merged, x)
print("split then merge is lossless")
Two rules cover nearly all of it. A split is reshape that introduces a new axis, which always requires the following transpose to put it where you want. A merge is transpose followed by reshape, in that order — reshaping first would interleave the wrong elements.
6. Einstein Notation as General Shape Control
import numpy as np
A = np.random.rand(3, 4)
B = np.random.rand(4, 5)
print("transpose :", np.einsum("ij->ji", A).shape) # (4, 3)
print("total sum :", np.einsum("ij->", A).shape) # ()
print("matmul :", np.allclose(np.einsum("ij,jk->ik", A, B), A @ B))
Q = np.random.rand(2, 4, 8, 16)
K = np.random.rand(2, 4, 8, 16)
S = np.einsum("bhid,bhjd->bhij", Q, K)
print("attention :", S.shape) # (2, 4, 8, 8)
assert S.shape == (2, 4, 8, 8)
Writing the einsum string forces you to name every axis and state whether it survives. That act alone catches most shape errors before they become silent broadcast bugs.
7. Worked Examples
7.1 Predict a broadcast
(5, 1, 3) + (4, 3)
aligned: (5, 1, 3)
(1, 4, 3)
result: (5, 4, 3)
7.2 Reduce a batch mean
X is (32, 128):
X.mean(axis=0) → (128,) mean over samples, per feature
X.mean(axis=1) → (32,) mean over features, per sample
7.3 Fix an axis misalignment
Want to subtract a per-feature mean from X (32, 128):
mu = X.mean(axis=0) shape (128,)
X - mu works: (32,128) - (128,) aligns fine
But to subtract a per-sample mean you need (32,1):
mu2 = X.mean(axis=1, keepdims=True) shape (32,1)
X - mu2
7.4 Insert a middle axis
(8,) → (8,1) x[:, None]
(3,8) → (3,1,8) x[:, None, :]
(3,8) → (3,8,1) x[..., None]
7.5 Count elements
(32, 3, 224, 224) → 32·3·224·224 = 4,816,896
bytes in float32 → 19,267,584 ≈ 18.4 MB per batch
8. Verify It in Python
import numpy as np
# --- 1. rank == len(shape) ----------------------------------------
for t in (np.array(1.0), np.zeros(4), np.zeros((3, 4)), np.zeros((2, 2, 2))):
assert t.ndim == len(t.shape)
print("ndim == len(shape) for all")
# --- 2. broadcasting rules ----------------------------------------
assert (np.zeros((3, 1, 4)) + np.zeros((2, 1))).shape == (3, 2, 4)
assert (np.zeros((3, 4)) + np.zeros((4,))).shape == (3, 4)
try:
np.zeros((3, 4)) + np.zeros((3,))
raise AssertionError("should have failed")
except ValueError:
print("incompatible shapes raise")
# --- 3. axis removal ----------------------------------------------
X = np.arange(6.0).reshape(2, 3)
assert X.sum(axis=0).shape == (3,)
assert X.sum(axis=1).shape == (2,)
assert X.sum().shape == ()
print("axis=k removes axis k")
# --- 4. keepdims preserves alignment ------------------------------
k = X.sum(axis=1, keepdims=True)
assert k.shape == (2, 1)
assert np.allclose(X / k, X / X.sum(axis=1, keepdims=True))
print("keepdims broadcasting works")
# --- 5. split then merge is lossless ------------------------------
B, T, d, H = 2, 8, 64, 4
x = np.random.rand(B, T, d)
q = x.reshape(B, T, H, d // H).transpose(0, 2, 1, 3)
assert q.shape == (B, H, T, d // H)
back = q.transpose(0, 2, 1, 3).reshape(B, T, d)
assert np.allclose(back, x)
print("head split/merge round-trips")
# --- 6. attention scores contract only the last axis --------------
scores = q @ q.swapaxes(-1, -2)
assert scores.shape == (B, H, T, T)
print("scores:", scores.shape)
# --- 7. einsum matches primitives --------------------------------
A = np.random.rand(3, 4); Bm = np.random.rand(4, 5)
assert np.allclose(np.einsum("ij,jk->ik", A, Bm), A @ Bm)
assert np.allclose(np.einsum("ij->ji", A), A.T)
assert np.isclose(np.einsum("ij->", A), A.sum())
print("einsum reproduces transpose, sum, matmul")
# --- 8. element count and memory ---------------------------------
batch = np.zeros((32, 3, 224, 224), dtype=np.float32)
assert batch.size == 32 * 3 * 224 * 224
print("batch elements:", batch.size, " bytes:", batch.nbytes)
print("all assertions passed")
9. Practice Questions
- State the rank and the meaning of each axis for
(32, 3, 224, 224). - Broadcast
(5,1,3)with(4,3)and give the result shape. - Why does
(3,4) + (3,)fail while(3,4) + (3,1)succeeds? - Give the output shapes of
sumonaxis=0,1, and-1for a(2,3)array. - When is
keepdims=Truenecessary rather than merely convenient? - Show the reshape + transpose that turns
(B,T,d)into(B,H,T,d/H). - Why must a merge be transpose-then-reshape rather than reshape-then-transpose?
- Translate
Σ_d Q[b,h,i,d]K[b,h,j,d]into an einsum string. - Compute the float32 memory for a
(16, 12, 2048, 64)attention activation. - NCHW and NHWC — which axes differ, and what breaks if you mix them?
10. Official References & Further Reading
Continue the Course
- Course Index: AI Mathematics Masterclass — All 48 Chapters
- Previous Chapter: Chapter 15: Rank, Null Space & Solvability of Linear Systems
- Next Chapter: Chapter 17: Determinants, Inverses & Numerical Conditioning
- Related Series: PyTorch Engineering Masterclass — All 16 Modules
