Chapter 16: Tensors, Axes & Broadcasting Mathematics

Chapter 16: Tensors, Axes & Broadcasting Mathematics

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 16 of 48

A tensor is just an array with a rank, and rank is just a count of axes. The difficulty is never the definition — it is the bookkeeping. Which axis is the batch? Which is the feature? What does axis=1 actually remove? This chapter makes axes and broadcasting fully explicit, then shows that every shape annotation in a transformer — (B, T, d) splitting into (B, H, T, d/H) and back — is one of a small set of split and merge operations you can name precisely.

1. Technical Audit & Core Learning Objectives

Component Common Tutorial Deficiency Engineering Standard Applied Here
Rank and axes Shapes given as tuples with no named meaning. Every axis named and its role stated.
Broadcasting Called “numpy magic” or explained only by example. Right-alignment rule stated, then verified against real shapes.
Axis reductions axis= described as “the direction” — ambiguous. Defined as the axis that is removed, with keepdims explained.
Transformer shapes Presented as a wall of unfamiliar tuples. Reduced to explicit split/merge steps.

Core learning objectives. After this chapter you should be able to:

  1. State the rank, shape, and axis meaning of any tensor in a model.
  2. Predict the result of broadcasting two shapes, or identify them as incompatible.
  3. Compute sum, mean, and max along any axis and predict the output shape.
  4. Use keepdims=True to preserve broadcasting alignment.
  5. Perform reshape, transpose, and axis insertion deliberately.
  6. Decompose an attention shape into its split and merge steps.

2. Rank Is a Count of Axes

Scalar, vector, matrix and rank-3 tensor with their shape tuples named
Figure 16.1: Rank is the length of the shape tuple. Each element is one axis.
import numpy as np

scalar = np.array(3.0)
vector = np.zeros(4)
matrix = np.zeros((3, 4))
batch  = np.zeros((32, 3, 224, 224))

for name, t in [("scalar", scalar), ("vector", vector), ("matrix", matrix), ("batch", batch)]:
    print(f"{name:8s} shape={str(t.shape):22s} ndim={t.ndim}  size={t.size}")
Shape Axes Named Convention
(B, T, d) batch, sequence length, model width Transformer activations
(B, H, T, d/H) batch, heads, sequence, per-head width Inside attention
(N, C, H, W) batch, channels, height, width PyTorch convolution (NCHW)
(N, H, W, C) batch, height, width, channels TensorFlow / channels-last

The NCHW versus NHWC difference is a pure axis-ordering convention, and mixing them across libraries is a common source of silently wrong results — the shapes are compatible enough to run, but every convolution sees the wrong axis as channels.

3. Broadcasting: Align From the Right

Broadcasting (3,1,4) against (2,1) aligned from the right producing (3,2,4)
Figure 16.2: Align trailing axes, then stretch any size-1 axis to match.
Two axes are compatible if:
   they are equal, or
   one of them is 1                      → that axis is stretched

Compare from the RIGHT. Missing leading axes count as 1.

  A: (3, 1, 4)
  B:       (2, 1)
  aligned:  (3,1,4)
            (1,2,1)      ← padded on the left with 1s
  result:   (3,2,4)

No data is copied; the stride on a stretched axis is 0.
import numpy as np

A = np.zeros((3, 1, 4))
B = np.zeros((2, 1))
print("A + B shape:", (A + B).shape)          # (3, 2, 4)

C = np.zeros((3, 4))
D = np.zeros((4,))
print("C + D shape:", (C + D).shape)          # (3, 4) -- D broadcasts across rows

try:
    np.zeros((3, 4)) + np.zeros((3,))
except ValueError as exc:
    print("incompatible:", str(exc)[:70])

The failure case is worth internalising: (3,4) + (3,) aligns 3 against 4, which is neither equal nor 1, so it raises. To add a per-row vector you need (3,1), which aligns as (1,3) against (3,4) — compatible. That is what keepdims=True gives you for free.

4. Axis Reductions

A 2x3 matrix reduced along axis 0 and axis 1 with resulting shapes shown
Figure 16.3: axis=k removes axis k. keepdims keeps a size-1 placeholder instead.
import numpy as np

X = np.array([[1.0, 2.0, 3.0],
              [4.0, 5.0, 6.0]])          # (2, 3)

print("axis=0 ->", X.sum(axis=0), X.sum(axis=0).shape)   # per column, (3,)
print("axis=1 ->", X.sum(axis=1), X.sum(axis=1).shape)   # per row,    (2,)
print("axis=-1->", X.sum(axis=-1).shape)                 # last axis,  (2,)
print("all    ->", X.sum(), X.sum().shape)               # scalar,     ()

keep = X.sum(axis=1, keepdims=True)
print("keepdims:", keep.shape)                           # (2, 1)
print("broadcast back:", (X / keep).shape)               # (2, 3)

axis=-1 is the idiomatic way to say “the last axis” without hard-coding its index, and it is what almost every transformer implementation uses — because it means the same thing regardless of how many leading batch axes exist.

5. Splitting and Merging Axes

Attention pipeline from (B,T,d) through head splitting to (B,H,T,d) with shapes at each stage
Figure 16.4: The attention pipeline as a sequence of named shape operations.
import numpy as np

B, T, d, H = 2, 8, 64, 4
x = np.random.rand(B, T, d)

# split the model axis d into (H, d/H) -- this INSERTS an axis
q = x.reshape(B, T, H, d // H).transpose(0, 2, 1, 3)
print("after split :", q.shape)                 # (B, H, T, d/H)

# scores: contract the last axis only
scores = q @ q.swapaxes(-1, -2)
print("scores      :", scores.shape)            # (B, H, T, T)

# merge heads back: transpose then reshape
merged = q.transpose(0, 2, 1, 3).reshape(B, T, d)
print("after merge :", merged.shape)            # (B, T, d)
assert np.allclose(merged, x)
print("split then merge is lossless")

Two rules cover nearly all of it. A split is reshape that introduces a new axis, which always requires the following transpose to put it where you want. A merge is transpose followed by reshape, in that order — reshaping first would interleave the wrong elements.

6. Einstein Notation as General Shape Control

Einsum expressions mapped to transpose, sum, diagonal, matmul and attention
Figure 16.5: One notation for every contraction. Indices absent from the output are summed.
import numpy as np

A = np.random.rand(3, 4)
B = np.random.rand(4, 5)

print("transpose :", np.einsum("ij->ji", A).shape)      # (4, 3)
print("total sum :", np.einsum("ij->", A).shape)        # ()
print("matmul    :", np.allclose(np.einsum("ij,jk->ik", A, B), A @ B))

Q = np.random.rand(2, 4, 8, 16)
K = np.random.rand(2, 4, 8, 16)
S = np.einsum("bhid,bhjd->bhij", Q, K)
print("attention :", S.shape)                          # (2, 4, 8, 8)
assert S.shape == (2, 4, 8, 8)

Writing the einsum string forces you to name every axis and state whether it survives. That act alone catches most shape errors before they become silent broadcast bugs.

7. Worked Examples

7.1 Predict a broadcast

(5, 1, 3) + (4, 3)
aligned:  (5, 1, 3)
          (1, 4, 3)
result:   (5, 4, 3)

7.2 Reduce a batch mean

X is (32, 128):
X.mean(axis=0) → (128,)  mean over samples, per feature
X.mean(axis=1) → (32,)   mean over features, per sample

7.3 Fix an axis misalignment

Want to subtract a per-feature mean from X (32, 128):
mu = X.mean(axis=0)          shape (128,)
X - mu                       works: (32,128) - (128,) aligns fine
But to subtract a per-sample mean you need (32,1):
mu2 = X.mean(axis=1, keepdims=True)   shape (32,1)
X - mu2

7.4 Insert a middle axis

(8,) → (8,1)      x[:, None]
(3,8) → (3,1,8)   x[:, None, :]
(3,8) → (3,8,1)   x[..., None]

7.5 Count elements

(32, 3, 224, 224) → 32·3·224·224 = 4,816,896
bytes in float32  → 19,267,584 ≈ 18.4 MB per batch

8. Verify It in Python

import numpy as np

# --- 1. rank == len(shape) ----------------------------------------
for t in (np.array(1.0), np.zeros(4), np.zeros((3, 4)), np.zeros((2, 2, 2))):
    assert t.ndim == len(t.shape)
print("ndim == len(shape) for all")

# --- 2. broadcasting rules ----------------------------------------
assert (np.zeros((3, 1, 4)) + np.zeros((2, 1))).shape == (3, 2, 4)
assert (np.zeros((3, 4)) + np.zeros((4,))).shape == (3, 4)
try:
    np.zeros((3, 4)) + np.zeros((3,))
    raise AssertionError("should have failed")
except ValueError:
    print("incompatible shapes raise")

# --- 3. axis removal ----------------------------------------------
X = np.arange(6.0).reshape(2, 3)
assert X.sum(axis=0).shape == (3,)
assert X.sum(axis=1).shape == (2,)
assert X.sum().shape == ()
print("axis=k removes axis k")

# --- 4. keepdims preserves alignment ------------------------------
k = X.sum(axis=1, keepdims=True)
assert k.shape == (2, 1)
assert np.allclose(X / k, X / X.sum(axis=1, keepdims=True))
print("keepdims broadcasting works")

# --- 5. split then merge is lossless ------------------------------
B, T, d, H = 2, 8, 64, 4
x = np.random.rand(B, T, d)
q = x.reshape(B, T, H, d // H).transpose(0, 2, 1, 3)
assert q.shape == (B, H, T, d // H)
back = q.transpose(0, 2, 1, 3).reshape(B, T, d)
assert np.allclose(back, x)
print("head split/merge round-trips")

# --- 6. attention scores contract only the last axis --------------
scores = q @ q.swapaxes(-1, -2)
assert scores.shape == (B, H, T, T)
print("scores:", scores.shape)

# --- 7. einsum matches primitives --------------------------------
A = np.random.rand(3, 4); Bm = np.random.rand(4, 5)
assert np.allclose(np.einsum("ij,jk->ik", A, Bm), A @ Bm)
assert np.allclose(np.einsum("ij->ji", A), A.T)
assert np.isclose(np.einsum("ij->", A), A.sum())
print("einsum reproduces transpose, sum, matmul")

# --- 8. element count and memory ---------------------------------
batch = np.zeros((32, 3, 224, 224), dtype=np.float32)
assert batch.size == 32 * 3 * 224 * 224
print("batch elements:", batch.size, " bytes:", batch.nbytes)
print("all assertions passed")

9. Practice Questions

  1. State the rank and the meaning of each axis for (32, 3, 224, 224).
  2. Broadcast (5,1,3) with (4,3) and give the result shape.
  3. Why does (3,4) + (3,) fail while (3,4) + (3,1) succeeds?
  4. Give the output shapes of sum on axis=0, 1, and -1 for a (2,3) array.
  5. When is keepdims=True necessary rather than merely convenient?
  6. Show the reshape + transpose that turns (B,T,d) into (B,H,T,d/H).
  7. Why must a merge be transpose-then-reshape rather than reshape-then-transpose?
  8. Translate Σ_d Q[b,h,i,d]K[b,h,j,d] into an einsum string.
  9. Compute the float32 memory for a (16, 12, 2048, 64) attention activation.
  10. NCHW and NHWC — which axes differ, and what breaks if you mix them?

Leave a Reply

Your email address will not be published. Required fields are marked *