Chapter 7: Scalars, Vectors & Vector Spaces

Chapter 7: Scalars, Vectors & Vector Spaces

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 7 of 48

A vector is the fundamental data object of machine learning. An embedding is a vector, a feature row is a vector, a bias is a vector, a gradient is a vector, and every activation in a transformer is a vector in some space. This chapter establishes what a vector space actually requires, why a basis is the smallest sufficient coordinate system, why dimension is not the same as the number of stored values, and why the geometry of embedding space is what makes semantic arithmetic work.

1. Technical Audit & Core Learning Objectives

Vector chapters usually list axioms and then show arrow diagrams without ever connecting the abstraction to a tensor shape. The result is a practitioner who can add arrows on paper but cannot explain why a 50,257-token vocabulary has 768 dimensions.

Component Common Tutorial Deficiency Engineering Standard Applied Here
Vector anatomy Components shown, magnitude and direction treated as separate topics. Magnitude and direction presented as the two facts a vector always carries simultaneously.
Axioms Ten axioms listed with no reason to care. Closure framed as the property that makes every later operation well-defined.
Span / basis Defined formally, never applied to a model. Applied to embeddings, low-rank adapters, and weight matrices.
Dimension Confused with “number of entries in the table”. Vocabulary size vs embedding dimension separated explicitly.

Core learning objectives. After this chapter you should be able to:

  1. Compute a vector’s magnitude and direction from its components.
  2. State the ten vector-space axioms and explain why closure is the load-bearing one.
  3. Decide whether a given set of vectors spans a space and whether it is independent.
  4. Distinguish a basis from a spanning set, and dimension from stored count.
  5. Explain why embedding dimension d is not the vocabulary size.
  6. Read semantic vector arithmetic as a statement about directions.

2. Anatomy: Magnitude and Direction

Vector from the origin to point (3,4) with components and magnitude annotated
Figure 7.1: A vector in ℝ² with its two components, its magnitude, and its direction stated separately.

A vector v ∈ ℝⁿ is an ordered n-tuple of real numbers. Two equivalent readings exist and both are used constantly:

  • Coordinates — a position, or a point in space.
  • Displacement — an arrow from the origin, carrying magnitude and direction.
import numpy as np

v = np.array([3.0, 4.0])
print("norms:", np.linalg.norm(v, 1), np.linalg.norm(v), np.linalg.norm(v, np.inf))
# 7.0  5.0  4.0

mag = np.linalg.norm(v)
direction = v / mag                     # unit vector, length 1
print("unit vector:", direction)        # [0.6 0.8]
print("angle (deg) :", np.degrees(np.arctan2(v[1], v[0])))   # 53.13

Normalising to unit length keeps the direction and discards the magnitude. This operation appears everywhere — cosine similarity, layer normalisation, and gradient clipping are all normalisations with different motivations but the same arithmetic.

3. The Ten Axioms, and Why Closure Matters

Ten vector space axioms with plain language descriptions
Figure 7.2: The ten axioms. Only the first two are closure; the remaining eight are bookkeeping.

A vector space is a set V of objects with two operations — addition and scalar multiplication — satisfying ten rules. Eight of them are so obvious that they are automatically true for anything you would call a vector. The two that carry weight are:

  1. Closure under addition. If u, v ∈ V then u + v ∈ V.
  2. Closure under scaling. If v ∈ V and c ∈ ℝ then c·v ∈ V.

Closure is what makes the space a self-contained world. Without it, adding two members could produce something outside the set and every later argument breaks. A concrete failure: the set of vectors with all-positive entries is not a vector space, because scaling by −1 leaves the set. That single fact is why probability distributions are points in a simplex rather than a vector space, and why operations on them need renormalisation.

Closure also explains why ReLU does not preserve a space: the set of outputs is not closed under negation, so a ReLU layer is a nonlinear map between vector spaces rather than a linear one within a vector space.

4. Span and Linear Independence

Two independent basis vectors spanning the whole plane, with a linear combination shown
Figure 7.3: The span of two independent vectors fills the plane. Any third vector in that plane is already reachable.

The span of a set of vectors is every linear combination you can form from them. Two independent vectors in the plane span the whole plane; a single vector spans only a line.

span{v₁}          = { c·v₁ : c ∈ ℝ }            a line
span{v₁, v₂}      = { c₁v₁ + c₂v₂ }             a plane (if independent)
span{v₁, 2v₁}     = { c₁v₁ + 2c₂v₁ } = span{v₁}  still a line — dependent

Linear independence means no vector in the set can be written as a combination of the others. Equivalently, c₁v₁ + … + cₖvₖ = 0 has only the trivial solution cᵢ = 0. Equivalently again, the matrix with these vectors as columns has full column rank.

import numpy as np

v1 = np.array([1.0, 0.0])
v2 = np.array([0.0, 1.0])
v3 = np.array([2.0, 3.0])          # reachable from v1 and v2

A = np.column_stack([v1, v2])
B = np.column_stack([v1, v2, v3])
print("rank with 2 vectors:", np.linalg.matrix_rank(A))    # 2
print("rank with 3 vectors:", np.linalg.matrix_rank(B))    # 2 -- v3 adds nothing

# dependence: is v3 in the span of v1, v2?
coeff, *_ = np.linalg.lstsq(A, v3, rcond=None)
print("v3 == 2*v1 + 3*v2 ->", np.allclose(A @ coeff, v3), coeff)

5. Basis and Dimension

Standard basis of R3 and the definition of dimension, with an embedding-table contrast
Figure 7.4: A basis is minimal and sufficient. Dimension is the size of any basis.

A basis of a vector space is a set that both spans the space and is linearly independent — minimal and sufficient at once. Remove a vector and the span shrinks; add one and it becomes redundant. The dimension is the number of vectors in any basis.

Object Dimension Why
Word embedding table d = 768 Each row lives in ℝ⁷⁶⁸; there are 50,257 rows
One-hot class vector k (number of classes) Standard basis of ℝᵏ
Image patch [C,H,W] C·H·W Flattening is a change of basis, not a change of information
Low-rank adapter r ≪ d LoRA constrains updates to a low-dimensional subspace

The table’s second row is the one that trips people up. An embedding table with 50,257 tokens and 768 dimensions is a matrix of shape (50257, 768). Its rows are points in a 768-dimensional space. The vocabulary size says how many points exist; the embedding dimension says how many coordinates each point has. Conflating the two produces the classic error of describing a model as “50,257-dimensional”.

LoRA is a basis argument in disguise: instead of updating all d×d weights, it constrains the update ΔW to BA with B ∈ ℝ^(d×r), so the update lies in an r-dimensional subspace. The claim that this suffices is a claim about the intrinsic dimension of fine-tuning updates.

6. Embedding Space Is a Metric Space

Word embeddings plotted in two dimensions showing a consistent gender direction between man-woman and king-queen
Figure 7.5: Semantic directions. The offset between man and woman is approximately the same as between king and queen.

Once every word is a vector, meaning can be manipulated by arithmetic — but only because the space has structure. The famous identity:

v_king − v_man + v_woman ≈ v_queen

states that the “gender” offset is a consistent direction, independent of the semantic axis. This is not a property of vectors in general; it is an empirical property of spaces learned by gradient descent on co-occurrence statistics. The mathematics permits it because directions compose additively.

In practice you use cosine similarity rather than raw distance, because embedding magnitudes vary with word frequency while directions carry the semantics. Chapter 8 develops the dot product; Chapter 11 compares the metrics properly.

7. Worked Examples

7.1 Normalise and interpret

v = (3, 4)      ‖v‖ = 5
v̂ = (0.6, 0.8)  unit vector, same direction

7.2 Test independence

Are (1,2,3), (2,4,6), (1,0,1) independent?

The second is exactly 2 × the first  →  dependent set.
Rank is 2, not 3, so they span only a plane in ℝ³.

7.3 Dimension of a basis

What is the dimension of span{(1,0,0), (0,1,0), (1,1,0)}?

The third is the sum of the first two  →  rank 2.
A basis is {(1,0,0), (0,1,0)}; dimension = 2.

7.4 Why positivity breaks a vector space

Let V = {(x,y) : x > 0, y > 0}. Is V a vector space?

No. Take v = (1,1) ∈ V and c = −1.
c·v = (−1,−1) ∉ V  →  closure under scaling fails.

7.5 Flattening changes basis, not information

A (3,224,224) image has 150,528 entries. Flattened it is a vector in ℝ^150528. No information is added or lost; only the coordinate description changed.

8. Verify It in Python

import numpy as np

# --- 1. magnitude and direction are separable ---------------------
v = np.array([3.0, 4.0])
assert np.isclose(np.linalg.norm(v), 5.0)
u = v / np.linalg.norm(v)
assert np.isclose(np.linalg.norm(u), 1.0) and np.allclose(u * 5, v)
print("unit vector:", u)

# --- 2. closure failure: positives are not a subspace -------------
p = np.array([1.0, 2.0])
assert not np.all((-1.0 * p) > 0)
print("scaling flipped the sign -> positivity is not closed")

# --- 3. dependent vectors do not raise rank ----------------------
A = np.column_stack([[1., 0.], [0., 1.]])
B = np.column_stack([[1., 0.], [0., 1.], [2., 3.]])
assert np.linalg.matrix_rank(A) == 2 and np.linalg.matrix_rank(B) == 2
print("rank A, B:", np.linalg.matrix_rank(A), np.linalg.matrix_rank(B))

# --- 4. a duplicate column is redundant --------------------------
C = np.column_stack([[1., 2., 3.], [2., 4., 6.], [1., 0., 1.]])
assert np.linalg.matrix_rank(C) == 2
print("rank of dependent 3x3:", np.linalg.matrix_rank(C))

# --- 5. vocabulary size is not dimension -------------------------
n_vocab, d_model = 50257, 768
E = np.zeros((n_vocab, d_model))
assert E.shape == (50257, 768)
print(f"{n_vocab} tokens, each a point in R^{d_model}")

# --- 6. low-rank update lives in a subspace ----------------------
d, r = 64, 4
Bm = np.random.randn(d, r); Am = np.random.randn(r, d)
dW = Bm @ Am
assert np.linalg.matrix_rank(dW) <= r
print("rank of BA:", np.linalg.matrix_rank(dW), "<= r =", r)
print("all assertions passed")

9. Practice Questions

  1. Compute the magnitude and unit vector of (−5, 12).
  2. State which two axioms are closure, and why the other eight are automatic for real tuples.
  3. Show that {(1,1), (1,−1)} is a basis of ℝ².
  4. Is {(1,2), (2,4), (3,6)} independent? Give the rank and the dimension of its span.
  5. Why is the set of all 3×3 symmetric matrices a vector space, while the set of all invertible 3×3 matrices is not?
  6. Explain why a 50,257-token embedding table is not "50,257-dimensional".
  7. Let V be the span of (1,0,1) and (0,1,1). Give a vector in ℝ³ that is not in V.
  8. What is the dimension of the space spanned by one-hot vectors of length 10?
  9. Explain LoRA as a basis statement.
  10. Why does flattening a CHW tensor not change its information content?

Leave a Reply

Your email address will not be published. Required fields are marked *