AI Reading
Quick summary of this article
AI Mathematics Masterclass — Chapter 7 of 48
A vector is the fundamental data object of machine learning. An embedding is a vector, a feature row is a vector, a bias is a vector, a gradient is a vector, and every activation in a transformer is a vector in some space. This chapter establishes what a vector space actually requires, why a basis is the smallest sufficient coordinate system, why dimension is not the same as the number of stored values, and why the geometry of embedding space is what makes semantic arithmetic work.
1. Technical Audit & Core Learning Objectives
Vector chapters usually list axioms and then show arrow diagrams without ever connecting the abstraction to a tensor shape. The result is a practitioner who can add arrows on paper but cannot explain why a 50,257-token vocabulary has 768 dimensions.
| Component | Common Tutorial Deficiency | Engineering Standard Applied Here |
|---|---|---|
| Vector anatomy | Components shown, magnitude and direction treated as separate topics. | Magnitude and direction presented as the two facts a vector always carries simultaneously. |
| Axioms | Ten axioms listed with no reason to care. | Closure framed as the property that makes every later operation well-defined. |
| Span / basis | Defined formally, never applied to a model. | Applied to embeddings, low-rank adapters, and weight matrices. |
| Dimension | Confused with “number of entries in the table”. | Vocabulary size vs embedding dimension separated explicitly. |
Core learning objectives. After this chapter you should be able to:
- Compute a vector’s magnitude and direction from its components.
- State the ten vector-space axioms and explain why closure is the load-bearing one.
- Decide whether a given set of vectors spans a space and whether it is independent.
- Distinguish a basis from a spanning set, and dimension from stored count.
- Explain why embedding dimension
dis not the vocabulary size. - Read semantic vector arithmetic as a statement about directions.
2. Anatomy: Magnitude and Direction
A vector v ∈ ℝⁿ is an ordered n-tuple of real numbers. Two equivalent readings exist and both are used constantly:
- Coordinates — a position, or a point in space.
- Displacement — an arrow from the origin, carrying magnitude and direction.
import numpy as np
v = np.array([3.0, 4.0])
print("norms:", np.linalg.norm(v, 1), np.linalg.norm(v), np.linalg.norm(v, np.inf))
# 7.0 5.0 4.0
mag = np.linalg.norm(v)
direction = v / mag # unit vector, length 1
print("unit vector:", direction) # [0.6 0.8]
print("angle (deg) :", np.degrees(np.arctan2(v[1], v[0]))) # 53.13
Normalising to unit length keeps the direction and discards the magnitude. This operation appears everywhere — cosine similarity, layer normalisation, and gradient clipping are all normalisations with different motivations but the same arithmetic.
3. The Ten Axioms, and Why Closure Matters
A vector space is a set V of objects with two operations — addition and scalar multiplication — satisfying ten rules. Eight of them are so obvious that they are automatically true for anything you would call a vector. The two that carry weight are:
- Closure under addition. If
u, v ∈ Vthenu + v ∈ V. - Closure under scaling. If
v ∈ Vandc ∈ ℝthenc·v ∈ V.
Closure is what makes the space a self-contained world. Without it, adding two members could produce something outside the set and every later argument breaks. A concrete failure: the set of vectors with all-positive entries is not a vector space, because scaling by −1 leaves the set. That single fact is why probability distributions are points in a simplex rather than a vector space, and why operations on them need renormalisation.
Closure also explains why ReLU does not preserve a space: the set of outputs is not closed under negation, so a ReLU layer is a nonlinear map between vector spaces rather than a linear one within a vector space.
4. Span and Linear Independence
The span of a set of vectors is every linear combination you can form from them. Two independent vectors in the plane span the whole plane; a single vector spans only a line.
span{v₁} = { c·v₁ : c ∈ ℝ } a line
span{v₁, v₂} = { c₁v₁ + c₂v₂ } a plane (if independent)
span{v₁, 2v₁} = { c₁v₁ + 2c₂v₁ } = span{v₁} still a line — dependent
Linear independence means no vector in the set can be written as a combination of the others. Equivalently, c₁v₁ + … + cₖvₖ = 0 has only the trivial solution cᵢ = 0. Equivalently again, the matrix with these vectors as columns has full column rank.
import numpy as np
v1 = np.array([1.0, 0.0])
v2 = np.array([0.0, 1.0])
v3 = np.array([2.0, 3.0]) # reachable from v1 and v2
A = np.column_stack([v1, v2])
B = np.column_stack([v1, v2, v3])
print("rank with 2 vectors:", np.linalg.matrix_rank(A)) # 2
print("rank with 3 vectors:", np.linalg.matrix_rank(B)) # 2 -- v3 adds nothing
# dependence: is v3 in the span of v1, v2?
coeff, *_ = np.linalg.lstsq(A, v3, rcond=None)
print("v3 == 2*v1 + 3*v2 ->", np.allclose(A @ coeff, v3), coeff)
5. Basis and Dimension
A basis of a vector space is a set that both spans the space and is linearly independent — minimal and sufficient at once. Remove a vector and the span shrinks; add one and it becomes redundant. The dimension is the number of vectors in any basis.
| Object | Dimension | Why |
|---|---|---|
| Word embedding table | d = 768 |
Each row lives in ℝ⁷⁶⁸; there are 50,257 rows |
| One-hot class vector | k (number of classes) |
Standard basis of ℝᵏ |
Image patch [C,H,W] |
C·H·W |
Flattening is a change of basis, not a change of information |
| Low-rank adapter | r ≪ d |
LoRA constrains updates to a low-dimensional subspace |
The table’s second row is the one that trips people up. An embedding table with 50,257 tokens and 768 dimensions is a matrix of shape (50257, 768). Its rows are points in a 768-dimensional space. The vocabulary size says how many points exist; the embedding dimension says how many coordinates each point has. Conflating the two produces the classic error of describing a model as “50,257-dimensional”.
LoRA is a basis argument in disguise: instead of updating all d×d weights, it constrains the update ΔW to BA with B ∈ ℝ^(d×r), so the update lies in an r-dimensional subspace. The claim that this suffices is a claim about the intrinsic dimension of fine-tuning updates.
6. Embedding Space Is a Metric Space
Once every word is a vector, meaning can be manipulated by arithmetic — but only because the space has structure. The famous identity:
v_king − v_man + v_woman ≈ v_queen
states that the “gender” offset is a consistent direction, independent of the semantic axis. This is not a property of vectors in general; it is an empirical property of spaces learned by gradient descent on co-occurrence statistics. The mathematics permits it because directions compose additively.
In practice you use cosine similarity rather than raw distance, because embedding magnitudes vary with word frequency while directions carry the semantics. Chapter 8 develops the dot product; Chapter 11 compares the metrics properly.
7. Worked Examples
7.1 Normalise and interpret
v = (3, 4) ‖v‖ = 5
v̂ = (0.6, 0.8) unit vector, same direction
7.2 Test independence
Are (1,2,3), (2,4,6), (1,0,1) independent?
The second is exactly 2 × the first → dependent set.
Rank is 2, not 3, so they span only a plane in ℝ³.
7.3 Dimension of a basis
What is the dimension of span{(1,0,0), (0,1,0), (1,1,0)}?
The third is the sum of the first two → rank 2.
A basis is {(1,0,0), (0,1,0)}; dimension = 2.
7.4 Why positivity breaks a vector space
Let V = {(x,y) : x > 0, y > 0}. Is V a vector space?
No. Take v = (1,1) ∈ V and c = −1.
c·v = (−1,−1) ∉ V → closure under scaling fails.
7.5 Flattening changes basis, not information
A (3,224,224) image has 150,528 entries. Flattened it is a vector in ℝ^150528. No information is added or lost; only the coordinate description changed.
8. Verify It in Python
import numpy as np
# --- 1. magnitude and direction are separable ---------------------
v = np.array([3.0, 4.0])
assert np.isclose(np.linalg.norm(v), 5.0)
u = v / np.linalg.norm(v)
assert np.isclose(np.linalg.norm(u), 1.0) and np.allclose(u * 5, v)
print("unit vector:", u)
# --- 2. closure failure: positives are not a subspace -------------
p = np.array([1.0, 2.0])
assert not np.all((-1.0 * p) > 0)
print("scaling flipped the sign -> positivity is not closed")
# --- 3. dependent vectors do not raise rank ----------------------
A = np.column_stack([[1., 0.], [0., 1.]])
B = np.column_stack([[1., 0.], [0., 1.], [2., 3.]])
assert np.linalg.matrix_rank(A) == 2 and np.linalg.matrix_rank(B) == 2
print("rank A, B:", np.linalg.matrix_rank(A), np.linalg.matrix_rank(B))
# --- 4. a duplicate column is redundant --------------------------
C = np.column_stack([[1., 2., 3.], [2., 4., 6.], [1., 0., 1.]])
assert np.linalg.matrix_rank(C) == 2
print("rank of dependent 3x3:", np.linalg.matrix_rank(C))
# --- 5. vocabulary size is not dimension -------------------------
n_vocab, d_model = 50257, 768
E = np.zeros((n_vocab, d_model))
assert E.shape == (50257, 768)
print(f"{n_vocab} tokens, each a point in R^{d_model}")
# --- 6. low-rank update lives in a subspace ----------------------
d, r = 64, 4
Bm = np.random.randn(d, r); Am = np.random.randn(r, d)
dW = Bm @ Am
assert np.linalg.matrix_rank(dW) <= r
print("rank of BA:", np.linalg.matrix_rank(dW), "<= r =", r)
print("all assertions passed")
9. Practice Questions
- Compute the magnitude and unit vector of
(−5, 12). - State which two axioms are closure, and why the other eight are automatic for real tuples.
- Show that
{(1,1), (1,−1)}is a basis ofℝ². - Is
{(1,2), (2,4), (3,6)}independent? Give the rank and the dimension of its span. - Why is the set of all 3×3 symmetric matrices a vector space, while the set of all invertible 3×3 matrices is not?
- Explain why a 50,257-token embedding table is not "50,257-dimensional".
- Let
Vbe the span of(1,0,1)and(0,1,1). Give a vector inℝ³that is not inV. - What is the dimension of the space spanned by one-hot vectors of length 10?
- Explain LoRA as a basis statement.
- Why does flattening a CHW tensor not change its information content?
10. Official References & Further Reading
- Goodfellow et al. — Linear Algebra (span, basis, norms)
- Deisenroth et al. — Mathematics for Machine Learning, Chapter 2
- Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models
- Mikolov et al. — Efficient Estimation of Word Representations (word2vec)
- 3Blue1Brown — Span, Basis and Linear Independence
Continue the Course
- Course Index: AI Mathematics Masterclass — All 48 Chapters
- Previous Chapter: Chapter 6: Calculus Intuition — Limits, Continuity & Rates of Change
- Next Chapter: Chapter 8: Vector Operations — Addition, Scaling, Norms & the Dot Product
- Related Series: PyTorch Engineering Masterclass — All 16 Modules
