Chapter 8: Vector Operations: Addition, Scaling, Norms & the Dot Product

Chapter 8: Vector Operations: Addition, Scaling, Norms & the Dot Product

AI Reading

Quick summary of this article

Preparing the AI summary…

AI Mathematics Masterclass — Chapter 8 of 48

Five operations cover almost every vector computation in a neural network: addition, scalar multiplication, the three norms, and the dot product. The dot product is the important one — it is simultaneously a classifier logit, a retrieval similarity, and a neuron’s pre-activation. This chapter derives it geometrically, shows why cosine similarity is the dot product with magnitude divided out, and demonstrates the one numerical property that matters: the dot product amplifies its inputs, so its scale grows with dimension.

1. Technical Audit & Core Learning Objectives

Vector operations are usually demonstrated with arrows and then left behind. What is missing is the connection between the algebra and the failures practitioners see: unbounded logits, similarity scores that depend on text length, and gradient norms that explode with depth.

Component Common Tutorial Deficiency Engineering Standard Applied Here
Addition / scaling Drawn as arrows, never linked to residuals or attention blending. Geometric rules connected to actual architectural operations.
Norms Only the L2 norm is taught; L1 and L∞ treated as trivia. All three with the regularisation each one induces.
Dot product Given as a formula with no geometric content. Derived as a projection, then shown as the shape every logit takes.
Scale Dot-product magnitude growth with dimension never mentioned. Explained as the reason attention divides by √d.

Core learning objectives. After this chapter you should be able to:

  1. Add and scale vectors geometrically and componentwise.
  2. Compute L1, L2, and L∞ norms and state which regulariser each corresponds to.
  3. Compute a dot product, a projection, and an angle between two vectors.
  4. Explain the sign of the dot product in terms of the angle.
  5. Derive the attention scaling factor √d from variance growth.
  6. Recognise the dot product inside a logit, a similarity score, and a weighted sum.

2. Addition and Scaling

Parallelogram construction for vector addition and scaling including a negative scale reversing direction
Figure 8.1: Addition closes the triangle; scaling changes magnitude by |c| and reverses direction when c is negative.
u + v = (u₁+v₁, u₂+v₂, …, uₙ+vₙ)     componentwise
c·u   = (c·u₁,  c·u₂,  …, c·uₙ)      componentwise

Commutative:  u + v = v + u
Associative:  (u+v)+w = u+(v+w)
Distributive: c(u+v) = cu + cv

Addition is the operation behind residual connections — x + f(x) adds two vectors of identical shape — and behind attention output, which is a weighted sum of value vectors. Scaling appears in learning-rate application, in gradient clipping, and in the temperature parameter of sampling.

3. The Three Norms

Unit balls for the L1 diamond, L2 circle and L-infinity square norms
Figure 8.2: Each norm draws a different unit ball. The shape of that ball is what each regulariser prefers.
Norm Formula Induced Regulariser Effect
L1 Σ|xᵢ| Lasso Drives weights exactly to zero — sparsity
L2 √Σxᵢ² Ridge / weight decay Shrinks all weights smoothly
L∞ max|xᵢ| Adversarial robustness (ε-ball) Bounds the single worst coordinate
import numpy as np

x = np.array([3.0, -4.0, 0.0])
print("L1 :", np.linalg.norm(x, 1))        # 7.0
print("L2 :", np.linalg.norm(x, 2))        # 5.0
print("Linf:", np.linalg.norm(x, np.inf))  # 4.0

# gradient clipping uses the L2 norm
g = np.array([10.0, 0.0, 0.0])
clip = 1.0
scaled = g * min(1.0, clip / np.linalg.norm(g))
print("clipped norm:", np.linalg.norm(scaled))   # 1.0

The shapes in Figure 8.2 explain the regularisation behaviour directly. The L1 ball has corners on the axes, so the optimum of a constrained problem tends to land exactly on a corner — where one coordinate is zero. The L2 ball is smooth, so the optimum lands somewhere on the curve with all coordinates non-zero. That single geometric fact is the whole difference between Lasso sparsity and Ridge shrinkage.

4. The Dot Product

Vector b projected onto vector a with the perpendicular dashed residual shown
Figure 8.3: The dot product is the length of the projection of one vector onto the other, scaled by that other’s length.
a · b = Σᵢ aᵢbᵢ = ‖a‖ ‖b‖ cos θ

Equivalent readings:
  componentwise   multiply matching entries, sum
  geometric       project one onto the other, multiply lengths
  angular         a scalar summary of alignment
Dot Product Angle Meaning
positive θ < 90° Pointing the same general way
zero θ = 90° Orthogonal — no shared component
negative θ > 90° Pointing away from each other
import numpy as np

a = np.array([4.0, 0.0])
b = np.array([2.0, 3.0])

dot = a @ b
print("dot:", dot)                                    # 8.0
cos = dot / (np.linalg.norm(a) * np.linalg.norm(b))
print("cos:", cos, " angle:", np.degrees(np.arccos(cos)))

proj = (dot / (a @ a)) * a                            # projection of b onto a
print("projection of b onto a:", proj)                # [2. 0.]
print("residual orthogonal:", np.isclose((b - proj) @ a, 0.0))

5. Cosine Similarity

Table of cosine values at 0, 45, 90, 135 and 180 degrees with interpretations
Figure 8.4: Cosine similarity is the dot product with both magnitudes divided out, so it lives in [−1, 1].
cos(u, v) = (u · v) / (‖u‖ ‖v‖)

Invariant to magnitude:  cos(2u, v) = cos(u, v)
This is why text retrieval normalises before comparing:
a long document and a short one about the same topic
should score 1.0, not "long beats short".

For unit vectors the cosine similarity and the dot product are identical, which is why retrieval pipelines normalise embeddings once at index time rather than dividing at query time.

6. One Operation, Three Disguises

Diagram showing the same dot product appearing as a classifier logit, a retrieval similarity and a neuron pre-activation
Figure 8.5: The dot product appears under three different names. Only the interpretation of the two vectors changes.
Context Expression What the Vectors Are
Classifier w·x + b weight vs input
Retrieval cos(q, k) query vs document key
Attention q·k / √d query vs key projections
Neuron Σ wᵢxᵢ incoming weights vs activations

6.1 Why attention divides by √d

If the components of q and k are independent with mean 0 and variance 1, their dot product over d dimensions has variance d and standard deviation √d. Changing d from 64 to 768 would make attention logits roughly 3.5× larger in magnitude, pushing softmax into saturation where gradients vanish. Dividing by √d restores unit variance, making the same temperature work at every width.

import numpy as np

rng = np.random.default_rng(0)
for d in (16, 64, 256, 1024):
    q = rng.normal(size=(2000, d))
    k = rng.normal(size=(2000, d))
    dots = np.sum(q * k, axis=1)
    print(f"d={d:5d}  std(dot)={dots.std():7.3f}  std(dot/√d)={ (dots/np.sqrt(d)).std():.3f}")

The second column grows like √d; the third is stable at 1.0. That stability is the entire justification for the scaling factor.

7. Worked Examples

7.1 Add, scale, and normalise

u = (1, 2),  v = (3, -1)
u + v      = (4, 1)
2u − 3v    = (2−9, 4+3) = (−7, 7)
‖2u − 3v‖  = √(49+49) = 9.899

7.2 Angle between two vectors

a = (1, 0),  b = (1, 1)
a·b = 1       ‖a‖ = 1, ‖b‖ = √2
cos θ = 1/√2 = 0.7071   →   θ = 45°

7.3 Norm comparison on the same vector

x = (3, -4)
L1 = 7, L2 = 5, L∞ = 4
Always:  L∞ ≤ L2 ≤ L1   for the same vector

7.4 Projection with a residual check

project b = (2,3) onto a = (4,0)
a·b = 8,  a·a = 16,  t = 0.5
proj = (2, 0)
residual = (0, 3),  and (0,3)·(4,0) = 0   ✓ orthogonal

7.5 Gradient clipping by norm

g = (30, 40),  ‖g‖ = 50,  clip = 1.0
g_clipped = g · (1/50) = (0.6, 0.8)     ‖g_clipped‖ = 1.0
Direction preserved, magnitude bounded.

8. Verify It in Python

import numpy as np

# --- 1. arithmetic identities --------------------------------------
u, v = np.array([1., 2.]), np.array([3., -1.])
assert np.allclose(u + v, [4., 1.])
assert np.allclose(2 * u - 3 * v, [-7., 7.])
print("linear combos ok")

# --- 2. norm ordering ----------------------------------------------
x = np.array([3., -4.])
n1, n2, ni = np.linalg.norm(x, 1), np.linalg.norm(x, 2), np.linalg.norm(x, np.inf)
assert ni <= n2 <= n1
print("Linf <= L2 <= L1:", ni, n2, n1)

# --- 3. dot product as projection ----------------------------------
a, b = np.array([4., 0.]), np.array([2., 3.])
assert np.isclose(a @ b, 8.0)
proj = (a @ b) / (a @ a) * a
assert np.allclose(proj, [2., 0.])
assert np.isclose((b - proj) @ a, 0.0)          # residual orthogonal
print("projection:", proj)

# --- 4. cosine is scale invariant ----------------------------------
def cos(u, v):
    return (u @ v) / (np.linalg.norm(u) * np.linalg.norm(v))

assert np.isclose(cos(a, b), cos(a * 7.5, b * 0.3))
assert np.isclose(cos(np.array([1., 0.]), np.array([1., 1.])), 1 / np.sqrt(2))
print("cosine invariant to magnitude")

# --- 5. orthogonal vectors have zero dot ---------------------------
assert np.isclose(np.array([1., 0.]) @ np.array([0., 1.]), 0.0)
print("orthogonal dot is zero")

# --- 6. dot product std grows as sqrt(d) ---------------------------
rng = np.random.default_rng(0)
stds = {}
for d in (16, 256, 1024):
    q = rng.normal(size=(4000, d)); k = rng.normal(size=(4000, d))
    stds[d] = np.sum(q * k, axis=1).std()
assert stds[1024] > 3 * stds[16]                 # grows with dimension
assert np.isclose(stds[1024] / np.sqrt(1024), 1.0, atol=0.1)
print("std(dot) by d:", {k: round(v, 2) for k, v in stds.items()})

# --- 7. clipping preserves direction -------------------------------
g = np.array([30., 40.])
gc = g * (1.0 / np.linalg.norm(g))
assert np.isclose(np.linalg.norm(gc), 1.0)
assert np.isclose(cos(g, gc), 1.0)
print("clipped:", gc)
print("all assertions passed")

9. Practice Questions

  1. Compute (2,3,−1) + (−1,0,4) and its L2 norm.
  2. For x = (1,2,2), compute L1, L2, and L∞.
  3. Find the angle between (1,0) and (1,√3).
  4. Show that u·v = 0 implies ‖u+v‖² = ‖u‖² + ‖v‖².
  5. Compute the projection of (3,4) onto (1,0), and the residual.
  6. Why is cosine similarity preferred over dot product for text retrieval?
  7. Derive why Var(q·k) = d when components are i.i.d. unit variance.
  8. What does dividing attention logits by √d protect against?
  9. Clip a gradient (6,8) to norm 2.0 and verify the direction is unchanged.
  10. Explain geometrically why the L1 ball’s corners produce sparse solutions.

Leave a Reply

Your email address will not be published. Required fields are marked *