Vector Operations
Adding, scaling, and comparing vectors.
On this page
Beginner: there are three operations you'll see constantly. Addition combines two vectors by adding matching positions (like merging two shopping lists item-by-item). Scalar multiplication stretches or shrinks a vector without changing its direction (multiply by a negative number and it flips to point the opposite way). The dot product is the most important for ML — multiply matching positions and add the results, giving one number that tells you how "aligned" two vectors are.
Intermediate: it helps to have a physical picture for each: addition is "walk along a, then keep walking along b" — where you end up is a+b. Scalar multiplication is "take the same walk, just longer or shorter, or backwards." The dot product is different in kind — it doesn't produce a new vector at all, it collapses two vectors down into a single number that measures similarity, which is exactly why it's the workhorse of machine learning: models are, at their core, endless dot products.
Advanced: the dot product secretly encodes both magnitude and angle at once — a·b = |a||b|cos(θ) — which means you can recover the angle between any two vectors purely from their coordinates, with no trigonometry drawn on paper required. This is the mathematical foundation of every "similarity score" in machine learning, from search engines to face recognition.
Largest when vectors point the same way (θ=0°), zero when perpendicular (θ=90°), negative when opposite. Dividing the dot product by both lengths gives cosine similarity, the single most common similarity metric in ML — it ranges from −1 to 1 regardless of how long the vectors are, which is why it's preferred over the raw dot product for comparing embeddings of very different magnitudes.
Start from the Law of Cosines applied to the triangle formed by a, b, and the vector a−b, with θ the angle between a and b:
Now expand the same left-hand side purely algebraically, using the dot product's own definition:
Both expressions equal the same quantity, so their right-hand sides are equal:
The ‖a‖² and ‖b‖² terms cancel from both sides, leaving −2‖a‖‖b‖cosθ = −2 a·b, and dividing by −2 gives exactly a·b = ‖a‖‖b‖cosθ.
Where this is used: this identity is the entire justification for cosine similarity (used above) and for every "attention score" computation in Transformers (section 1.3's expert note) — it's what lets a pure dot product stand in for "how aligned are these two directions" without ever computing an angle directly.
a (blue) is drawn first, then b (orange) starts where a ends. The result a+b (green) is the straight line back to the new tip. Drag either dot afterward.
Let a = [2, 3] and b = [4, 1].
| Operation | Calculation | Result |
|---|---|---|
| a + b | [2+4, 3+1] | [6, 4] |
| a − b | [2−4, 3−1] | [−2, 2] |
| 3a | [3×2, 3×3] | [6, 9] |
| a · b | (2×4)+(3×1) | 8+3 = 11 |
Now try c = [1, -2]: a · c = (2×1)+(3×-2) = 2−6 = −4. A negative dot product tells you immediately, without drawing anything, that a and c point in broadly opposite general directions (the angle between them is greater than 90°).
Cosine similarity is one line of NumPy, and it's what powers semantic search, recommendation engines, and duplicate-detection systems in production.
import numpy as np
a = np.array([2, 3])
b = np.array([4, 1])
dot = np.dot(a, b) # 11
cosine_sim = dot / (np.linalg.norm(a) * np.linalg.norm(b))
print(round(cosine_sim, 3)) # 0.789 -> fairly similar direction
# This exact pattern is how search engines rank documents
# against a query embedding, and how recommenders compare users.- Recommendation systems use the dot product (as cosine similarity) to measure how similar two users' taste vectors are, or how well a user vector matches a product vector.
- Neural networks — every neuron computes a dot product between its input and weight vector before applying an activation function; this single operation, repeated billions of times, is most of what a forward pass "is."
- Search engines — semantic search embeds your query and every document into the same vector space, then ranks documents by dot product / cosine similarity to the query.
- Physics — work done by a force is the dot product of the force vector and the displacement vector:
W = F · d, which is exactly why pushing sideways on a wall (perpendicular to any real displacement) does zero work.
- Using raw dot product as "similarity" without normalizing — a long vector pointing in a slightly wrong direction can out-score a short vector pointing exactly right. Cosine similarity fixes this by dividing out the magnitudes.
- Forgetting that
a·b = 0means perpendicular, not "unrelated" or "small" — orthogonality is an exact geometric statement, not a vague one.
Going deeper
Two vectors are orthogonal when their dot product is exactly 0. This underlies the attention mechanism in Transformers: attention scores are the dot product between "query" and "key" vectors — a high score means "pay attention to this token."
In practice, attention scores are usually scaled by 1/√d (where d is the dimensionality) before the softmax — this is "scaled dot-product attention," and the scaling exists purely to stop dot products from exploding in high dimensions, which would otherwise push the softmax into regions with vanishing gradients.
At the master level: a set of mutually orthogonal unit vectors is called orthonormal — the columns of every rotation matrix and every U/V matrix in an SVD (section 1.10) are orthonormal by construction, which is exactly what guarantees those matrices preserve lengths and angles (they never distort space, only reorient it).
If a · b = 0, what does that tell you about the two vectors?
They're orthogonal (perpendicular) — no directional overlap.