Vector Embeddings & Mathematical Distance Metrics
Understand how discrete human text is projected into continuous high-dimensional vector spaces, and how distance functions (Cosine, Dot Product, Euclidean L2) quantify semantic meaning.
1. Anatomy of a High-Dimensional Vector Embedding
An embedding is a translation of discrete, categorical tokens (words, sentences, images, audio) into a dense vector of continuous floating-point numbers in a high-dimensional vector space:
f(text) → v ∈ Rd where d ∈ {384, 768, 1024, 1536, 3072}Inside this d-dimensional coordinate space, geometric proximity corresponds directly to semantic conceptual similarity. Concepts that share similar contextual usage—such as "king" and "queen", or "Kubernetes" and "container orchestration"—are mapped closely together, while semantically disjoint concepts are placed far apart.
The Linear Subspace Property
High-dimensional transformer models preserve relational geometry:vec("King") - vec("Man") + vec("Woman") ≈ vec("Queen"). Vector databases exploit this topological geometry to execute semantic fuzzy searches across millions of documents.
2. Core Distance Metrics: Cosine vs. Dot Product vs. Euclidean
Every vector database query computes distance between a query vector q and candidate document vectors x. The choice of distance metric directly impacts search latency, memory access patterns, and retrieval accuracy:
| Metric | Formula | Range | Magnitude Sensitivity | Primary Use Case |
|---|---|---|---|---|
| Cosine Similarity | cos(θ) = (A · B) / (||A|| × ||B||) | [-1.0, +1.0] | Magnitude Invariant: Only measures angle between vectors. | Natural language text where chunk length varies (short sentences vs. long paragraphs). |
| Dot Product (Inner Product) | A · B = ∑ (Ai × Bi) | (-∞, +∞) | Magnitude Dependent: Longer vectors yield larger values. | Unit-normalized vectors, recommendation ranking, neural search with fixed-length inputs. |
| Euclidean Distance (L2) | d(A, B) = √(∑ (Ai - Bi)2) | [0.0, +∞) | Strongly Dependent: Measures physical straight-line distance. | Computer vision embeddings, geospatial coordinates, image feature vectors. |
| Manhattan Distance (L1) | d(A, B) = ∑ |Ai - Bi| | [0.0, +∞) | Strongly Dependent: Grid-based taxi-cab distance. | High-dimensional sparse data, outlier-resilient clustering, product attribute search. |
3. The Normalization Trick: Unifying Cosine & Dot Product
Computing raw Cosine Similarity requires calculating two L2 norms (||A|| = √∑ Ai2) plus a division on every single vector comparison:
# Unnormalized Cosine Distance: Requires 2 square roots and 1 division per pair
cosine_sim = np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))In production vector databases, running millions of square roots and division operations destroys CPU throughput. Instead, vector databases require or automatically perform L2 Unit Normalization during data ingestion:
u_normalized = u / ||u||2 ⇒ ||u_normalized|| = 1.0When vectors u and v are normalized to unit length (||u|| = 1 and ||v|| = 1):
- Cosine Similarity equals Dot Product:
cos(θ) = (u · v) / (1 × 1) = u · v. The database executes a single SIMD-accelerated BLAS dot product instruction! - Squared Euclidean Distance is directly proportional to Cosine Distance:Minimizing Euclidean distance is mathematically identical to maximizing Cosine similarity for unit vectors.
||u - v||2 = ||u||2 + ||v||2 - 2(u · v) = 1 + 1 - 2 × cos(θ) = 2(1 - cos(θ))
4. Industry Embedding Models: Dimensions & Trade-offs
Selecting the right embedding model sets the foundation of your vector index. Increasing dimensions improves nuance at the cost of RAM and search latency:
| Model | Provider | Dimensions | Context Window | Key Strengths |
|---|---|---|---|---|
| text-embedding-3-small | OpenAI | 1,536 (truncatable to 512) | 8,191 tokens | Matryoshka representation, ultra-cheap API, strong baseline |
| text-embedding-3-large | OpenAI | 3,072 (truncatable to 1024) | 8,191 tokens | Top MTEB multilingual retrieval score, nuanced enterprise search |
| embed-english-v3.0 | Cohere | 1,024 | 512 tokens | Asymmetric search support (search_document vs search_query) |
| bge-m3 | BAAI (Open Source) | 1,024 | 8,192 tokens | Multi-functionality: outputs Dense, Sparse (Lexical), and Multi-Vector ColBERT |
| all-MiniLM-L6-v2 | HuggingFace / SBERT | 384 | 256 tokens | Ultra-lightweight (80MB model), sub-millisecond local CPU inference |
vector[:512]) and re-normalize it to reduce vector storage costs by 66%with less than a 2% degradation in retrieval accuracy.5. Production Benchmark: Vectorized Similarity Matrix in Python
Here is a complete Python script using NumPy to compute batch vector distances across thousands of documents in milliseconds using matrix multiplication:
import numpy as np
# Simulate a database of 10,000 document vectors (1536 dimensions)
NUM_DOCS = 10000
DIMENSIONS = 1536
np.random.seed(42)
docs_matrix = np.random.randn(NUM_DOCS, DIMENSIONS).astype(np.float32)
# Step 1: Normalize all document vectors to unit length in-place
norms = np.linalg.norm(docs_matrix, axis=1, keepdims=True)
docs_normalized = docs_matrix / norms
# Step 2: Create a query vector and normalize it
query_vector = np.random.randn(DIMENSIONS).astype(np.float32)
query_normalized = query_vector / np.linalg.norm(query_vector)
# Step 3: Compute batch Cosine Similarity via Matrix-Vector Dot Product
# Shape: [10000, 1536] x [1536] -> [10000]
cosine_scores = np.dot(docs_normalized, query_normalized)
# Step 4: Extract Top-5 Most Similar Documents
top_k_indices = np.argsort(cosine_scores)[::-1][:5]
print("=== Top 5 Retrieved Document IDs & Cosine Scores ===")
for rank, idx in enumerate(top_k_indices, 1):
print(f"Rank {rank}: Doc ID #{idx} | Similarity Score: {cosine_scores[idx]:.5f}")ANN Algorithms: HNSW & IVF
Continue to the next advanced retrieval and vector search engineering module.
Explore Related Tools
Hand-picked utilities and calculators related to this tool.
