Natural Language Processing (NLP): Tokenization, Semantic Embeddings & Transformers
Natural Language Processing bridges the semantic divide between human symbolic language and machine tensor calculations. Explore how raw unstructured strings transform into high-dimensional vector spaces and self-attentive generative models.
1. The Text-to-Vector Pipeline: Tokenization & Dense Embeddings
Computers cannot process raw characters directly. NLP relies on a multi-stage numerical pipeline:
Subword Tokenization (BPE)
Algorithms like Byte-Pair Encoding (BPE) and WordPiece iteratively merge frequent character pairs, building compact vocabularies (32k–128k tokens) that balance vocabulary size with out-of-vocabulary flexibility.
Dense Vector Embeddings
Each token is mapped to a continuous dense vector e ∈ ℝ^d (e.g. d = 768 or 4096). Semantic relationships correspond to geometric angles measured by Cosine Similarity.
Positional Encodings
Because Transformers process all tokens simultaneously without inherent sequence awareness, sinusoidal or rotary position embeddings (RoPE) are added to inject word order information.
2. The Transformer Architecture & Multi-Head Self-Attention
Introduced in 2017 by Vaswani et al., the Scaled Dot-Product Self-Attention mechanism enables every token to dynamically attend to every other token in the sequence:
| Architecture Paradigm | Attention Masking | Primary Use Case | Exemplars |
|---|---|---|---|
| Encoder-Only (Bidirectional) | Bidirectional (attends to past and future tokens) | Classification, NER, semantic search embeddings | BERT, RoBERTa, DeBERTa |
| Decoder-Only (Autoregressive) | Causal lower-triangular mask (attends to past tokens only) | Open-ended text generation, coding, conversational agents | GPT-4o, LLaMA-3, Claude, Gemini |
| Encoder-Decoder (Seq2Seq) | Bidirectional encoder + Causal autoregressive decoder | Language translation, abstractive summarization | T5, BART, Whisper |
