Large Language Models: Architectures, Scaling & MoE
Large Language Models (LLMs) represent the apex of causal sequence modeling. By training deep autoregressive architectures on internet-scale text corpora, these networks develop in-context few-shot learning capabilities, world modeling, and complex algorithmic reasoning.
1. Architectural Blueprint of Modern Decoder-Only LLMs
While original Transformers used an Encoder-Decoder structure (e.g., T5), modern frontier LLMs (Llama 3, Mistral, Gemma 2, Claude, DeepSeek) are universally causal decoder-only. Over iterations, modern models standardized on three major architectural improvements:
Replaces standard LayerNorm with Root Mean Square Normalization (RMSNorm). It omits mean-centering and scales strictly by root mean square, reducing compute overhead by ~7% while stabilizing gradients in 100+ layer deep networks.
Replaces simple ReLU or GELU activations with Swish Gated Linear Units (SwiGLU):SwiGLU(x) = (x · W_1) ⊗ Swish(x · W_2) · W_3
Consistently yields lower validation perplexity at identical parameter counts.
Injected into Query and Key states at every single layer, ensuring relative token position decay across massive 128k+ context windows without needing absolute position tables.
2. Scaling Laws: Kaplan vs. Chinchilla
Training multi-billion parameter networks requires millions of GPU-hours. Determining how to divide compute budgetC ≈ 6ND between parameter count N and token countD is governed by empirical scaling laws:
L(N, D) = E + A / N^α + B / D^β
Compute-optimal ratio: Tokens (D) should be ~20× the Parameter count (N).
| Model Era | Parameters (N) | Training Tokens (D) | Tokens / Param Ratio | Status |
|---|---|---|---|---|
| GPT-3 (2020) | 175 Billion | 300 Billion | 1.7 : 1 | Severely undertrained (Kaplan law paradigm) |
| Chinchilla (2022) | 70 Billion | 1.4 Trillion | 20 : 1 | Compute-optimal for training FLOPs |
| Llama 3 (2024) | 8 Billion | 15.0 Trillion | 1,875 : 1 | Inference-optimal overtraining (maximizes edge performance) |
Why Overtrain? While training an 8B model on 15T tokens is suboptimal in terms of pure training FLOPs, it yields an ultra-compact model that is radically cheaper to serve at inference time across billions of user requests.
3. Mixture-of-Experts (MoE) Architecture
Dense models activate all weights for every single token. As models scale past 100B parameters, inference latency becomes intolerable.Sparse Mixture-of-Experts (MoE) replaces dense FFN layers with multiple expert blocks controlled by a routing router:
Softmax Router Mechanics
Given hidden state x, a router computes gating logits H(x) = x · W_g. The Top-k softmax values select which experts evaluate the token:
Real-World Case: Mixtral 8x7B & DeepSeek-V3
- Mixtral 8x7B: 46.7 Billion total parameters, but only 12.9B parameters active per token (Top-2 routing out of 8 experts).
- DeepSeek-V3: Multi-Head Latent Attention (MLA) combined with 256 fine-grained routed experts + 1 shared expert, achieving frontier performance at 1/10th the inference compute cost.
4. Serving & Inference Optimization: The KV-Cache Bottleneck
During autoregressive generation, computing Key and Value tensors for previously generated tokens over and over is redundant. Inference engines cache them in GPU memory as the KV-Cache.
Memory (Bytes) = 2 × Layers × Heads_KV × Head_Dim × Seq_Len × Precision_Bytes
For an 8B model with 32 layers, sequence length 8192, and FP16 precision, a single user session consumes ~1GB of GPU RAM just for the KV-cache!
PagedAttention (vLLM)
Inspired by OS virtual memory paging, PagedAttention breaks KV-caches into non-contiguous blocks of memory. This eliminates memory fragmentation and enables dynamic batching, increasing serving throughput by 2x–4x.
Speculative Decoding
A small, fast "draft model" (e.g. 1B params) generates candidate tokens rapidly. The large target model (70B) verifies all tokens in a single parallel forward pass, boosting decoding speeds by 2x–3x without altering output probabilities.
5. Production Code: Low-Latency Streaming with vLLM & Ollama
The following Python snippet connects to an OpenAI-compatible high-performance local serving engine (vLLM / Ollama) with token streaming:
import os
from openai import OpenAI
# Connect to high-throughput local engine (vLLM or Ollama serving Llama 3)
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY" # vLLM local instance
)
stream = client.chat.completions.create(
model="meta-llama/Meta-Llama-3-8B-Instruct",
messages=[
{"role": "system", "content": "You are a distributed systems architect."},
{"role": "user", "content": "Explain how PagedAttention solves KV-cache memory fragmentation."}
],
temperature=0.3,
max_tokens=600,
stream=True
)
print("Streaming Response:
")
for chunk in stream:
content = chunk.choices[0].delta.content or ""
print(content, end="", flush=True)Module Key Takeaways
- Modern LLMs universally employ decoder-only architectures with RMSNorm, SwiGLU, and RoPE.
- Chinchilla scaling laws established that training data and model parameters should scale symmetrically (~20 tokens/param).
- Sparse MoE models decouple model parameter capacity from per-token inference FLOPs using learned routing.
- Inference serving throughput is memory-bandwidth bounded; PagedAttention and GQA mitigate KV-cache bloat.
Explore Related Tools
Hand-picked utilities and calculators related to this tool.
