How Generative AI Works: Under the Hood

1. The Autoregressive Text Generation Pipeline

Autoregressive Large Language Models (LLMs) operate in a loop: predicting one token at a time, appending it to the context, and repeating until an End-of-Sequence (<EOS>) token is generated.

2. Mathematical Sampling & Decoding Strategies

At the final layer of an LLM, the model outputs an unnormalized vector of scores called logits z ∈ ℝ^V, where V is the vocabulary size. How we convert these logits into a chosen token dictates the model's creativity and coherence.

3. How Diffusion Models Generate Images

Unlike autoregressive text generation which works sequentially, visual generative models (e.g., Stable Diffusion, Flux, Midjourney) rely on stochastic differential equations and denoising diffusion:

4. Custom Implementation: Temperature & Nucleus Sampler

Here is how sampling algorithms are mathematically coded in PyTorch inside modern inference engines (vLLM/Hugging Face):

import torch
import torch.nn.functional as F

def sample_next_token(logits: torch.Tensor, temperature: float = 0.7, top_p: float = 0.9) -> int:
    """
    Applies temperature scaling and Nucleus (Top-p) filtering to raw model logits.
    """
    # 1. Apply temperature scaling
    if temperature > 0:
        logits = logits / temperature
    else:
        # Greedy decoding if temperature is 0
        return torch.argmax(logits, dim=-1).item()

    # 2. Convert logits to probabilities
    probs = F.softmax(logits, dim=-1)

    # 3. Sort probabilities in descending order
    sorted_probs, sorted_indices = torch.sort(probs, descending=True)
    cumulative_probs = torch.cumsum(sorted_probs, dim=-1)

    # 4. Remove tokens with cumulative probability above the threshold (Nucleus)
    sorted_indices_to_remove = cumulative_probs > top_p
    # Shift indices to the right to keep the first token above the threshold
    sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
    sorted_indices_to_remove[..., 0] = 0

    # 5. Zero out removed tokens and renormalize
    sorted_probs[sorted_indices_to_remove] = 0.0
    sorted_probs = sorted_probs / torch.sum(sorted_probs, dim=-1, keepdim=True)

    # 6. Sample from filtered multinomial distribution
    next_sorted_token_idx = torch.multinomial(sorted_probs, num_samples=1)
    next_token_id = sorted_indices.gather(-1, next_sorted_token_idx).item()

    return next_token_id