Decoding Hyperparameters: Temperature, Top-P & Penalties
A prompt is only half of the generation equation. How the model translates raw logits into actual output tokens is dictated by sampling hyperparameters. In this chapter, understand the probability mathematics behind Temperature,Top-P (Nucleus Sampling), and repetition penalties to achieve rock-solid reproducibility in production.
1. Temperature Scaling: The Softmax Modifier
Before generating a token, the neural network computes unbounded real scores (logits $z_i$) for every word in its vocabulary. To convert logits into a probability distribution, PyTorch and inference engines apply Softmax scaled by Temperature (T):
Collapses the distribution so the single token with the highest logit receives probability 1.0. Completely deterministic. Mandatory for SQL generation, code, math, and JSON extraction.
Preserves the natural shape of the learned distribution while allowing slight variation between runs. Ideal for general conversational agents, copywriting, and customer support.
Flattens the distribution toward a uniform random pick. Low-probability tokens become frequent, resulting in hallucinations, syntactical breakdown, and incoherent output.
2. Top-P (Nucleus Sampling) vs. Top-K Truncation
Even at low temperatures, a long tail of thousands of rare tokens can cause erratic slips. Sampling cutoff algorithms dynamically prune this tail:
Top-P (Nucleus Sampling)
Sorts tokens by probability and keeps only the top candidates whose cumulative sum reaches threshold p (e.g. 0.90).
Top-K Truncation
Strictly limits sampling to the top k highest-probability tokens regardless of cumulative mass (e.g. k = 40).
OpenAI Golden Rule
OpenAI recommends altering either temperature OR top_p, but never tweaking both simultaneously. For deterministic business workflows, keep top_p = 1.0 and set temperature = 0.0.
3. Frequency Penalty vs. Presence Penalty
When generating long documents, models can fall into repetitive feedback loops where reciting a phrase increases the probability of reciting it again:
| Parameter | Formula Modification | Primary Effect | Recommended Setting |
|---|---|---|---|
frequency_penalty | logit -= count * penalty | Penalizes words proportionally to how many times they already appeared in the text. Prevents repetitive lists. | 0.2 - 0.5 (Max 2.0) |
presence_penalty | logit -= (1 if count > 0 else 0) * penalty | One-time flat penalty if the word has appeared at least once. Encourages introducing fresh topics and diverse vocabulary. | 0.1 - 0.4 (Max 2.0) |
4. Production Hyperparameter Recipes
| Application Domain | Temperature | Top-P | Frequency Penalty | Seed |
|---|---|---|---|---|
| JSON Extraction / Pydantic | 0.0 | 1.0 | 0.0 | 42 (Strictly reproducible) |
| Code Generation & Bug Fixing | 0.1 | 0.95 | 0.0 | 1337 |
| RAG Enterprise Q&A | 0.2 | 1.0 | 0.1 | None |
| Creative Storytelling / Brainstorming | 0.85 | 0.90 | 0.4 | None |
Chain-of-Thought & Tree of Thoughts
Continue to the next hands-on prompt engineering guide in this series.
Explore Related Tools
Hand-picked utilities and calculators related to this tool.
