AI Explainedby Milad Fahmy

AI Explained

Top-p vs top-k sampling: what the knobs actually cut

Top-k, top-p (nucleus) and min-p sampling all cut the long tail of unlikely tokens. How each one decides where to cut, and why a fixed k fits badly.

Milad FahmyNov 12, 20262 min read

On this page
  1. Quick recap: scores, then a roll
  2. The problem is the tail
  3. Top-k: a fixed number
  4. Top-p: the nucleus
  5. Min-p: a cutoff that moves
  6. Defaults, stacking, and the future

You've seen settings like top_k = 50 and top_p = 0.9. They're on almost every model playground. But what do they actually cut?

Quick recap: scores, then a roll

At each step a language model scores every possible next token, turns the scores into probabilities, and samples one. Temperature reshapes those odds; top-k, top-p and min-p decide which tokens are allowed in the roll at all.

The problem is the tail

After "The sky is", a few tokens are likely: blue, clear, gray. Then come thousands of unlikely ones. Each is tiny, but together they are not. Sample from the full list long enough and you will eventually pick nonsense. The nucleus sampling paper describes the fix as "truncating the less reliable tail of the distribution".

Top-k: a fixed number

Top-k keeps only the k likeliest tokens and throws away the rest. A 2018 story-generation paper by Fan et al. sampled "from the k = 10 most likely candidates". Set k to one and you get greedy decoding: always the single likeliest token.

The catch is that one fixed k fits badly. Sometimes only two words make sense next; sometimes hundreds do.

  • On a flat distribution, a small k cuts good options.
  • On a peaked one, a big k lets junk in.

Top-p: the nucleus

Top-p, or nucleus sampling, comes from Holtzman et al.'s 2019 paper The Curious Case of Neural Text Degeneration, published at ICLR 2020. Sort the tokens by probability and keep the smallest set whose probabilities add up to at least p.

Because it's based on probability mass, not a count, the set adapts. With p = 0.9, a confident step might keep two tokens, and an open-ended one two hundred (illustrative numbers). In the paper's words, the size "will adjust dynamically based on the shape of the probability distribution".

With either method, the surviving tokens' odds are rescaled to add up to one, and then the model samples.

Min-p: a cutoff that moves

Min-p is a newer idea. It drops any token far less likely than the top one: the threshold uses "the top token's probability as a scaling factor". With min_p = 0.1 and a top token at 0.60, anything under 0.06 goes. When the model is sure, the cutoff is strict; when it isn't, it's looser.

Defaults, stacking, and the future

Setting Hugging Face Transformers default
top_k 50
top_p 1.0 (keeps everything)
min_p not set

You can stack these, but each one shrinks the pool further. Change one at a time and test on your real prompts.

And on some new models, the knobs are going away: Google's docs say "Gemini 4 and Gemini 3.6+ models ignore custom values for temperature, top_p, and top_k."

Cut the tail, keep the sense.

Sources

#top-p#top-k#nucleus sampling#min-p#llm sampling

Watch