What is top-p sampling in simple terms?

It is a rule for picking the next word. The model sorts its candidate words by probability, keeps the fewest that together reach the chosen share p, and picks one of those at random in proportion to their probabilities. Everything below the cutoff is ignored.

What is the difference between top-p and top-k?

Top-k keeps a fixed number of candidates, such as the 10 most likely. Top-p keeps however many it takes to reach a share of the probability, so it keeps few words when the model is confident and more when it is unsure. Holtzman and colleagues argue that one fixed k suits some steps and not others.

Is nucleus sampling the same as top-p?

Yes. Nucleus sampling is the name from the 2019 paper by Holtzman and colleagues. Provider documentation calls the setting top_p, and OpenAI's reference describes it as an alternative to sampling with temperature, called nucleus sampling.

What is a good top-p value?

There is no universal one. Documented defaults differ: Hugging Face's library uses 1.0 and Google's Gemini documentation lists 0.95. For some current models the setting cannot be changed at all, so check the documentation of the model you use.

Should I change temperature and top-p together?

Provider advice, as of 2026-10-11, is to change one, not both. OpenAI's reference says it generally recommends altering one or the other, and Google's documentation recommends keeping defaults for Gemini 3.x models.

LLM · SAMPLING

What Is Top-p (Nucleus) Sampling? How AI Picks Among Likely Words

Top-p sampling, also called nucleus sampling, is a rule for choosing a language model's next word. The model sorts its candidates by probability, keeps the smallest group whose probabilities add up to p, and draws one word from that group. The unlikely tail is dropped, and the group grows or shrinks with the model's confidence.

One-pager titled "What is top-p sampling?" with the subtitle "Keep the smallest set that adds up to p". Six candidate next words for one step, sorted by probability: tea 55%, coffee 20%, water 12%, milk 6%, juice 3% and others 4%, with a running total column of 55%, 75%, 87%, 93%, 96% and 100%. The first four rows sit inside an amber frame with amber bars, and the total 93% is highlighted. An amber line below them reads "p = 90%: stop at 93%", and the last two rows are dimmed but still readable. An arrow runs from the frame to a box reading "Draw one word, from the 4 kept". A REMEMBER band reads: p is a share, not a count; group grows when unsure; cuts the tail only. The numbers are illustrative. — One-pager: six candidate words sorted by probability, the running total passing p = 90%, the kept four in an amber frame and the tail dropped.

Keep the smallest group that adds up to p

At every step a language model scores each token in its vocabulary. The next-token prediction page covers where those scores come from. Top-p is a filter applied to them before one token is drawn.

Holtzman and colleagues introduced it in 2019 under the name nucleus sampling. The steps are short. Sort the candidates from most to least probable. Add the probabilities up from the top. Stop as soon as the running total reaches p. The words above the stop point are the nucleus, and the rest are discarded.

The kept probabilities no longer add up to 1, so each one is divided by the kept total. In the figure, 55% becomes 59%. The model then draws one token from the rescaled group, in proportion to those numbers.

Notice that p is a share of probability, not a number of words. The figure's running total reaches 87% after three words, which is short of 90%, so a fourth word is added and the total reaches 93%. The word that crosses the line stays in.

I’d like a cup of ___

Top-p90%

Smaller poolWhole list

WordChanceTotalDrawn

tea55%55%59%
coffee20%75%22%
water12%87%13%
milk6%93%6%
juice3%96%out
others4%100%out

p = 90%: stop at 93%. The pool is 4 of 6 words.

Press the button to draw 10 words from the pool.

Illustrative numbers. “Drawn” is each kept word’s share after the pool is rescaled to 100%.

Why the cutoff adapts, and how top-k differs

Because the cutoff is a share of probability, the size of the group depends on the shape of the distribution at that step. When one word holds most of the probability, the running total passes p almost at once and the group is tiny. When probability is spread across many words, the total climbs slowly and the group is large. Holtzman and colleagues describe the candidate pool as expanding and contracting dynamically with the model's confidence.

The older approach is top-k, which always keeps a fixed number of candidates. Fan, Lewis and Dauphin used it in 2018 for story generation, sampling from the 10 most likely words. Their reason was that completely random sampling can introduce very unlikely words that damage the text.

Holtzman and colleagues point to the weakness of one fixed number. Where the model is spread across tens or hundreds of reasonable words, a small k risks bland text. Where one or a few words hold nearly all the probability, a large k lets in inappropriate candidates. The figure shows both with k = 3: on the confident step it keeps two extra words that hold 5% between them, and on the uncertain step it throws away 35% of the probability.

The authors report that nucleus sampling gave text closer to human text in diversity without losing fluency or coherence. Their tests used GPT-2, so they show the idea works, not which setting a current product uses.

The two ends of the dial

At p = 1, the smallest group whose probabilities add up to 1 is every token the model gives any probability to. Nothing is filtered, so sampling runs over the full distribution. Hugging Face's Transformers library documents a default top_p of 1.0 (documentation v5.19.0), and its code skips the filter when the value is not below 1.

As p shrinks, so does the group. Once p is no larger than the top candidate's own probability, that candidate alone reaches p, the group holds one token, and that token wins every time. Hugging Face's code always keeps at least one token. At this end, top-p behaves like always taking the most likely word, the greedy decoding described on the next-token page.

The tail is what the filter targets. Holtzman and colleagues found that sampling straight from the full distribution produced incoherent text, and blamed an unreliable tail of tens of thousands of low-probability tokens that add up to a lot of probability in total.

There is no universal best p. Google's Gemini documentation lists a default topP of 0.95, as of its page updated 2026-09-17. Hugging Face's library defaults to 1.0. Defaults differ by system, so read the one for the system you use.

How it fits with temperature

Temperature and top-p both act before a token is drawn, and they do different jobs. Temperature reshapes all the probabilities, flatter or sharper. Top-p removes candidates from the bottom. The temperature page explains the first, and this page covers the second.

The order is not the same everywhere. In the Hugging Face Transformers code, read on 2026-10-11, temperature scaling comes first, then top-k, then top-p, so the running total is computed on already reshaped probabilities. Google's Gemini documentation describes topK first, then topP, with the final token selected using temperature sampling. The same two numbers can therefore give different groups in different systems.

Vendor advice, as of 2026-10-11. OpenAI's API reference says it generally recommends altering temperature or top_p but not both. Google's Gemini documentation strongly recommends keeping temperature, topK and topP at their defaults for Gemini 3.x models. Anthropic's API reference marks top_p as deprecated: models released after Claude Opus 4.6 do not support setting it, and values below 0.99 are rejected with an error.

The documents give no reason for the one-at-a-time advice. Both settings change what can be picked, so changing one at a time is the way to see which did what.

What top-p does not do

Top-p only removes candidates the model already rated as unlikely. It adds no knowledge, and it cannot bring in a word the model did not score. If the model puts 92% on a wrong answer, that answer is the top candidate, so it is always in the group.

That is why top-p does not stop hallucinations. A hallucination is fluent text the model rates as probable, and the filter keeps probable text. It can remove wild picks from the tail, which is the failure Holtzman and colleagues measured, but a wrong answer at the top of the distribution passes straight through. The hallucination page covers why.

A group with several words still means a random draw, so the same prompt can give different answers. Top-p also has no say over length, tone or instructions. Those come from the prompt and the training.

FAQ

What is top-p sampling in simple terms?: It is a rule for picking the next word. The model sorts its candidate words by probability, keeps the fewest that together reach the chosen share p, and picks one of those at random in proportion to their probabilities. Everything below the cutoff is ignored.
What is the difference between top-p and top-k?: Top-k keeps a fixed number of candidates, such as the 10 most likely. Top-p keeps however many it takes to reach a share of the probability, so it keeps few words when the model is confident and more when it is unsure. Holtzman and colleagues argue that one fixed k suits some steps and not others.
Is nucleus sampling the same as top-p?: Yes. Nucleus sampling is the name from the 2019 paper by Holtzman and colleagues. Provider documentation calls the setting top_p, and OpenAI's reference describes it as an alternative to sampling with temperature, called nucleus sampling.
What is a good top-p value?: There is no universal one. Documented defaults differ: Hugging Face's library uses 1.0 and Google's Gemini documentation lists 0.95. For some current models the setting cannot be changed at all, so check the documentation of the model you use.
Should I change temperature and top-p together?: Provider advice, as of 2026-10-11, is to change one, not both. OpenAI's reference says it generally recommends altering one or the other, and Google's documentation recommends keeping defaults for Gemini 3.x models.

Sources

Last updated 2026-10-11