Sampling, Logprobs, and Decoding Control

31.1 From Logits to a Token #

At each position, the model produces a logit vector over the vocabulary, and the sampler transforms it and selects. The function below is that sampler, written out in full so the order of operations is visible.

import numpy as np

def sample(logits, temperature=1.0, top_p=1.0, top_k=0, rng=None):
    rng = rng or np.random.default_rng()
    logits = np.asarray(logits, dtype=np.float64)

    if temperature == 0:
        return int(np.argmax(logits))                    # greedy

    logits = logits / temperature                        # 1. scale

    if top_k:                                            # 2. truncate by count
        keep = np.argpartition(logits, -top_k)[-top_k:]  # exactly k indices
        mask = np.zeros(logits.shape, dtype=bool); mask[keep] = True
        logits = np.where(mask, logits, -np.inf)

    probs = np.exp(logits - logits.max())
    probs /= probs.sum()

    if top_p < 1.0:                                      # 3. truncate by mass
        order  = np.argsort(-probs)
        cum    = np.cumsum(probs[order])
        keep   = order[:np.searchsorted(cum, top_p) + 1]
        mask   = np.zeros_like(probs); mask[keep] = 1
        probs *= mask; probs /= probs.sum()

    return int(rng.choice(len(probs), p=probs))          # 4. sample

The order matters. Temperature is applied before normalization, so it changes the shape of the distribution. Top-k and top-p truncate after, so they remove candidates without reshaping the survivors’ relative probabilities.

31.2 What Each Parameter Actually Does #

31.3 Reading Logprobs #

Logprobs convert the model from an opaque generator into an instrument. Where a provider exposes them, they are the most useful diagnostic available.

from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-5.6-terra",
    messages=[{"role": "user",
               "content": "Classify sentiment as positive/negative/neutral. "
                          "Reply with one word.\n\nText: It arrived on time."}],
    logprobs=True,
    top_logprobs=5,
    max_completion_tokens=4,
)

import math
for tok in resp.choices[0].logprobs.content:
    print(f"{tok.token!r}  p={math.exp(tok.logprob):.4f}")
    for alt in tok.top_logprobs:
        print(f"    {alt.token!r:12} {math.exp(alt.logprob):.4f}")

# 'neutral'  p=0.5120
#     'neutral'     0.5120
#     'positive'    0.4610      ← the model is nearly split
#     'negative'    0.0180

That output reveals something no amount of asking ever would. The model reported “neutral” with the confidence of a coin flip between two categories. A system that logs only the answer sees a confident classification; a system that logs the distribution sees a case that needs review.

31.4 Calibration, Properly #

Verbalized confidence—asking the model “how sure are you?”—and token logprobs are two candidate uncertainty signals, and which is better is a question about your task rather than a settled ranking. They measure different quantities: a logprob is a genuine next-token likelihood, while what you usually want is the probability that the answer is correct, and the provenance of a number does not decide whether it predicts that. The assumption that the self-reported one must be worse because it is “just generated text” does not survive the published comparison—Tian et al. report GPT-4’s verbalized confidence on TriviaQA at an expected calibration error of 0.024 against 0.078 for the model’s own label probability, and title the result accordingly. Evaluate both on your data. Whichever you pick needs calibration against ground truth before you can act on a threshold, and that requirement is the part that does not vary.

import math

def token_confidence(logprobs_content, positions=None) -> float:
    """Geometric mean probability across selected token positions.
    Use positions to restrict to the tokens that carry the answer."""
    toks = logprobs_content if positions is None else [logprobs_content[i] for i in positions]
    if not toks:
        return 0.0
    return math.exp(sum(t.logprob for t in toks) / len(toks))

def route_by_confidence(conf: float, low=0.55, high=0.90) -> str:
    if conf >= high: return "auto_accept"
    if conf >= low:  return "human_review"
    return "escalate_to_stronger_model"

Two design notes apply. Restrict the calculation to the tokens that carry the answer—averaging over boilerplate dilutes the signal toward 1.0 and makes everything look confident. And the thresholds must be fitted on labeled data from your own task; the numbers above are placeholders, and using them unfitted is worse than not routing at all.

The calibration procedure runs as follows: collect a few hundred labeled cases, compute confidence for each, bin by confidence decile, and plot observed accuracy per bin. A well-calibrated model produces a diagonal. Most do not, and the deviation is what your thresholds must correct for.

31.5 Stop Sequences and Structural Control #

Stop sequences terminate generation the moment a given string appears. They are underused and they are a genuine cost lever because output is the expensive lane (§21.4).

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=4096,
    stop_sequences=["</analysis>", "\n\n---"],
    messages=[{"role": "user", "content": prompt}],
)
# stop_reason == "stop_sequence" says a stop sequence fired;
# the stop_sequence field says which one.

Two patterns cover most uses. Using a closing tag as a stop sequence lets you generate exactly one structured block and stop, without paying for a trailing explanation—though the API omits the matched sequence from the returned text, so re-append the closing tag if a parser downstream expects it. And prefilling the assistant turn constrains the opening:

messages = [
    {"role": "user", "content": "Extract the fields as JSON."},
    {"role": "assistant", "content": "{"},        # prefill: forces JSON start
]

Prefilling is cheap and effective where it is supported, and support is narrowing—including on the model in the call just above it. Anthropic’s current generation rejects a trailing assistant turn with HTTP 400 on Sonnet 5, Opus 5 and the Fable and 4.6-through-4.8 families, and the documented replacement is a structured output or a system instruction.⁷⁶ Where prefill does work, it eliminates the “Sure! Here’s the JSON:” preamble entirely rather than asking the model not to produce it; where it does not, reach for §32 instead.

31.6 Seeds #

Some providers accept a seed for reproducibility. It fixes the pseudorandom stream used by the sampler and nothing else. It does not address floating-point non-associativity, batch-dependent kernel selection, MoE routing variability, silent infrastructure changes, or which fleet in which geography served the request (§5.2).

A seed reduces variance without producing reproducibility. Treating it as though it did will build systems whose failures cannot be reproduced.