Top-p sampling, also called nucleus sampling, keeps the smallest group of tokens whose probabilities add up to p, then samples from that group. The shortlist adapts automatically to how confident the model is.
Top-k sampling is a decoding method that keeps only the k most likely next tokens, renormalises their probabilities and samples from that shortlist, adding variety while blocking very unlikely words.
Beam search is a decoding method that keeps several candidate sequences alive at once and expands them step by step, choosing the sequence with the best overall probability rather than the best next token.
Greedy decoding is the simplest way to generate text from a language model: at every step it picks the single most likely next token, without any randomness.
Model alignment is the work of making an AI system behave in line with human intentions, values and safety expectations, covering both training methods and the deeper research problem of specifying what we actually want.
Preference tuning is the training stage where a model learns from comparisons between better and worse responses, rather than from single correct answers, making its output more helpful, safe and aligned with what people want.
Time to first token (TTFT) is the delay between sending a request and the model producing its first output token. It is the metric users feel most, because it determines how long a screen stays blank.
Inference latency is the time between sending a request to a model and receiving the response. For LLMs it breaks down into queueing, prompt processing and token generation.
Maximum output tokens is the setting that caps how many tokens a model may generate in one response. It controls cost and latency, and if set too low, it cuts answers off mid-sentence.
Presence penalty is an API setting that applies a fixed, one-time reduction to the probability of any token that has already appeared, encouraging a model to introduce new words and topics.
Frequency penalty is an API setting that reduces a token's likelihood in proportion to how many times it has already appeared in the output, discouraging overused words while allowing occasional repeats.
Repetition penalty is a decoding setting that lowers the probability of tokens that have already appeared, discouraging a language model from repeating the same words or phrases in a loop.
Model pruning removes weights, neurons or entire components from a trained neural network to make it smaller and faster, ideally with minimal loss of accuracy.
Speculative decoding speeds up text generation by having a small, fast model draft several tokens ahead, which the large model then verifies in a single pass, accepting the ones it agrees with.
Streaming inference delivers a model's output token by token as it is generated, rather than waiting for the complete response. It makes applications feel far faster without changing total generation time.