LLMs

Time to first token (TTFT) is the delay between sending a request and the model producing its first output token. It is the metric users feel most, because it determines how long a screen stays blank.

Inference latency is the time between sending a request to a model and receiving the response. For LLMs it breaks down into queueing, prompt processing and token generation.

Maximum output tokens is the setting that caps how many tokens a model may generate in one response. It controls cost and latency, and if set too low, it cuts answers off mid-sentence.

Presence penalty is an API setting that applies a fixed, one-time reduction to the probability of any token that has already appeared, encouraging a model to introduce new words and topics.

Frequency penalty is an API setting that reduces a token's likelihood in proportion to how many times it has already appeared in the output, discouraging overused words while allowing occasional repeats.

Repetition penalty is a decoding setting that lowers the probability of tokens that have already appeared, discouraging a language model from repeating the same words or phrases in a loop.

Model pruning removes weights, neurons or entire components from a trained neural network to make it smaller and faster, ideally with minimal loss of accuracy.

Speculative decoding speeds up text generation by having a small, fast model draft several tokens ahead, which the large model then verifies in a single pass, accepting the ones it agrees with.

Streaming inference delivers a model's output token by token as it is generated, rather than waiting for the complete response. It makes applications feel far faster without changing total generation time.

Batch inference is processing multiple requests together rather than one at a time. It raises hardware throughput and lowers cost per request, at the cost of some individual latency.

Tokens per second measures how quickly a language model generates output. It is reported either per user, describing how fast text streams, or as total throughput across all concurrent requests.

Sparsity in LLMs means that a large share of a model's weights or activations are zero, or that only part of the model is used for any given input, which can reduce memory and computation.

GGUF is a model file format used by llama.cpp and tools built on it, such as Ollama and LM Studio. It packs quantized weights, tokenizer and metadata into a single file designed for local inference.

AWQ (activation-aware weight quantization) is a quantization method that identifies the small fraction of weights most important to model output, based on activation magnitudes, and protects them by scaling before quantizing.

GPTQ is a post-training quantization method that compresses a model's weights to 3 or 4 bits in one pass, using a small calibration dataset and second-order information to minimise the resulting error.