AI Glossaries

Batch inference is processing multiple requests together rather than one at a time. It raises hardware throughput and lowers cost per request, at the cost of some individual latency.

Tokens per second measures how quickly a language model generates output. It is reported either per user, describing how fast text streams, or as total throughput across all concurrent requests.

Sparsity in LLMs means that a large share of a model's weights or activations are zero, or that only part of the model is used for any given input, which can reduce memory and computation.

GGUF is a model file format used by llama.cpp and tools built on it, such as Ollama and LM Studio. It packs quantized weights, tokenizer and metadata into a single file designed for local inference.

AWQ (activation-aware weight quantization) is a quantization method that identifies the small fraction of weights most important to model output, based on activation magnitudes, and protects them by scaling before quantizing.

GPTQ is a post-training quantization method that compresses a model's weights to 3 or 4 bits in one pass, using a small calibration dataset and second-order information to minimise the resulting error.

Tensor parallelism splits individual weight matrices across multiple GPUs so they compute a single layer together, exchanging partial results. It is the standard way to run very large models within one server.

Data parallelism is the most common way to train models on multiple GPUs: each device holds a full copy of the model, processes a different slice of the batch, and gradients are averaged across devices.

Model parallelism is the practice of splitting a single model across several GPUs because it is too large to fit on one, dividing either its layers, its individual weight matrices, or both.

Mixture of experts (MoE) is an architecture where each token is routed to a small number of expert sub-networks rather than passing through the entire model, giving large capacity at lower compute cost per token.

Pipeline parallelism splits a model's layers into stages placed on different devices, which process different micro-batches at the same time, like an assembly line.

A decoder-only model is a transformer built from a single stack of decoder layers that generates text one token at a time using causal attention. Most modern large language models use this design.

A feed-forward network (FFN) is the part of a transformer block that processes each token's representation independently through two linear layers with an activation in between. It holds most of a transformer's parameters.

8-bit quantization stores model weights using eight bits each, halving memory compared with 16-bit precision while usually preserving quality almost exactly.

4-bit quantization stores a model's weights using only four bits each, cutting memory use to roughly a quarter of 16-bit precision and making large models runnable on consumer hardware.