LLMs

Tensor parallelism splits individual weight matrices across multiple GPUs so they compute a single layer together, exchanging partial results. It is the standard way to run very large models within one server.

Data parallelism is the most common way to train models on multiple GPUs: each device holds a full copy of the model, processes a different slice of the batch, and gradients are averaged across devices.

Model parallelism is the practice of splitting a single model across several GPUs because it is too large to fit on one, dividing either its layers, its individual weight matrices, or both.

Mixture of experts (MoE) is an architecture where each token is routed to a small number of expert sub-networks rather than passing through the entire model, giving large capacity at lower compute cost per token.

Pipeline parallelism splits a model's layers into stages placed on different devices, which process different micro-batches at the same time, like an assembly line.

A decoder-only model is a transformer built from a single stack of decoder layers that generates text one token at a time using causal attention. Most modern large language models use this design.

A feed-forward network (FFN) is the part of a transformer block that processes each token's representation independently through two linear layers with an activation in between. It holds most of a transformer's parameters.

8-bit quantization stores model weights using eight bits each, halving memory compared with 16-bit precision while usually preserving quality almost exactly.

4-bit quantization stores a model's weights using only four bits each, cutting memory use to roughly a quarter of 16-bit precision and making large models runnable on consumer hardware.

A causal language model predicts each token using only the tokens that came before it. This left-to-right constraint, enforced by causal masking, is what makes a model able to generate text.

Adapter tuning is a parameter-efficient fine-tuning method that inserts small trainable layers, called adapters, inside a frozen pretrained model, so only those layers are trained for each new task.

QLoRA is a fine-tuning method that loads a base model in 4-bit precision and trains small LoRA adapters on top, cutting memory use enough to fine-tune very large models on a single GPU.

Data curation for LLMs is the process of selecting, cleaning, filtering, deduplicating and balancing the text used to train a model. It has a direct effect on model quality, safety and cost.

Synthetic data for LLM training is text generated by AI models rather than collected from humans, used to create instruction examples, reasoning traces and domain data where real data is scarce or private.

Continued pretraining, also called domain-adaptive pretraining, takes an existing pretrained model and trains it further on large amounts of domain or language-specific text before any task fine-tuning.