AI Glossaries

A causal language model predicts each token using only the tokens that came before it. This left-to-right constraint, enforced by causal masking, is what makes a model able to generate text.

Adapter tuning is a parameter-efficient fine-tuning method that inserts small trainable layers, called adapters, inside a frozen pretrained model, so only those layers are trained for each new task.

QLoRA is a fine-tuning method that loads a base model in 4-bit precision and trains small LoRA adapters on top, cutting memory use enough to fine-tune very large models on a single GPU.

Data curation for LLMs is the process of selecting, cleaning, filtering, deduplicating and balancing the text used to train a model. It has a direct effect on model quality, safety and cost.

Synthetic data for LLM training is text generated by AI models rather than collected from humans, used to create instruction examples, reasoning traces and domain data where real data is scarce or private.

Continued pretraining, also called domain-adaptive pretraining, takes an existing pretrained model and trains it further on large amounts of domain or language-specific text before any task fine-tuning.

Catastrophic forgetting is when a neural network loses previously learned abilities after being trained on new data. In LLMs it often appears after fine-tuning, when general skills or safety behaviour degrade.

Gradient clipping limits how large gradients can be before they update a model's weights. It prevents exploding gradients and the sudden loss spikes that can destabilise or ruin a training run.

Gradient accumulation is a training technique that sums gradients across several small batches before performing one weight update, letting you simulate a large batch size on limited GPU memory.

Cross-entropy loss measures how far a model's predicted probability distribution is from the correct answer. It is the standard loss function for classification and for training language models.

A learning rate scheduler adjusts the learning rate over the course of training, typically warming it up from near zero and then decaying it, which improves stability early and precision later.

The learning rate is the setting that controls how big a step a model takes when updating its weights during training. Too high and training becomes unstable; too low and it takes far too long.

Batch size is the number of training examples a model processes together before updating its weights. It affects memory use, training speed, stability and the final quality of the model.

A training epoch is one complete pass of the training algorithm through the entire training dataset. Models usually train for several epochs, though large language model pretraining often uses only one pass or less.

Perplexity is a measure of how well a language model predicts a sample of text. It can be read as the average number of equally likely choices the model is considering at each token, so lower is better.