LLMs

Catastrophic forgetting is when a neural network loses previously learned abilities after being trained on new data. In LLMs it often appears after fine-tuning, when general skills or safety behaviour degrade.

Gradient clipping limits how large gradients can be before they update a model's weights. It prevents exploding gradients and the sudden loss spikes that can destabilise or ruin a training run.

Gradient accumulation is a training technique that sums gradients across several small batches before performing one weight update, letting you simulate a large batch size on limited GPU memory.

Cross-entropy loss measures how far a model's predicted probability distribution is from the correct answer. It is the standard loss function for classification and for training language models.

A learning rate scheduler adjusts the learning rate over the course of training, typically warming it up from near zero and then decaying it, which improves stability early and precision later.

The learning rate is the setting that controls how big a step a model takes when updating its weights during training. Too high and training becomes unstable; too low and it takes far too long.

Batch size is the number of training examples a model processes together before updating its weights. It affects memory use, training speed, stability and the final quality of the model.

A training epoch is one complete pass of the training algorithm through the entire training dataset. Models usually train for several epochs, though large language model pretraining often uses only one pass or less.

Perplexity is a measure of how well a language model predicts a sample of text. It can be read as the average number of equally likely choices the model is considering at each token, so lower is better.

At every generation step, an LLM produces a probability distribution over its entire vocabulary: a number for each possible next token, all adding up to 1. Sampling settings decide how a token is chosen from it.

A logit is the raw, unnormalised score a model produces for each possible output before those scores are converted into probabilities. In an LLM, there is one logit per token in the vocabulary at every generation step.

An LLM's vocabulary is the fixed list of tokens the model can recognise and generate. Every input is broken into these tokens, and every output is chosen from them.

Autoregressive generation is the process of producing output one token at a time, where each newly generated token is fed back in as part of the input for the next step.

Next-token prediction is the training objective behind most large language models: given a sequence of tokens, predict the one that comes next. Repeated at massive scale, this simple task produces broad language ability.

Masked language modeling (MLM) is a training method where random tokens in a text are hidden and the model learns to predict them using context from both sides. It is the objective behind BERT and similar encoder models.