An embedding is a learned list of numbers that represents something, such as a word, token, image or product, so that similar things have similar numbers. Embeddings are how neural networks turn symbols into meaning they can compute with.
A context window is the maximum amount of text, measured in tokens, that a large language model can consider at one time. It includes the system prompt, conversation history, documents and the model's own response.
Tokenization is the process of splitting text into tokens, the small units a language model can process, and mapping each to a number. Most modern LLMs use subword tokenizers such as byte-pair encoding (BPE).
A token is the basic unit of text a large language model reads and generates. A token can be a whole word, part of a word, a punctuation mark or a space. LLM context limits and pricing are measured in tokens.
A transformer is a neural network architecture that uses a mechanism called attention to relate every part of an input to every other part. Introduced in 2017, it is the foundation of nearly all modern large language models.
GPT stands for Generative Pre-trained Transformer. It is OpenAI's family of large language models, which power ChatGPT, and the name describes how the models work: they generate text, are pre-trained on large datasets and use the transformer architecture.
A large language model (LLM) is an AI model trained on enormous amounts of text to understand and generate human language. LLMs power chat assistants, writing tools, coding assistants and AI search.
A small language model (SLM) is a compact language model, typically with millions to a few billion parameters, designed to run efficiently on phones, laptops or low-cost servers while handling focused language tasks well.
A foundation model is a large AI model trained on broad, diverse data at scale that can be adapted to a wide range of tasks. Large language models, image generators and multimodal models are common examples.
An open-weight model is an AI model whose trained weights are published so anyone can download, run and often fine-tune it. It differs from fully open-source AI, which also shares training code and data details.
Model distillation, or knowledge distillation, trains a smaller 'student' model to imitate a larger 'teacher' model. The student learns from the teacher's outputs and becomes cheaper and faster to run while keeping much of the teacher's ability.
Model quantization reduces the precision of a model's numbers, for example from 16-bit to 8-bit or 4-bit, so it uses less memory and runs faster, usually with only a small loss in quality.
DPO (direct preference optimization) is a method for aligning language models with human preferences. It trains directly on pairs of preferred and rejected responses, avoiding the separate reward model and reinforcement learning loop used in RLHF.
RLHF (reinforcement learning from human feedback) is a training method where people compare AI responses, a reward model learns their preferences, and the language model is then optimised with reinforcement learning to produce responses people prefer.
Instruction tuning is a fine-tuning step that trains a pretrained language model on many examples of instructions paired with good responses, turning a text-continuing base model into an assistant that follows requests.