At every generation step, an LLM produces a probability distribution over its entire vocabulary: a number for each possible next token, all adding up to 1. Sampling settings decide how a token is chosen from it.
A logit is the raw, unnormalised score a model produces for each possible output before those scores are converted into probabilities. In an LLM, there is one logit per token in the vocabulary at every generation step.
An LLM's vocabulary is the fixed list of tokens the model can recognise and generate. Every input is broken into these tokens, and every output is chosen from them.
Autoregressive generation is the process of producing output one token at a time, where each newly generated token is fed back in as part of the input for the next step.
Next-token prediction is the training objective behind most large language models: given a sequence of tokens, predict the one that comes next. Repeated at massive scale, this simple task produces broad language ability.
Masked language modeling (MLM) is a training method where random tokens in a text are hidden and the model learns to predict them using context from both sides. It is the objective behind BERT and similar encoder models.
An embedding is a learned list of numbers that represents something, such as a word, token, image or product, so that similar things have similar numbers. Embeddings are how neural networks turn symbols into meaning they can compute with.
A context window is the maximum amount of text, measured in tokens, that a large language model can consider at one time. It includes the system prompt, conversation history, documents and the model's own response.
Tokenization is the process of splitting text into tokens, the small units a language model can process, and mapping each to a number. Most modern LLMs use subword tokenizers such as byte-pair encoding (BPE).
A token is the basic unit of text a large language model reads and generates. A token can be a whole word, part of a word, a punctuation mark or a space. LLM context limits and pricing are measured in tokens.
A transformer is a neural network architecture that uses a mechanism called attention to relate every part of an input to every other part. Introduced in 2017, it is the foundation of nearly all modern large language models.
GPT stands for Generative Pre-trained Transformer. It is OpenAI's family of large language models, which power ChatGPT, and the name describes how the models work: they generate text, are pre-trained on large datasets and use the transformer architecture.
A large language model (LLM) is an AI model trained on enormous amounts of text to understand and generate human language. LLMs power chat assistants, writing tools, coding assistants and AI search.
A small language model (SLM) is a compact language model, typically with millions to a few billion parameters, designed to run efficiently on phones, laptops or low-cost servers while handling focused language tasks well.
A foundation model is a large AI model trained on broad, diverse data at scale that can be adapted to a wide range of tasks. Large language models, image generators and multimodal models are common examples.