AI Glossaries

Prefix tuning is a parameter-efficient fine-tuning method that learns trainable vectors inserted into the attention keys and values at every layer of a frozen model, giving more control than input-only soft prompts.

Prompt tuning is a parameter-efficient fine-tuning method that learns a small set of continuous vectors, called a soft prompt, which are prepended to the input while the model's weights stay frozen.

An encoder-decoder model, also called a sequence-to-sequence model, uses one component to read and understand the input and another to generate the output, connected by cross-attention.

RoPE (rotary positional embedding) encodes a token's position by rotating its query and key vectors by an angle proportional to that position, so attention naturally depends on the distance between tokens.

Positional encoding is the method transformers use to represent the order of tokens in a sequence, because the attention mechanism by itself treats input as an unordered set.

An attention head is a single attention unit inside a transformer layer, with its own query, key and value projections. Each head learns to focus on a particular kind of relationship between tokens.

Multi-head attention runs several attention operations in parallel, each with its own learned projections, so a model can capture different kinds of relationships at once and then combine them.

Self-attention is a mechanism where each position in a sequence attends to every other position in the same sequence, using queries, keys and values to decide how much information to take from each.

An attention mechanism is a technique that lets a neural network focus on the most relevant parts of its input when producing each part of its output, instead of treating all input equally.