An embedding is a learned list of numbers (a vector) that represents something such as a word, token, sentence, image or product, placed so that similar things end up close together. Embeddings are how neural networks turn discrete symbols into meaningful numbers they can compute with. Every LLM starts by converting each token into an embedding.
This page covers embeddings as a core concept inside models. For using embeddings in search and databases, see What is a Vector Embedding?
Key takeaways
- An embedding maps an item to a point in a many-dimensional space.
- Distance and direction in that space capture similarity and relationships.
- Embeddings are learned during training, not hand-designed.
- LLMs use embedding layers to turn tokens into vectors, and refine them into contextual embeddings.
- The same idea applies to images, audio, users, products and more.
An embedding explained in simple terms
Picture a map of foods. Mangoes sit near papayas, biryani near pulao, and ice cream far from pickles. Nobody placed them by hand; the positions came from studying how foods are described and eaten together. An embedding is that kind of map, except with hundreds or thousands of dimensions instead of two, so it can capture many kinds of similarity at once.
How embeddings are learned
Embeddings start as random numbers. During training, the model adjusts them so that items used in similar contexts move closer together. The idea echoes a linguistic principle often summarised as “you shall know a word by the company it keeps”. Word2vec (2013, Tomas Mikolov and colleagues at Google) made word embeddings famous. It showed that vector arithmetic could capture relationships: king − man + woman ≈ queen.
Static vs contextual embeddings
| Type | Meaning | Example |
|---|---|---|
| Static | One fixed vector per word, whatever the context | Word2vec, GloVe |
| Contextual | The vector changes depending on surrounding words | Transformer layers in BERT and LLMs |
A static embedding gives “bank” one vector. A contextual model produces different vectors for “river bank” and “bank loan”. Modern transformers start with static token embeddings and then refine them layer by layer into contextual ones.
A worked example: how an LLM uses embeddings
- The word “Pune” becomes a token ID, say 48213.
- The embedding layer looks up row 48213 in a large table and returns a vector of, for example, 4,096 numbers.
- Positional information is added so the model knows where the token appears.
- Transformer layers mix this vector with those of surrounding tokens. After many layers, the vector for “Pune” in “flights to Pune” carries meaning about travel and location.
- At the output, the model compares its final vector with the embeddings of all tokens to predict the next token.
The embedding table is one of the model’s largest sets of weights: vocabulary size × embedding dimension.
Types of embeddings
- Token embeddings: inside every language model.
- Sentence and document embeddings: one vector per passage, used in search and clustering.
- Image embeddings: produced by vision encoders.
- Multimodal embeddings: text and images in a shared space, as in CLIP.
- User and item embeddings: used in recommendation systems.
- Graph embeddings: represent nodes in networks.
Why embeddings matter
- They let models generalise: learning about “cat” helps with “kitten”.
- They make similarity measurable, enabling semantic search and recommendations.
- They compress rich information into compact, reusable form.
Limitations
- Embeddings reflect biases in training data; early word embeddings showed gender stereotypes.
- Dimensions are rarely interpretable individually.
- Embeddings from different models cannot be mixed.
- Static embeddings ignore context and multiple meanings.