An encoder is a neural network component that converts raw input, such as text, an image or audio, into a compact numeric representation that captures its most important features. This representation is often called an embedding, a latent vector or a hidden representation. Other parts of a system then use it to classify, compare, search or generate output, often through a decoder.
Key takeaways
- An encoder turns input into numbers that capture meaning.
- Encoders can stand alone (encoder-only models such as BERT) or pair with decoders.
- Their outputs power semantic search, classification and recommendation.
- Many generative systems use encoders, for example to turn a text prompt into guidance for an image model.
- An encoder’s quality depends on what it was trained to preserve.
An encoder explained in simple terms
Imagine summarising a long book into a short index card that still lets someone tell what the book is about and how it compares to other books. You lose the exact wording, but keep the essence. An encoder does this for data. A sentence becomes a list of numbers. Similar sentences get similar numbers, even if they use different words. “How do I reset my password?” and “I forgot my login” end up close together.
How an encoder works
- Input preparation: text is split into tokens; images into pixels or patches; audio into short frames.
- Initial embedding: each token or patch is mapped to a vector.
- Processing layers: layers of a neural network, such as convolutions or transformer attention, mix information so each part is understood in context.
- Output representation: either one vector per token, or one pooled vector summarising the whole input.
Types of encoders
| Encoder type | Built with | Input | Example use |
|---|---|---|---|
| Text encoder | Transformer | Sentences, documents | Semantic search, classification |
| Image encoder | CNN or vision transformer | Images | Image search, visual question answering |
| Audio encoder | CNN or transformer | Speech, sound | Speech-to-text |
| Sequence encoder | RNN, LSTM | Time series, older text models | Forecasting, early translation |
| Autoencoder’s encoder | Any of the above | Any | Compression, anomaly detection (autoencoder) |
Encoder-only models
Some models use only an encoder. BERT, released by Google in 2018, is the best-known example. It reads text in both directions at once, making it strong at understanding tasks: classification, sentiment, named entity recognition and search ranking. It does not generate free-flowing text. Encoder-only models are often smaller and cheaper than generative models, and remain widely used for embeddings and classification.
Encoders in generative AI
Even systems that generate content often rely on encoders:
- Text-to-image models use a text encoder to turn a prompt into guidance for image generation.
- Multimodal models use image and audio encoders to feed visual or sound information into a language model.
- Retrieval systems encode documents and questions so the most relevant passages can be found and passed to a language model.
A worked example: semantic search
A help centre encodes all its 2,000 articles into vectors once. When a user types “can’t log in”, the same encoder turns the question into a vector. The system finds articles whose vectors are closest, often using cosine similarity, and shows “Resetting your password” first, even though the words differ. This is how meaning-based search works.
Encoder vs decoder vs encoder-decoder
| Architecture | What it does | Examples |
|---|---|---|
| Encoder-only | Understands input, outputs representations | BERT, many embedding models |
| Decoder-only | Generates output one token at a time | GPT-style language models |
| Encoder-decoder | Encoder reads input; decoder generates a new sequence | Translation models, T5 |
Limitations
- Encoding compresses information; some detail is always lost.
- Encoders reflect the biases and gaps in their training data.
- Embeddings from different encoders are not interchangeable.
- Long documents may need splitting, since encoders have input length limits.