Vector search is the operation of finding the stored embeddings closest to a query vector. Comparing a query against every stored vector is exact but slow, so production systems use approximate nearest neighbour (ANN) indexes that find almost all the right results in a fraction of the time. Vector search is the engine inside a vector database.
Key takeaways
- Similarity is measured with cosine similarity, dot product or Euclidean distance.
- Exact search is O(n); ANN indexes make search sublinear.
- HNSW is the most widely used ANN index.
- The key trade-off is recall versus speed and memory.
- Filtering by metadata alongside search needs careful index support.
Vector search explained in simple terms
Finding the nearest restaurant by checking the distance to every restaurant in the country works, but it is slow. In practice you use a map with neighbourhoods and roads, so you only check nearby candidates. ANN indexes build that road network in vector space.
Similarity measures
| Measure | What it captures | Notes |
|---|---|---|
| Cosine similarity | Angle between vectors | Most common for text embeddings |
| Dot product | Angle and magnitude | Equivalent to cosine when vectors are normalised |
| Euclidean (L2) distance | Straight-line distance | Common in image and geometric data |
Use whatever the embedding model was trained for; most text models expect cosine or normalised dot product.
Main ANN index types
| Index | How it works | Strengths | Trade-offs |
|---|---|---|---|
| HNSW | A multi-layer graph where search hops between neighbours | Excellent recall and speed | High memory use |
| IVF | Vectors are clustered; only the nearest clusters are searched | Memory efficient | Recall depends on clusters probed |
| Product quantization (PQ) | Compresses vectors into small codes | Huge memory savings | Lower precision |
| IVF-PQ | Clustering plus compression | Scales to billions | Accuracy loss |
| Flat (exact) | Compare with everything | Perfect recall | Only for small collections |
HNSW, described by Yury Malkov and Dmitry Yashunin, is the default in most vector databases because of its strong speed-to-recall balance.
Recall: the metric that matters
Recall@k is the share of true nearest neighbours that the approximate search actually returns. A recall of 0.95 at k = 10 means 9.5 of the 10 genuinely closest items are found on average.
Worked example: raising HNSW’s search parameter (often `ef_search`) from 50 to 200 might raise recall from 0.92 to 0.99, while increasing query latency from 3 ms to 12 ms. For RAG, where a reranker refines results anyway, 0.92 is often fine. For legal discovery, missing 8% of relevant items may be unacceptable.
Filtering and search together
Real queries usually include conditions: only this customer’s documents, only the current policy version, only the last year. Combining filters with ANN search is harder than it sounds. Pre-filtering can leave too few candidates in the graph; post-filtering can throw away most results. Good vector databases implement filtered search natively, and this is often a key differentiator.
Tuning vector search
- Choose the index type by collection size and memory budget.
- Tune build parameters for index quality and search parameters for recall.
- Retrieve more candidates than you need, then rerank.
- Measure recall on a labelled sample, not just latency.
Limitations
- Approximate results mean occasional misses.
- High-dimensional distances become less discriminative.
- Index building takes time and memory.
- Similarity does not guarantee relevance to the question asked.