You add semantic search to your app. The model embeds the query, the database returns a list of neighbors, and the results look almost right, but sometimes wrong in ways that are hard to explain. A question about "database migrations" surfaces a result about "schema evolution" (fine), but also one about "immigration policy" (not fine). The query is clear. The embedding model has no idea what you meant.
This is the fundamental challenge of vector search: it retrieves what is geometrically close in embedding space, not what is logically correct for your application. Understanding why that happens, and what controls it, is what separates a rough proof-of-concept from a reliable search system.
TLDR
- Vector search encodes text or images as high-dimensional embeddings, then finds stored vectors geometrically close to a query vector.
- Closeness is set by a similarity metric (cosine, dot product, or L2), and the metric must match what the embedding model was trained with.
- Exact search over billions of vectors is too slow, so production uses approximate nearest-neighbor indexes like HNSW or IVFFlat that trade a little recall for a large speedup.
- Retrieval quality depends more on embedding model, chunking, and metric alignment than on which index you pick.
What problem vector search solves#
Keyword search works by matching tokens. If a user searches "fast database," a keyword engine looks for documents containing those exact words. It misses documents that say "low-latency storage" or "high-throughput persistence" even if they answer the same question.
Vector search solves the vocabulary mismatch problem by encoding meaning rather than tokens. A good embedding model places semantically similar content close together in vector space regardless of the exact words used. "Fast database," "low-latency storage," and "high-throughput data layer" can end up near each other because they describe the same concept.
This makes vector search useful for semantic document retrieval, recommendation systems, duplicate detection, image search, and the retrieval step in RAG (retrieval-augmented generation) pipelines. It also makes it sensitive to what the embedding model learned, which is why irrelevant results appear when a query lands in an unexpected neighborhood.
Mental model: meaning as a location#
Think of an embedding as GPS coordinates in a very high-dimensional space. Each piece of content (a sentence, a paragraph, a product description) gets mapped to a specific location. Similar content lands in the same neighborhood. Dissimilar content lands far away.
A vector search query asks: "What are the nearest neighbors of this query's location?" The database computes distances from the query point to every stored point and returns the closest ones.
The challenge is that "neighborhood" is defined by geometry, not domain logic. If you embed both technical articles and news articles into the same space, a query about "Java" might land equidistant between results about the programming language and results about Indonesian geography. The model sees patterns from its training data; it does not understand your product's intent.
| Concept | Analogy |
|---|---|
| Embedding | GPS coordinates in high-dimensional space |
| Embedding model | The function that assigns coordinates |
| Vector index | A spatial lookup structure for those coordinates |
| Similarity search | Finding the nearest coordinate neighbors |
| Recall | What fraction of true neighbors were returned |
How vector search works step by step#
End to end, a query runs through the same six stages every time: the content is embedded once at ingestion, then each incoming query is embedded, searched against the index, filtered, and optionally re-ranked before results return.
Embed the content
An embedding model (such as OpenAI's text-embedding-3-small, a sentence-transformers model, or a domain-specific model) converts your text, image, or other content into a fixed-length array of floating-point numbers. A 1536-dimensional embedding is a list of 1536 numbers that together encode the meaning of the input.
Store the vectors
Each embedding is stored alongside its source document ID. In a dedicated vector database like Pinecone, Qdrant, or Weaviate, this is the primary storage. In PostgreSQL with pgvector, vectors are stored as a column of type vector(1536) alongside your other SQL data.
Build an index
A flat search compares the query vector against every stored vector. That is exact, but O(n) in time. For millions of vectors, this is too slow. An ANN index like HNSW builds a graph structure that allows navigating to nearest neighbors in logarithmic time by skipping most of the dataset.
Embed the query
At query time, the user's input goes through the same embedding model. The resulting query vector must be produced by the same model that produced the stored vectors. Mixing models breaks the geometry.
Search the index
The vector database finds the k nearest neighbors to the query vector according to a similarity metric. It returns the IDs of matching documents, which your application uses to fetch the original content.
Filter and rank
Most production systems apply metadata filters before or after the ANN search (e.g., filter by date, category, or user) and optionally re-rank results using a cross-encoder or LLM scoring step for better precision.
Similarity metrics: how closeness is measured#
The similarity metric determines what "close" means. The three most common are cosine similarity, dot product, and L2 (Euclidean) distance.
Cosine similarity measures the angle between two vectors, ignoring their magnitude. It returns 1 for identical direction (most similar) and 0 for perpendicular vectors. This makes it robust to differences in vector length, which is why it is a common default for text embeddings.
Dot product is the product of magnitudes and cosine similarity combined. For normalized vectors (length = 1), cosine similarity and dot product produce identical rankings. Most modern embedding models return normalized vectors, which is why you often see "use dot product for normalized embeddings": it is computationally cheaper and equivalent. The Weaviate distance metrics guide explains the relationships between these metrics clearly.
L2 (Euclidean) distance measures the straight-line distance between two points. It is sensitive to vector magnitude, which means it encodes both direction and scale. Some embedding models, particularly those trained with contrastive learning on magnitude-sensitive tasks, may perform better with L2.
Always match the metric to the model
Use the same similarity metric that the embedding model was trained with. As Zilliz notes in their FAQ on metric mismatch, using the wrong metric can degrade retrieval quality in ways that are subtle and hard to diagnose. Results are returned, but they are not the most semantically relevant ones.
Key components: ANN indexes#
Exact nearest-neighbor search is O(n). For a million vectors at 1536 dimensions, this is prohibitively slow for interactive applications. ANN indexes solve this with a controllable accuracy-speed tradeoff.
HNSW#
HNSW (Hierarchical Navigable Small World) is the most widely used ANN algorithm in production vector databases. Introduced by Malkov and Yashunin (arXiv:1603.09320), it builds a multi-layer graph of proximity connections. The top layers have long-range connections for fast navigation. The bottom layer contains all vectors with short-range connections for precise lookup.
At query time, HNSW starts at a random entry point in the top layer and greedily follows edges toward closer neighbors, descending layer by layer. This yields logarithmic query complexity on most datasets.
The pgvector extension for PostgreSQL supports HNSW indexes on vector columns. pgvector's official README describes the index as offering better query performance than IVFFlat in terms of the speed-recall tradeoff, though with slower build times and higher memory usage. An HNSW index can be created on an empty table since it does not require a training step.
-- Create a table with vector column
CREATE TABLE documents (
id SERIAL PRIMARY KEY,
content TEXT,
embedding vector(1536)
);
-- Create an HNSW index for cosine similarity search
CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);
-- Query: find 5 nearest neighbors
SELECT id, content, 1 - (embedding <=> '[0.1, 0.2, ...]') AS similarity
FROM documents
ORDER BY embedding <=> '[0.1, 0.2, ...]'
LIMIT 5;
The key HNSW parameters are m (number of connections per node) and ef_construction (size of the candidate list during build). Higher values improve recall but increase build time and memory. The ef_search parameter at query time controls how thoroughly the graph is explored, where higher values improve recall at the cost of query speed.
IVFFlat#
IVFFlat (Inverted File with Flat Quantization) divides vectors into lists (Voronoi cells) and assigns each vector to its nearest centroid. At query time, only the probes closest cells are searched. This is faster to build and uses less memory than HNSW, but has lower query performance at equivalent recall.
pgvector's documentation recommends creating an IVFFlat index only after the table has data, since it requires a training step to determine centroids. A suggested starting point is rows / 1000 lists for up to one million rows.
CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);
-- At query time, control the recall-speed tradeoff:
SET ivfflat.probes = 10;
FAISS#
Meta's FAISS (Facebook AI Similarity Search) library is a foundational toolkit for building ANN indexes. It supports dozens of index types, GPU acceleration, and product quantization for compressed-domain search. Many vector databases (including Weaviate) use FAISS internally. It is best used as a library when you need fine-grained control over index structure or GPU-accelerated search, rather than as an end-user database.
| Index type | Build speed | Query speed | Memory | Best for |
|---|---|---|---|---|
| Flat (exact) | Fast | Slow at scale | High | Small datasets, ground truth |
| IVFFlat | Fast | Good | Medium | Moderate scale, fast builds |
| HNSW | Slow | Fast | High | Large scale, high recall targets |
| FAISS + PQ | Medium | Very fast | Low | Billion-scale with compression |
Where vector search goes wrong#
Most vector search failures are not algorithm bugs. They are embedding, chunking, or evaluation mismatches.
Wrong embedding model. A general-purpose model trained on web text may not understand your domain vocabulary. A medical records system needs embeddings trained on clinical text; a code search tool needs embeddings trained on source code. Using an off-the-shelf model for a specialized domain often produces neighborhood pollution: irrelevant results that are geometrically adjacent because the model has weak domain signal.
Poor chunking strategy. Embedding long documents as a single vector blends too many topics into one location. A 20-page technical report averaged into one embedding cannot reliably surface specific paragraphs. Chunking strategy (how you split documents before embedding) has a larger practical impact on retrieval quality than the choice of ANN index.
Metric-model mismatch. Configuring the index to use L2 distance for embeddings trained with cosine objectives silently degrades results.
Low recall at fixed k. HNSW with default parameters can miss true neighbors when ef_search is too small. If your evaluation shows that ground-truth matches are not appearing in top-10 results, increasing ef_search is usually the first lever to pull.
Missing metadata filters. Returning the most semantically similar document regardless of its publication date, access permission, or product category produces results that are correct in embedding space but wrong for the user's actual context.
No evaluation baseline. Teams often tune chunking, prompts, and index parameters without measuring recall@k. Without a labeled evaluation set, you cannot tell whether changes improve retrieval quality.
How to debug and reason about retrieval#
When results look wrong, the debug loop has three layers.
First, inspect the embeddings. Embed a few representative queries and their expected results, then compute the similarity directly. If expected results score low (e.g., cosine similarity below 0.7), the problem is in the embedding model or chunking, not the index.
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
query_embedding = embed("what is database migration")
expected_doc_embedding = embed("schema evolution and migration strategy")
print(cosine_similarity(query_embedding, expected_doc_embedding))
# If < 0.7, the model may not understand your domain
Second, check index recall. Run a brute-force exact search on a sample and compare its results to your ANN index results at the same k. If exact search returns significantly better results, increase ef_search (HNSW) or probes (IVFFlat).
Third, evaluate end-to-end. Build a labeled set of (query, expected document) pairs and measure recall@5 and recall@10 across the full pipeline. This is the only reliable way to know whether tuning improved anything.
Practical examples#
RAG document retrieval#
In a retrieval-augmented generation pipeline, the retrieval step uses vector search to find the chunks most likely to answer a question. The chunks are then passed to a language model as context. See how to build a RAG search pipeline for a practical walkthrough. The embedding quality and chunk size are usually the largest drivers of answer quality.
Semantic product search#
An e-commerce catalog with 2 million products can use vector search so that a query for "lightweight hiking boots for summer" surfaces sandals, trail runners, and approach shoes, not just exact keyword matches. Combined with metadata filters (in-stock, size, price range), this gives users relevant results even when they do not know the exact product name.
Duplicate detection#
Embed all support tickets at ingestion time. New tickets can be checked against stored embeddings for similarity above a threshold (e.g., cosine similarity > 0.92). High-similarity pairs are flagged as likely duplicates, reducing agent workload without keyword matching.
Start with exact search
During development, use an exact flat index or brute-force search to establish ground truth. Switch to HNSW only after you have confirmed the embedding model produces useful neighborhoods. Optimizing the ANN index before validating embeddings is a common ordering mistake.
Key Takeaways#
Vector search retrieves by geometric proximity in embedding space, not by semantic understanding. The embedding model defines what "close" means. The ANN index defines how efficiently you can navigate that space.
HNSW is the default choice for most production use cases. It offers the best speed-recall tradeoff and can be built on an empty table. IVFFlat is useful when build speed and memory matter more than query speed at maximum recall.
Retrieval quality depends primarily on embedding model selection, chunking strategy, and similarity metric alignment, not on index type. Measure recall@k against a labeled set before tuning parameters. Filter metadata after (or during) the vector search to enforce business constraints that geometry cannot express.
FAQ#
What is the difference between vector search and semantic search?
They refer to the same general idea but at different levels of abstraction. Semantic search is the product goal: return results that match meaning, not just keywords. Vector search is the technical mechanism: embed content and queries into a shared vector space, then find nearest neighbors. Semantic search can also be implemented with hybrid approaches combining keywords and vectors.
Can I use vector search in PostgreSQL without a separate vector database?
Yes. The pgvector extension adds a vector column type and both HNSW and IVFFlat index types to PostgreSQL. This is a practical starting point if your team already operates PostgreSQL and your dataset is under roughly 10 million vectors. For larger scale or more specialized features (hybrid search, multitenancy, filtering before ANN), a dedicated vector database may offer better ergonomics.
How many dimensions should my embeddings have?
Use the dimensionality your chosen embedding model produces. OpenAI's text-embedding-3-small produces 1536-dimensional vectors by default. Sentence-transformers models typically produce 384 or 768 dimensions. Higher dimensions can encode more nuance but cost more storage and index build time. Some models support dimensionality reduction (e.g., OpenAI's dimensions parameter) which can reduce storage at a small recall cost.
Does vector search replace keyword search?
Rarely on its own. Keyword search handles exact matches, product codes, names, and cases where precision matters more than recall. Vector search handles synonyms, paraphrases, and semantic variants. Production search systems often use hybrid search, combining BM25 keyword scores with vector similarity scores, which consistently outperforms either approach alone.
What is recall@k and why does it matter?
Recall@k measures what fraction of relevant documents appear in the top k results. If a dataset has 10 truly relevant documents for a query and your search returns 7 of them in the top 10, your recall@10 is 0.7. ANN indexes trade some recall for speed, so measuring recall@k against an exact search baseline tells you how much accuracy you are giving up for the speed gain.
