Vector Databases & Semantic Search

Semantic search retrieves content based on meaning rather than requiring exact keyword matches. Vector databases make this practical at scale, but production systems must balance retrieval quality, latency, cost, freshness, filtering, and operational complexity. These questions focus on the design decisions behind reliable vector retrieval.

1. What is an embedding, and how is it used in semantic search?

Answer: An embedding is a numerical vector that represents content—such as text, an image, or another object—in a learned semantic space. An embedding model maps similar meanings to vectors that are close under a chosen similarity or distance measure.

How semantic search works

1. Prepare content: Clean and chunk documents into meaningful searchable units.
2. Generate document vectors: Pass each chunk through an embedding model and store its vector with a stable ID and metadata.
3. Embed the query: Convert the user's query into a vector using a compatible model and configuration.
4. Retrieve neighbors: Search for vectors nearest to the query vector using cosine similarity, dot product, or an appropriate distance metric.
5. Apply constraints: Enforce tenant, authorization, language, and other metadata filters.
6. Use the results: Return passages, optionally rerank them, and provide them to a downstream application or RAG model.

Example: A search for “How can I reset my login credentials?” may retrieve a document titled “Password recovery procedure,” even though the query and document share few exact words.

Important caveats: Embeddings do not guarantee truth or relevance. Results depend on the model, chunking, language, domain, and similarity metric. Keep the model version and vector dimension consistent between indexed documents and queries.

2. How would you choose a vector database for an enterprise application?

Answer: Start with workload requirements and representative retrieval tests rather than selecting a product by popularity. Compare recall, latency, filtering, update patterns, scale, security, operational effort, and total cost under realistic conditions.

Dimension	Questions to ask
Scale and capacity	How many vectors? What dimensions, metadata size, growth rate, and query concurrency?
Retrieval quality	What recall@k or task-level quality is needed? Which metrics and index types are supported?
Latency and throughput	What are p50/p95/p99 latency and sustained QPS under realistic filters?
Metadata filtering	Can it efficiently filter by tenant, ACL, date, category, or status?
Update and deletion	How well does it handle frequent upserts, deletes, compaction, and index rebuilds?
Operations and security	Managed or self-hosted? Backup, restore, encryption, isolation, audit, and access controls?
Cost and portability	What are storage, memory, compute, replication, network, and migration costs?

Selection process

1. Define workload shape: vector count, dimensions, metadata, filters, write rate, query rate, and freshness requirements.
2. Create a representative benchmark dataset and realistic query set with relevance judgments.
3. Compare exact search and candidate ANN configurations for recall, p95 latency, throughput, and cost.
4. Test authorization filters, updates, deletes, backup/restore, and failure recovery—not only happy-path queries.
5. Choose the simplest option that meets the service-level objectives and can be operated reliably.

Interview tip: A database that wins an unfiltered benchmark may perform poorly with selective ACL filters or frequent updates. Benchmark the actual workload, including filters and concurrency.

3. What is the difference between exact and approximate nearest-neighbor search?

Answer: Exact nearest-neighbor (ENN) search evaluates candidates exhaustively to find the true nearest vectors under the selected metric. Approximate nearest-neighbor (ANN) search uses an index or search strategy to find likely neighbors much faster, while accepting some risk of missing the exact top results.

Aspect	Exact search	Approximate search
Method	Compare query with all eligible vectors	Navigate an index or search selected candidates
Recall	Exact top-k for the defined metric and candidate set	May miss some true nearest neighbors
Latency at scale	Can be expensive for large collections	Usually lower with suitable index and tuning
Best use	Small datasets, ground-truth evaluation, strict exactness	Large collections with latency or throughput constraints

How to tune ANN: Increase search effort or candidate breadth to improve recall, then measure the latency and compute cost. Use exact search on a representative sample to establish ground truth and calculate ANN recall@k.

Trade-off: ANN is not automatically better. For small datasets or highly selective filters, exact search may be simpler and competitive. Choose based on measured service-level objectives.

4. How does HNSW work at a high level?

Answer: HNSW stands for Hierarchical Navigable Small World. It is a graph-based ANN index. Each vector is a node connected to selected nearby nodes. The graph has multiple layers: sparse upper layers support long-range navigation, while denser lower layers support local refinement.

Query process, simplified

1. Start at an entry point in the top layer.
2. Move to neighboring nodes that are closer to the query, repeating until no better move is found at that layer.
3. Descend to the next layer, using the current candidate as the starting point.
4. At the bottom layer, explore a broader candidate set and return the best matches found.

Conceptual HNSW layers

Upper layer: sparse long-range links

↓ navigate closer

Middle layer: more candidate routes

↓ refine search

Bottom layer: dense local-neighbor graph

Important parameters

• M: Approximate limit on graph connections per node; higher values often increase memory use and can improve connectivity/recall.
• efConstruction: Search breadth used while building the graph; higher values can improve index quality but increase build time.
• efSearch: Search breadth during queries; higher values often improve recall at the cost of latency and compute.

Trade-offs: HNSW can offer strong recall and low latency, but it may use substantial memory and require careful handling of updates, deletes, and rebuilds. Actual parameter names and behavior depend on the database implementation.

5. How would you migrate to a new embedding model without disrupting production?

Answer: Treat the embedding-model change as a versioned index migration. Vectors produced by different embedding models are generally not directly comparable, even if their dimensions happen to match. Re-embed documents and queries with compatible model versions.

1. Record current state: Capture model version, preprocessing, chunking settings, dimensions, metric, index configuration, and baseline retrieval metrics.
2. Build a parallel index: Generate new vectors into a separate namespace or index while the existing index continues serving production.
3. Backfill in batches: Use checkpoints, content-hash caching where valid, bounded retries, and progress tracking.
4. Validate coverage: Compare expected document/chunk IDs with successfully embedded and indexed IDs; test metadata and ACL filters.
5. Evaluate quality: Use a fixed query set to measure recall, ranking quality, answer quality, latency, and cost.
6. Shadow or canary: Compare new-model results against production traffic or a small approved traffic slice without changing all users at once.
7. Switch safely: Use a versioned alias or routing configuration to move traffic to the new index, with rollback ready.
8. Retire deliberately: Keep the old index until the new version meets acceptance criteria and rollback is no longer required.

Interview tip: Do not mix old document vectors with new query vectors unless the system explicitly supports that compatibility. Keep model version, index version, and query encoder version aligned.

6. What happens when a vector index contains stale embeddings?

Answer: Retrieval may surface outdated, misleading, or deleted content even if the source system is correct. This can reduce answer quality and, in sensitive systems, expose information that should no longer be available.

• Content drift: A source document changes, but the old chunk/vector remains indexed.
• Deleted content: A removed document remains searchable because deletion did not propagate.
• Version mismatch: The query encoder or index uses a different model version than expected.
• Metadata drift: Permissions, tenant ownership, or classification changed while index metadata stayed old.

Detection and remediation

1. Track source version/hash and indexed version/hash for each document.
2. Monitor source-to-index lag, stale-document counts, and deletion propagation time.
3. Use version-aware upserts and tombstones to suppress retired content immediately.
4. Reprocess only changed documents unless model or preprocessing changes require a broader rebuild.
5. Run reconciliation to find missing, duplicated, or outdated chunks.
6. Ensure the retrieval layer excludes inactive versions and applies current authorization.

Key point: An index can be operationally healthy but semantically stale. Monitor freshness and source-version alignment alongside latency and availability.

7. How would you improve retrieval latency for millions of vectors?

Answer: Measure latency by stage first, then tune the index, filtering, query path, and infrastructure based on the actual bottleneck. Optimize p95/p99 latency and recall together; a fast response that returns poor results is not a successful optimization.

Optimization checklist

• Measure: Separate query embedding time, vector search, metadata filtering, reranking, network overhead, and LLM generation.
• Tune ANN: Adjust search breadth and index parameters while measuring recall@k and p95 latency.
• Filter efficiently: Index frequently used metadata and test highly selective tenant or ACL filters.
• Reduce candidates wisely: Retrieve a suitable top-k, then rerank a smaller candidate set if needed.
• Right-size vectors: Evaluate embedding dimensions and supported compression/quantization options against quality loss.
• Keep hot data close: Use suitable memory/cache policies and place services to reduce network hops.
• Scale and isolate: Partition or shard where useful, provision for concurrency, and isolate heavy indexing jobs from latency-sensitive queries.
• Cache carefully: Cache query embeddings or safe results where useful, but include tenant, permissions, query context, and model/index versions in cache keys as appropriate.

Benchmark plan: Use representative vectors and real query distributions. Record p50/p95/p99 latency, QPS, recall@k, filter selectivity, index build/update time, memory, and cost. Test during concurrent writes as well as steady-state reads.

Example: If p95 latency increases only for queries with strict tenant filters, investigate filtered ANN behavior and metadata indexing before simply adding more compute. If latency rises during backfills, isolate ingestion workloads or reduce indexing concurrency.

8. When would you use hybrid search instead of vector search alone?

Answer: Hybrid search combines semantic vector retrieval with lexical retrieval, such as BM25 or keyword search. It is useful when meaning matters but exact terms—product IDs, error codes, names, legal clauses, version strings, or technical identifiers—must also match reliably.

Approach	Strength	Limitation
Keyword / lexical	Exact terms, IDs, rare names, literal matches	May miss semantically similar wording
Vector	Meaning and paraphrases	May rank semantically similar but exact-term-mismatched results highly
Hybrid	Combines literal and semantic signals	Needs fusion/ranking configuration and evaluation

How to implement it

1. Run lexical and vector searches against the same eligible corpus.
2. Combine candidate lists using a rank-fusion approach such as Reciprocal Rank Fusion (RRF), or use a calibrated weighted score when score scales are comparable.
3. Optionally rerank the combined candidates with a reranker.
4. Evaluate with real queries, including exact identifiers, paraphrases, ambiguous queries, and domain-specific terms.

Example: For “error E1042 after agent version 7.3,” lexical search can preserve the exact error code and version while vector search can find troubleshooting documentation that describes the same issue in different language.

Security note: Apply tenant and authorization constraints to both retrieval paths before combining results. Hybrid search must not introduce a permission bypass through the lexical index.

Quick Revision Table

Question	Key points
Embeddings	Vectors, semantic similarity, compatible models, metadata
Choose vector DB	Recall, p95 latency, filters, scale, updates, security, total cost
Exact vs ANN	Ground truth vs speed/recall trade-off
HNSW	Layered graph, M, efConstruction, efSearch
Model migration	Parallel index, backfill, evaluate, canary, rollback
Stale vectors	Version/hash tracking, tombstones, reconciliation
Latency at scale	Profile stages, tune ANN, filters, caching, isolation
Hybrid search	Lexical + semantic retrieval, fusion, evaluation

Scenario-Based Follow-Up Questions

Expand each scenario to review the core ideas a strong answer should cover.

Scenario 1: Search is fast, but users say the results are less relevant after a model upgrade.

Verify that query and document embeddings use the same model version, compare retrieval metrics on a fixed query set, and check preprocessing and chunking changes. Build a parallel index if needed, compare recall and ranking quality, and roll back the active version if acceptance criteria fail.

Scenario 2: p95 latency doubles when a large backfill begins.

Separate query and indexing resource usage, inspect CPU/memory and queue depth, reduce or cap indexing concurrency, and isolate backfill workloads if supported. Measure recall and latency after tuning; do not sacrifice required authorization filters just to improve speed.

Scenario 3: A semantic search misses an exact product code.

Add lexical retrieval for identifiers and combine its results with vector candidates using rank fusion or calibrated ranking. Ensure normalization preserves punctuation and token boundaries in identifiers, then evaluate both exact-code queries and semantic paraphrases.

Scenario 4: The new index is only 92% complete at release time.

Keep the old index active unless the release policy explicitly allows partial coverage. Compare expected and indexed IDs, retry transient failures, inspect permanent errors, and promote only after completeness, access-control, and retrieval-quality checks pass.

Chapter Summary

• Embeddings represent content in a semantic space; query and document encoders must be compatible.
• Select a vector database using realistic workload benchmarks, not only product features.
• ANN improves search efficiency with a recall/latency trade-off; HNSW uses a layered graph.
• Migrate embedding models with parallel indexes, evaluation, controlled rollout, and rollback.
• Track source and index versions to detect stale embeddings and propagate deletions.
• Optimize retrieval latency with measurement, index tuning, efficient filters, and workload isolation.
• Use hybrid retrieval when both semantic meaning and exact terms matter.

Practice Challenge

Design semantic search for 10 million enterprise document chunks. Define your vector database selection criteria, ANN evaluation approach, HNSW tuning plan, embedding-model migration process, freshness monitoring, latency SLOs, and when you would enable hybrid search.