Evaluation Datasets & Regression Testing for RAG

Reliable Retrieval-Augmented Generation (RAG) applications need more than a few example prompts. They need a representative, versioned evaluation dataset that tests whether the system retrieves the right evidence and produces accurate, grounded, safe answers. A good evaluation process helps teams compare chunking strategies, embedding models, prompts, and retrieval configurations without relying only on subjective impressions.

Interview framework

Explain the evaluation objective, representative query coverage, trusted reference evidence, separate retrieval and answer metrics, leakage controls, repeatable test runs, and decision criteria for accepting a change.

1. How would you create an evaluation dataset for an enterprise RAG application?

Answer: I would start from real user tasks and business risks, then create a curated, versioned set of queries with trusted evidence and expected outcomes. The dataset should represent both common requests and high-impact edge cases.

Define the scope: list user personas, key workflows, supported domains, and the failure modes that matter—such as incorrect policy answers, missed evidence, stale content, or unauthorized retrieval.
Sample real queries: use privacy-approved and redacted production queries, support tickets, search logs, and domain-expert examples. Include paraphrases and different levels of ambiguity.
Build ground truth: for each query, record expected answer points, relevant document IDs or passages, acceptable answer variations, and whether the correct behavior is to abstain or ask a clarification.
Cover meaningful slices: include departments, document types, query complexity, rare topics, exact identifiers, time-sensitive questions, multilingual queries if applicable, and permission-sensitive cases.
Review with experts: have subject-matter experts validate relevance and answerability. Track disagreements and adjudicate ambiguous cases.
Version and govern: store dataset version, source document version, annotation guidelines, labels, reviewer, and date. Apply access controls and remove sensitive data.
Keep a locked holdout: maintain a test set that is not repeatedly used to tune prompts or retrieval settings.

A useful evaluation record might contain:

{
  "query_id": "q-1042",
  "query": "What is the approval limit for this expense type?",
  "expected_answer_points": ["Limit depends on expense category and role"],
  "relevant_sources": ["policy-17#section-4"],
  "answerable": true,
  "risk_level": "high",
  "slice": ["policy", "permissions"]
}

Important: do not treat one exact reference sentence as the only valid answer. For many enterprise questions, grade factual claims, required points, evidence support, and appropriate abstention rather than exact wording.

2. What is the difference between an evaluation dataset and a training dataset?

Answer: A training dataset is used to fit or adapt a model. An evaluation dataset measures how well a system performs on examples that should not be used to optimize that same run. In RAG, evaluation can cover the entire system—retriever, reranker, prompt, model, tools, and output policy—not just model weights.

Dimension	Training dataset	Evaluation dataset
Purpose	Learn model parameters or behavior	Measure quality and compare candidates
Use during tuning	Directly used for optimization	Validation can guide tuning; final holdout stays protected
Labels	Training targets or preference labels	Reference evidence, expected points, relevance, safety outcomes
Main risk	Biased or noisy learning examples	Leakage, unrepresentative coverage, overfitting to benchmark
Refresh cadence	Based on model-development strategy	Updated as product scope and production failures evolve

Nuance: RAG does not always require fine-tuning. You can use an evaluation dataset to compare chunking, retrieval, reranking, context construction, and prompting without training the language model. Keep the final test set separate from examples used to tune those choices.

3. How would you prevent data leakage between training and test sets?

Answer: Prevent examples or information from the test set from influencing model training, retrieval tuning, prompt design, annotation decisions, or repeated manual optimization. Leakage can happen even when the rows are not identical.

Deduplicate before splitting: identify exact duplicates and near-duplicates using normalized text, hashes, or similarity checks.
Split by source or entity when needed: keep chunks from the same source document, document family, customer, or incident in one split if cross-split overlap would make evaluation unrealistic.
Use time-based splits for changing knowledge: evaluate on later queries or later document versions when the production question is about future performance.
Protect holdout labels: do not use final-test answers to create prompt examples, tune thresholds, or repeatedly select the best configuration.
Audit retrieval overlap: ensure test answers are not inadvertently copied into training examples, synthetic prompts, or special-case rules.
Control benchmark exposure: limit access to the locked set and track every evaluation run against its version.

Example: If one policy document is split into 100 chunks, randomly placing 80 chunks in training and 20 in testing may inflate performance because the sets contain nearly identical context. Split at the document or document-family level where appropriate.

Interview tip: separate a development/validation set used for iteration from a locked final test set used for release decisions. If the final test set is repeatedly inspected and optimized against, it gradually becomes another validation set.

4. How do you evaluate retrieval quality separately from answer quality?

Answer: Evaluate the retrieval stage and generation stage independently, then evaluate the complete RAG pipeline. This helps identify whether a failure happened because relevant evidence was not found or because the model failed to use the evidence correctly.

Layer	Useful metrics	What it tells you
Retrieval	Recall@k, Precision@k, MRR, nDCG	Whether relevant evidence appears, and how highly it ranks
Answer	Correctness, completeness, groundedness, citation accuracy	Whether the response is correct and supported by evidence
End-to-end	Task success, appropriate abstention, latency, cost, safety failures	Whether the user task succeeds within operational constraints

Retrieval definitions: Recall@k measures the proportion of known relevant items found in the top k results. Precision@k measures how many of the top k results are relevant. MRR rewards ranking the first relevant result highly; nDCG accounts for graded relevance and ranking position.

Answer evaluation: use expert labels or a carefully calibrated judge to score correctness, completeness, grounding in retrieved context, citation support, and refusal/abstention when evidence is insufficient. Validate automated judge scores against human reviews, especially for high-risk domains.

Diagnostic example: low retrieval recall points toward chunking, indexing, query rewriting, filters, or embeddings. High retrieval recall but low groundedness suggests context ordering, prompt design, model behavior, or answer-generation problems.

5. How would you build a regression-testing pipeline for an LLM application?

Answer: Build an automated, versioned evaluation pipeline that runs on every material change to the model, prompt, retrieval configuration, embedding model, chunking logic, or source-data pipeline. Combine hard checks with graded quality measures.

Version all inputs: record evaluation dataset version, code commit, model/version, prompt, retriever settings, index snapshot, and embedding configuration.
Run deterministic checks: validate schemas, required fields, tool permissions, citation format, refusal behavior, and output contracts.
Run retrieval tests: calculate Recall@k, MRR/nDCG, and relevant-source coverage.
Run answer evaluations: score correctness, completeness, groundedness, citation support, and appropriate abstention.
Run safety and access tests: include unauthorized queries, prompt injection attempts, sensitive data, stale sources, and tool misuse.
Compare against baseline: calculate aggregate and slice-level changes, confidence intervals where appropriate, latency, and cost.
Gate deployment: block releases for critical regressions, investigate meaningful drops, and use canary or shadow evaluation for uncertain changes.
Store artifacts: save inputs, outputs, evidence IDs, metric results, judge version, and run metadata for reproducibility.

Code / prompt / index change
  -> run fixed evaluation dataset
  -> retrieval + answer + safety checks
  -> compare with approved baseline
  -> inspect regressions by slice
  -> pass gate, roll back, or require review

Important: LLM outputs can vary across runs. Use deterministic checks where possible, repeat evaluations for noisy tests, and avoid treating tiny score changes as meaningful without considering variance and business impact.

6. What are the risks of using synthetic data for evaluation?

Answer: Synthetic data can expand coverage cheaply, but it may not reflect real user language, real ambiguity, true document complexity, or actual failure modes. If the same model or prompt generates and grades the examples, the benchmark may favor that model's style and assumptions.

Distribution mismatch: generated queries may be cleaner and more explicit than production requests.
Label errors: synthetic reference answers or evidence mappings may be incorrect or oversimplified.
Model bias: the generator may repeat its own blind spots, stereotypes, and preferred reasoning patterns.
Benchmark circularity: using the same model family to generate examples and judge outputs can reward stylistic similarity rather than factual quality.
Weak edge-case coverage: generated examples may miss real permission conflicts, outdated sources, acronyms, typos, and complex multi-hop questions.
False confidence: a large synthetic set can create precise-looking metrics without representing real risk.

Best practice: use synthetic data to supplement—not replace—real, privacy-safe production examples and expert-authored cases. Validate samples manually, track which examples are synthetic, measure performance separately by origin, and include adversarial and rare cases designed by domain experts.

Interview tip: synthetic data is excellent for coverage expansion and controlled perturbations, but the final release decision should rely on a trusted evaluation set with independently validated labels.

7. How would you identify gaps in a golden evaluation dataset?

Answer: Compare the dataset against real production traffic, product requirements, source coverage, and observed failures. A golden dataset should evolve deliberately, while retaining a stable core for longitudinal comparison.

Analyze production query patterns: cluster privacy-safe queries by intent, topic, complexity, language, and user group. Compare their distribution with the evaluation set.
Review failure logs: turn confirmed retrieval misses, hallucinations, citation errors, stale-content cases, and unauthorized-access attempts into new test cases.
Build a coverage matrix: track topics, document types, source systems, query complexity, risk level, freshness requirements, and access-control scenarios.
Measure slice performance: look for groups with too few examples or consistently weaker results, not just aggregate score.
Run expert review: ask domain experts to identify high-impact questions that are absent, ambiguous, or incorrectly labeled.
Track dataset health: measure duplicate rate, label disagreement, outdated references, unanswerable examples, and examples whose source documents no longer exist.
Promote carefully: version new cases, document why they were added, and preserve a stable core set for comparison across releases.

Example: A policy assistant may score well overall but have almost no examples involving recently updated policies or users with restricted access. Aggregate quality hides these gaps. Add targeted cases and report results for those slices separately.

8. How would you measure the effect of changing a chunking strategy or embedding model?

Answer: Run a controlled experiment where the candidate change is the primary difference. Keep the evaluation queries, relevance labels, source snapshot, downstream model, prompt, and other retrieval settings fixed where possible.

Define the hypothesis: for example, smaller chunks may improve passage precision but lose context; a new embedding model may improve semantic recall for domain terminology.
Build comparable indexes: index the same source snapshot using the baseline and candidate chunking/embedding configurations. Record versions and preprocessing choices.
Run the same queries: evaluate each configuration against the same held-out queries and relevance judgments.
Measure retrieval: compare Recall@k, Precision@k, MRR/nDCG, duplicate result rate, and coverage by query slice.
Measure end-to-end answers: compare grounded correctness, completeness, citation support, abstention, latency, token usage, and cost.
Inspect failures: review queries where the candidate improves or regresses. Segment by long documents, tables, exact identifiers, acronyms, and multi-hop questions.
Assess significance and risk: use paired comparisons on the same examples, estimate uncertainty, and apply minimum thresholds for critical slices.
Deploy safely: use shadow traffic or a canary, monitor real behavior, and keep the baseline index available for rollback.

Chunking trade-off: small chunks can improve focused matching but may omit context; large chunks preserve context but can add irrelevant text and consume more tokens. Overlap can preserve boundary context but increases index size and duplicate retrieval.

Embedding trade-off: a new model may improve semantic retrieval but changes vector dimensions, distance behavior, latency, cost, and index compatibility. Usually build a separate index and compare before switching production traffic.

Fair comparison rule

Change one major variable at a time when diagnosing impact. If multiple components must change together, treat the result as a system-level experiment and avoid attributing the gain to one component.

Quick Revision Table

Topic	Key takeaway
Evaluation dataset	Representative queries, trusted evidence, expected outcomes, risk slices, and versioning.
Training vs evaluation	Training optimizes behavior; evaluation measures it. Keep a locked final holdout.
Leakage prevention	Deduplicate and split by document/entity/time where appropriate; protect holdout labels.
Retrieval vs answer quality	Use Recall@k/MRR/nDCG for retrieval and correctness/groundedness/citation metrics for answers.
Regression testing	Version inputs, run repeatable tests, compare baselines, inspect slices, gate critical regressions.
Synthetic evaluation data	Useful for breadth, but validate against real queries and independent expert labels.
Golden-set gaps	Use production patterns, failure analysis, coverage matrices, and expert review.
Chunking/embedding changes	Run controlled, paired experiments and measure retrieval, answers, latency, and cost.

Scenario-Based Follow-up Questions

Practise explaining your measurement strategy and release decisions aloud.

Scenario 1: Retrieval recall improves, but answer quality gets worse. Why?

The retriever may return more relevant but redundant or noisy chunks, displacing the key evidence or exceeding the model's effective context. Inspect ranking, context ordering, chunk size, citation alignment, and answer metrics—not retrieval recall alone.

Scenario 2: Your evaluation score improves after prompt tuning. How do you know it generalizes?

Check that the prompt was tuned on validation data, not the locked test set. Evaluate on the holdout and important slices, review regression cases, and verify that the test examples were not used in few-shot prompts or hand-crafted rules.

Scenario 3: A synthetic benchmark reports 98% accuracy, but users complain. What do you investigate?

Compare synthetic examples with real query distributions, inspect label quality and judge bias, and sample production failures. Build a human-reviewed evaluation set with realistic ambiguity, source freshness, access-control, and rare-case coverage.

Scenario 4: A new embedding model boosts average recall but increases cost and latency. How do you decide?

Compare gains by risk-critical slice and end-to-end task success, then weigh latency and cost against the product SLO and business value. Test batching, index parameters, or reranking alternatives; canary the candidate and retain a rollback path.

Chapter Summary

A trustworthy RAG evaluation program uses representative queries, independently validated evidence, leakage-resistant splits, separate retrieval and answer metrics, and repeatable regression tests. Synthetic data can expand coverage, but real-world relevance and trusted labels determine whether scores are meaningful.

Practice challenge: You are replacing a chunking strategy and embedding model in an enterprise RAG system. Design an experiment that measures retrieval recall, grounded answer quality, citation correctness, latency, cost, and performance for permission-sensitive queries. Explain how you prevent leakage and decide whether to release.

Search This Blog