Evaluation must follow the system layers
An enterprise RAG system is a pipeline. A single score cannot explain whether a failure came from document processing, retrieval, reranking, generation, permissions, or stale data. A useful evaluation programme separates those layers and reconnects them to the user task.
Begin with representative questions
Collect real question patterns from intended users. Include straightforward lookups, comparisons, multi-document questions, ambiguous requests, questions with no answer, and requests the user is not authorised to resolve.
For each question, record the expected evidence and the qualities of an acceptable response. Perfect wording is less important than correct support.
Evaluate retrieval first
Measure whether the right passages appear in the candidate set and whether irrelevant passages dominate the context. Review performance by document type, query type, department, and access scope. Aggregate results can hide important weak segments.
Useful retrieval checks include:
- evidence recall;
- ranking quality;
- metadata-filter correctness;
- citation granularity;
- sensitivity to document updates.
Evaluate the generated response
Review factual support, completeness, citation accuracy, instruction adherence, uncertainty, and usefulness. Automated model-based evaluators can help with scale, but they should be calibrated against human review and treated as measurement tools—not ground truth.
Measure operational quality
Users experience more than answer text. Track end-to-end latency, retrieval latency, model latency, token usage, error rates, timeout behaviour, and cost by request type. Security and permissions need independent tests.
Create a regression loop
Every material failure should become a candidate test case. Run the suite when prompts, chunking, embeddings, retrievers, rerankers, models, or data connectors change.
The purpose of evaluation is not to produce an impressive dashboard. It is to create enough evidence to make safe engineering decisions, detect regressions, and understand where the system should improve next.