A useful demo is not yet a reliable retrieval system
A RAG prototype often succeeds on a small, familiar set of documents. Production introduces a different problem: changing source data, ambiguous questions, access controls, retrieval noise, long documents, latency constraints, and users who ask questions the original builder never anticipated.
The failure is rarely “the model is not smart enough.” More often, the system has no disciplined way to decide what to retrieve, how much evidence is sufficient, or when it should decline to answer.
Common failure modes
The ingestion pipeline loses important structure
Tables, headings, footnotes, page boundaries, and document versions carry meaning. Flattening every file into arbitrary chunks can make the most relevant evidence difficult to recover.
Retrieval is treated as one setting
Different questions benefit from different strategies. Exact identifiers may need keyword search; conceptual questions may need semantic search; complex questions may need query decomposition or metadata filters. One top-k vector query is not a complete retrieval architecture.
Generated answers are evaluated without retrieval
An answer can sound good even when the right evidence was never retrieved. Evaluate the pipeline in layers: ingestion quality, retrieval recall and precision, reranking, answer groundedness, citation correctness, latency, and cost.
The system cannot fail safely
Production systems need explicit behaviour for missing context, conflicting sources, stale documents, and unsupported requests. “I do not have enough evidence” is a feature when the alternative is an authoritative-sounding invention.
A more dependable path
Start with representative questions and expected evidence. Build a small evaluation set before tuning prompts. Instrument every retrieval and generation step. Review failures by category, then improve the narrowest layer responsible.
Production RAG is an information system with a language model inside it—not a language model with a document folder attached.
The goal is not to maximise how often the system answers. It is to increase how often useful answers are supported by the right evidence, for the right user, within acceptable operational limits.