Three Retrieval Failures That Are Actually Data Preparation Failures

When a retrieval augmented generation (RAG) system gives a wrong answer, the first thing most teams look at is the prompt. It is the most visible lever, and rewording it produces a quick, satisfying win on the one query that failed. But in the majority of enterprise RAG deployments, the prompt is not where the problem lives, and neither, in most cases, is the retriever itself. Retrieval only ever surfaces what the corpus makes available to it.

When the answer is wrong, the fault can also sit one layer earlier: in how documents were structured and prepared before retrieval ever ran. These failures are uniquely dangerous because they are invisible. The user sees a fluent, confident response and has no way to know it was built on an irrelevant, incomplete, or outdated source.

Retrieval still deserves to be evaluated as its own component, in isolation, with its own metrics, rather than folded into a vague sense that the model seems off. Three failure modes account for most of what goes wrong, and each can be tested directly at the retrieval layer: relevance, coverage, and noise. But testing there only tells you where the failure surfaces, not where it originates. All three are frequently manufactured upstream, in how documents were turned into a corpus in the first place.

1. Relevance: Are the retrieved passages actually about the question?

Relevance is the most fundamental retrieval metric and the most common place RAG systems fail. Consider a customer service inquiry into whether a refund applies to a specific product. The retriever returns a returns policy page, topically adjacent, with a high similarity score. But it is the general consumer policy, not the enterprise contract addendum that actually governs the specific account. Handed plausible material, the model does what models do and synthesizes a confident answer from the wrong document. This is not an edge case. A large share (~70%) of the passages a RAG system retrieves do not contain the information needed to answer the query at all. When that happens, hallucination is the likeliest outcome.

Here is where ingestion quality becomes decisive. If that governing addendum lived in a scanned PDF whose table structure was flattened into an undifferentiated block of text during processing, the passage that mattered may never have been embedded as a coherent, retrievable unit. The retriever did not fail to find a relevant document. A relevant document was never cleanly represented. This is what "garbage in, garbage out" means for RAG, and it is why document processing quality, not just embedding choice, sits at the front of the relevance problem.

2. Coverage: Do the retrieved passages contain the whole answer?

Even when retrieval surfaces relevant material, it may not surface all of it. Coverage failures are subtler than relevance failures and more dangerous. Suppose a pricing query returns the document confirming a customer's tier but misses the exhibit describing the exception that applies to their contract. The result is an answer that is partially correct, and partial correctness is far harder to catch than an obvious error because the answer checks out.

Coverage failures are very often chunking failures, and chunking is an ingestion decision. Recent RAG research keeps landing on the same diagnosis. A system can retrieve the correct document family yet still fail because the passage that carries the answer is missing, buried or split away from its context during chunking. For instance, a table row may get detached from its column header, or an exception clause can be severed from the rule it modifies. These are coverage gaps created before a single vector is computed.

This is exactly the case for processing that preserves document structure. ABBYY Document AI is built around that principle, recognizing tables, preserving headers and footnotes, and keeping related elements together, so the passage that carries the answer is not torn apart during ingestion.

3. Noise: How much irrelevant material rides along?

The third failure is the mirror image of the second. Instead of missing the answer, the system buries it. Teams often respond to weak relevance by widening the net, retrieving twenty passages instead of five, which inadvertently hands the model a triagяe job it was never designed to do. The evidence here is unusually clear. Simply reordering the same set of retrieved passages can change the model's answer. Performance is strongest when the key passage sits at the very beginning or end of the context and drops sharply when it lands in the middle. It's important to note that not all noise is equal. Random passages do little harm, but "distracting" passages, which are irrelevant yet semantically related, actively degrade answer quality.

Noise also traces back to the corpus. When ingestion classifies and cleanly bounds documents instead of dumping raw text into the index, the candidate pool is smaller and sharper, with fewer plausible but wrong distractors competing with the truth. Turning documents into clean, well bounded, structured data, which is ABBYY's thesis about making enterprise content fluent to LLMs, is as much a noise reduction strategy as a retrieval one.

Testing for all three

None of this can be measured without ground truth. For each query in your evaluation set, you need to know which documents or passages actually contain the answer. Building that takes real effort, but without it you cannot tell a retrieval problem from a prompt problem when an answer comes back wrong. With ground truth in hand, the classic information retrieval metrics do the job, including precision at k for relevance, recall for coverage, and mean reciprocal rank for how well the best passage surfaces. They predate LLMs by decades and remain the right tools.

Here is the pattern to internalize: When retrieval is good, even a mediocre prompt produces acceptable answers. When retrieval is poor, even an expertly crafted prompt cannot save you. For any system whose prompts are already tuned, the next dollar is almost always better spent on retrieval, and specifically on the document quality that drives retrieval. Test the retriever as its own component. Then trace every failure back to where the corpus was made.

That is the difference between AI you hope works and AI you know works.