RAG troubleshooting: find where the evidence was lost
Trace a wrong RAG answer from its source through indexing, retrieval and context assembly. Separate missing evidence from a model misreading good evidence.
Debug a RAG answer by following its evidence: the source document, indexed text, retrieved passages, model input and final claims. Fix the stage that lost or distorted the needed information. Changing the prompt cannot retrieve a fact that never reached the searchable data.
The new manual allows CSV export with a header-only file when no rows match. The assistant says export fails. First check which manual it found, whether it received the empty-result exception and whether its answer kept that exception. Those are three different possible failures.
A wrong answer does not tell you whether the search failed or the model ignored a correct result. Save the intermediate evidence before trying a larger model, a longer prompt or a different database. Otherwise, a change can hide the symptom without explaining its cause.
Keep the failed question and its actual inputs#
Chapter 6 of Albada’s book separates retrieval from generation: first find relevant information, then supply it to a model producing the response. That distinction is the starting point for this diagnostic. Retrieving extra information does not, by itself, establish that it is current, complete or correctly used.
Our illustrative product has two manuals. Version 2.2 rejects CSV export when no rows match. Version 2.3 writes a header-only file. A user asks what the version 2.3 web application does, but the assistant answers with the old behaviour.
Keep that question, the product variant, authorised data scope, source revisions, search settings, actual returned passages, assembled model input and answer. Apply appropriate access and retention controls to diagnostic records; debugging is not a reason to copy every private document into an unrestricted log.
Check that the right source became searchable#
Start at the document itself. Does the version 2.3 manual actually contain the empty-result rule? If it does not, the next step is to obtain a trustworthy source or report the gap. A better search ranking cannot supply missing documentation.
If the rule exists, inspect the extracted text. A parser may have omitted a table footnote, or the ingestion job may still contain the previous revision. Compare the indexed passage with the actual manual rather than trusting a successful ingestion status alone.
Preserve product, version and access metadata when dividing a document into passages. A passage saying “an empty result produces a header-only file” becomes ambiguous when detached from the version and product it describes. In this example, keeping the heading and exception together matters more than choosing a universal passage length.
Check the user’s allowed corpus too. If they cannot access the needed source, report that limitation according to the product’s policy. Removing access filters until a test passes is not a retrieval fix.
Try it: locate the lost exception
- Inspect the authorised manual.
- Obtain a verified rule or report the gap.
- Do not invent it through prompting.
Search tuning cannot retrieve a fact absent from the source.
Invented manuals and trace; no retrieval system is called.
Inspect what the search returned#
Now compare the returned passages with the evidence the question needs. Did the version filter select 2.3? Did the relevant exception rank below the returned set? Did a desktop guide displace the web guide because both discuss CSV export?
Test queries that preserve exact identifiers as well as natural-language descriptions. Azure AI Search’s hybrid-search documentation explains how its implementation combines keyword and vector results. It also identifies product codes, names and specialised terms as cases where exact matching can be valuable. That is not a claim that one hybrid configuration is best for every corpus.
For the example, compare the same failed question using the existing search and a version-filtered alternative. Record whether each returns the passage containing the exception. If increasing the result count adds the needed passage, also inspect what irrelevant material it introduces. More retrieved text is not automatically a better final input.
A similarity score ranks candidates under a retrieval method. It is not a certificate that a passage is current or that its claim is true. Source scope and version remain separate checks.
Check what reached the model#
Suppose the search returned the right exception, but a summarising step retained only “CSV export is supported”. The retrieval succeeded; the evidence was lost afterwards. Inspect the actual next model input, not just the search log.
The same problem can occur when the application truncates results or carries forward an old summary saying that empty exports fail. Keep source references with condensed findings and mark superseded assumptions. Context engineering examines exactly this difference between available information and information the model received.
Retrieved content must remain evidence, not an instruction with permission to change the task. A manual can describe an export operation without authorising the assistant to export anyone’s data. Keep the boundary between retrieved text and instructions intact while debugging.
Separate a missing fact from a misread fact#
Run a diagnostic with the correct version 2.3 passage supplied directly. If the model still says export fails, investigate interpretation, conflicting instructions or output construction. If it answers correctly, inspect the normal retrieval and input-assembly path. One successful diagnostic is evidence about that run, not proof that the generation stage never fails.
Check the final claim and its citation separately. The answer should preserve the condition: no matching rows produces a header-only file in the specified product version. A link to the right manual does not rescue text that reverses its exception.
When sources genuinely disagree, keep the disagreement visible until their scope or authority resolves it. Do not silently choose whichever passage appeared first. Nor should the system turn an unsuccessful search into a confident claim that the feature does not exist.
Add graph retrieval for a question that needs it#
The chapter also introduces knowledge graphs. Microsoft’s GraphRAG builds entities, relationships and community summaries, then uses those structures for different kinds of queries. Its documentation distinguishes broad corpus questions from queries about particular entities.
That offers a different candidate design for questions such as which components share a dependency across many documents. Our single-manual exception does not establish a need for that extra processing. A stale source can produce a stale graph too, and extracted relationships still need to be checked.
Compare the additional representation on questions that need relationships or corpus-wide synthesis. Keep a simpler retrieval baseline and account for building and updating the derived data. A graph is a design choice to test, not the compulsory next stage after vector search.
Keep the failure as a repeatable check#
Save both version questions: the old version should still return its old rule, while the new version should retain its exception. Include the wrong product variant and a source the caller cannot access. Check retrieved evidence and final claims independently. This is a proposed fixture, not a measured benchmark.
Add the cases to a versioned evaluation set. If the fix updated the manual or index, describe it as that. Memory and model training are different mechanisms, and changing the wrong one can leave the original failure untouched.
Questions people also ask
3 QUESTIONSDoes RAG require a vector database?
No. Retrieval can use keyword search, structured lookups, vector similarity or a combination. Choose a retrieval method for the information and questions involved. A vector store is one implementation choice, not the definition of RAG.
Does retrieving the correct document guarantee a correct answer?
No. The relevant exception may be absent from the returned passage, removed while assembling the input or misread during generation. A correct document identifier is useful evidence, but it does not establish that the answer preserves what the document says.
Should I fine-tune the model when a manual changes?
First identify the failure. A changed manual usually requires making its current content available to retrieval and checking the resulting answer. Updating documents or the search index is not itself model training. Fine-tuning addresses a different part of the system and needs a separate evaluation.