What to check when a production RAG answer is wrong

When a production RAG answer is wrong, check five things in this order and stop at the first failure: whether the source was indexed, whether retrieval returned it, whether that passage survived into the prompt, whether the model used the passage it was given, and whether the answer is the wrong shape of a right fact. Swapping the model is not on the list until those five have a written result. A wrong answer that is already live is a pipeline incident. Treat it as one.

The order is the pipeline, and the pipeline is where the documented failures sit. An experience report on three RAG deployments — research, education, and biomedical — catalogued seven failure points. The first four happen before the model has finished its job: missing content, a miss of the top-ranked documents, a passage retrieved and then dropped before the prompt, and a passage that is in the prompt but not extracted (Barnett, Kurniawan, Thudumu, Brannelly & Abdelrazek, "Seven Failure Points When Engineering a Retrieval Augmented Generation System", 3rd International Conference on AI Engineering, Lisbon, April 2024). Their biomedical run indexed 15,000 documents and 1,000 question-and-answer pairs, generated answers with GPT-4, and scored them with OpenAI's evals before the misses were read by hand. Their first takeaway is the one a live incident needs: "validation of a RAG system is only feasible during operation." Robustness, they write, "evolves rather than designed in at the start." RAG is a data problem wearing an AI costume argues why the model is the wrong place to begin. This page is the sequence once you already hold one wrong answer.

Check one: was the source indexed at all? Barnett's first failure point, missing content, is a question that cannot be answered from the documents you hold. The unhappy version does not refuse. It answers from a neighbouring document, because the query was close enough to something that does exist. The test is one phrase. Take a sentence from the document a person says should have won — a policy number, a clause, a product name that is not generic — and search the index for that string, not for the user's question. If the string is absent, stop. Re-embedding, reranking, and a newer model will reproduce the same wrong answer, because none of them can retrieve a passage that was never stored. Two ordinary reasons the string is missing: the file arrived after the last full index, or the extractor dropped it — a scanned page, a sheet inside a workbook, a comment on a ticket — and the chunker never saw the text. Either reason is an ingestion fix, and it is cheaper than a model comparison.

Check two: did retrieval return it? The second failure point is missed top-ranked documents. The answer is in the index and did not place inside the top K the next stage is allowed to see. K is a latency and cost choice, not a property of the question. Run the same query with K raised far enough to see whether the right chunk appears anywhere in the ranked list, and write down the rank if it does. A chunk at rank 40 is a retrieval problem. A chunk that never appears is still check one, or it is a representation problem: the embedding of the chunk does not sit near the embedding of the question. Exact identifiers fail this way often — an error code, a SKU, a clause number — because similarity is not identity. Put a keyword match beside the embedding, and a filter for document status and effective date in front of the ranker, before you touch the generator. The same study found that putting the file name and the chunk number into the retrieved context helped the reader extract the right span. If the right chunk is already inside the top K, leave the retriever alone.

Check three: did the passage survive into the prompt? The third failure point is not in context. The documents came back from the index, and a consolidation step dropped them before generation, usually because the token budget could not hold every hit. The retrieval log looks healthy, which is why this check gets skipped. Open the prompt that was actually sent. If the winning passage is not in that string, the generator never had a chance, and every opinion of the model is an opinion of a different input. Raising K feels like the remedy. Researchers at Amazon AWS AI, with collaborators at Shanghai Jiao Tong University and Westlake University, evaluated 8 RAG systems across 10 domains and then varied how much context a system received. Moving from 5 chunks to 20 raised claim recall from 61.5 to 77.6 — more of the facts required for a correct answer were now available — while faithfulness moved from 88.1 to 92.2, noise sensitivity from 34.0 to 35.4, and end-to-end F1 only from 51.7 to 53.4 (Ru, Qiu, Hu, Zhang, Shi et al., "RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation", arXiv, 2024). Coverage jumped. The answer barely moved, because a generator tends to trust a relevant chunk as a whole: the extra facts arrive tied to extra noise. Test the larger window against the answer. Do not ship it because the log got longer.

Check four: did the model use the passage it was given? The fourth failure point is not extracted. The answer sits in the context, and the model did not pull it out, typically because the context is noisy or contradicts itself. This is the first check that is allowed to blame the generator, and only this one. The split that makes the blame legitimate is faithfulness: of the claims in the answer, what share can be inferred from the context that was actually sent. Es, James, Espinosa-Anke and Schockaert, at Exploding Gradients and Cardiff University, define it that way. They split the answer into statements and ask whether each statement follows from the context ("Ragas: Automated Evaluation of Retrieval Augmented Generation", arXiv, 2023). On WikiEval — 50 Wikipedia pages, questions about events after the start of 2022 — two annotators agreed on faithfulness in around 95% of cases, and the Ragas faithfulness score matched the human preference on 95% of pairwise comparisons. That 0.95 is their metric on their set. What you copy is the split. If the right passage is in the prompt and the claim is not supported by it, you have a generation defect: an instruction that requires a citation, a refusal when the context does not contain the fact, or a model that is worse at extraction. If the right passage is not in the prompt, a high faithfulness score is a trap. The model was loyal to the wrong pages. The same framework separates answer relevance — whether the reply addresses the question — from context relevance, whether the retrieved context is focused. One thumbs-up on the final answer collapses the three and sends the next week to the wrong module.

Check five: is the answer the wrong shape of a right fact? The last three failure points in the Deakin catalogue are not hallucinations. Wrong format: the user asked for a table or a list and received a paragraph. Incorrect specificity: the fact is true and too general, or true and narrower than the decision in front of the user. Incomplete: part of the answer is in the context and missing from the reply, as when "what are the key points in documents A, B and C?" is answered from A alone. A user will still call that wrong, and a metric that only checks whether each sentence is grounded will still call it right. The test is a comparison against the shape you promised, written down before you look at the output. If checks one through four passed and the shape is wrong, the fix is the instruction and the example, not the index. The same experience report records a lesson that cuts against a slogan: on their teaching assistant, a larger context, 8K against 4K, produced more accurate responses, contrary to earlier measurements on GPT-3.5. Window size is a variable you measure on your own questions.

Five checks need five artefacts from the same incident: the user's question, the chunk identifiers retrieval returned and their ranks, the exact prompt that was sent, the answer the user saw, and the passage a person says should have won. Without those five you have a complaint, and a complaint is not enough to decide which module to change. Changing two modules in one release is how a team finishes the week unable to say which change mattered. That record is also how robustness evolves. The Deakin report's second finding is that it is not designed in on day one. It accumulates, one incident at a time, somewhere the next miss will actually be read.

There is a version of this where an engineering engagement is the wrong purchase. If check one fails because the document was never written — the rule lives in a person's head, and two people cannot produce the same current file — the work is curation and an owner. A retriever cannot mint the missing source, and indexing the confusion only serves it faster. If you do not yet have one wrong answer with those five artefacts, you do not have a production failure. You have a system that has not been observed. That is a two-week AI readiness assessment: systems, data, and security mapped before a build is scoped, findings yours whether or not anyone builds afterwards. A demo that has to impress a room once does not need an index that stays current. Teams that have not started a pilot, and projects whose blocker is an undefined goal, belong there too. Name the goal, or finish the observation, before you buy a repair.

When the answer is already live, someone has acted on it, and checks one through four point at a pipeline you cannot see from the chat window, the remaining work is specific. An audit of what is salvageable. An evaluation harness with a threshold. Guardrails before anything acts on its own. One baseline number the work has to move — tickets deflected, hours saved, cycle time — agreed before the rebuild, and reported against, so the question of whether it is working has a figure rather than a mood. That is what an AI pilot failed to reach production engagement is for: finished in 8–12 weeks, with the audit in the first week or two, so a pilot that cannot be saved is called unsalvageable before the larger invoice starts. The metric is agreed up front. If the engagement does not move it, the build is on us and you do not pay for it. The five checks are that audit's first afternoon. The harness that stops the same failure shipping again is AI evaluation and observability. Retrieval treated as its own deliverable, rather than a library call beside a model, is the core of generative AI development services.

Haystack, an AI data-intelligence platform, is the shape of check one when it has to keep being true. Business and customer data lived in silos. Teams could not find files, insights, or metrics. Fragmented tools broke the workflow. The existing setup failed under high data loads, and nothing in it surfaced what the data already held. The platform we built is one search layer across documents, data lakes, and repositories: real-time indexing, so a new point is searchable when it lands rather than at the next batch, and Golang services on the indexing path, because that path is where load had already broken the previous system. An index that updates when the corpus updates is the only version of check one that is still true a week later. A nightly batch that was fine on a small corpus is how check one starts failing without anyone merging a bad prompt.

Run it on the last wrong answer before you open a comparison of models. Search the index for a sentence from the document that should have won. Record the rank of that chunk, or that it never appeared. Confirm the chunk is inside the prompt that was sent, not merely in the retrieval log. Mark each claim as supported by that prompt or not. Compare the shape of the answer with the shape you promised. The first failure is the one you fix. Bring the five artefacts to that audit. Without them the first day reconstructs an incident you already lived through. Whether the feature should still be serving the next customer while you run the checks is a separate decision: when to turn a live AI feature off.