RAG is a data problem wearing an AI costume

Ask why a RAG system works in the demo and falls over in production, and the honest answer is almost never the model. It's the retrieval pipeline underneath it: how a document gets chunked, what metadata survives that split, and whether anyone is measuring "did we fetch the right passage" as its own number instead of grading the final answer and hoping retrieval gets credit it didn't earn. Teams routinely spend weeks comparing one model against another and minutes on the pipeline that decides what either model actually sees.

An engineering study of three production RAG deployments — in research, education, and biomedical domains — found the failures clustering into seven repeatable points: content that was never indexed in the first place, a search step that misses the right chunk even though it exists in the index, a context window too small to hold every relevant passage at once, and formatting or completeness failures on the generation side (Barnett, Kurniawan, Thudumu, Brannelly & Abdelrazek, "Seven Failure Points When Engineering a Retrieval Augmented Generation System", IEEE/ACM CAIN, January 2024). None of the seven live inside the model. They live in the pipeline a team built before the model ever saw a prompt — which is exactly why swapping models rarely fixes a RAG system that's failing in production.

Chunking is where most of that damage starts, and it's usually invisible, because a badly chunked document still returns a result — just not the right one. Splitting by a fixed token count cuts straight through headings, table rows, and the boundary between a question and its answer, so the embedding for a chunk ends up representing half of one idea and half of another, and similarity search matches on neither cleanly. Structure-aware chunking — splitting at section and table boundaries instead of every N tokens, and carrying the parent heading along with each chunk so it stays interpretable out of context — fixes a class of retrieval misses that no amount of prompt engineering on the generation side can touch, because by the time the model sees the chunk, the damage is already done.

Even a pipeline that retrieves the right passage isn't safe once it reaches the model's context window. Researchers at Stanford tested six model families — including GPT-3.5, GPT-4, and Claude — on tasks where the correct answer was buried at different positions inside a stack of retrieved documents, and found a consistent U-shaped performance curve: accuracy was highest when the relevant passage sat first or last in the context, and dropped by roughly 20 to 30 percentage points when the same passage sat in the middle instead (Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", Transactions of the Association for Computational Linguistics, 2024). A retrieval pipeline can rank the right chunk in position eight of twelve and still lose most of the value of having found it at all — the argument for reranking the retrieved set so the strongest passage lands at the edges of the context, not just for retrieving it in the first place.

The failure mode that shows up most often in practice is quieter than a hallucination and harder to notice: a support or research assistant answers confidently from a document that was correct six months ago and has since been superseded, because the retrieval index has no field for effective date or document status and ranks purely on semantic similarity. The stale passage reads as the best match because nothing in the pipeline ever told it otherwise. Fixing this is not a prompting problem — it's adding a metadata field and a filter before the ranking step ever runs, which is a data-engineering task, not a model one.

The practical fix is to stop grading retrieval by proxy. Build a golden set of real questions paired with the passage that should answer each one, and score recall@k against that set on its own, independent of whatever the model does with the passage afterward. A system that retrieves the right passage 95% of the time and generates well from it is a different problem, with a different fix, than one that retrieves the wrong passage 40% of the time and generates flawlessly from bad material. Conflating the two into a single end-to-end accuracy number is the single most common reason teams spend weeks debugging the wrong half of the system. Pure vector similarity also isn't the whole answer on its own: a hybrid search that combines keyword matching (BM25) with embedding similarity catches exact-term queries — a product SKU, an error code, a client's own name for something — that a pure embedding search routinely misses, because semantic similarity is measuring meaning, not identity. A cross-encoder reranking pass over the top candidates from either method, run before generation, is usually cheaper than it sounds and catches the cases where the fast first-pass ranker got the order wrong but had the right passage somewhere in its shortlist.

This is infrastructure work before it's an AI feature, and it has to behave like infrastructure: holding up as the corpus grows, not just in a fifty-document pilot. Haystack, an AI data-intelligence platform we built to unify search across documents, data lakes, and repositories, was designed around that constraint directly — real-time indexing so newly ingested data is searchable immediately, and correlation across structured and unstructured sources rather than a single flat index, because the version that works cleanly at fifty documents and the version that still answers correctly under real production load are not the same system.

Retrieval quality is what decides whether an LLM application gets trusted, and it's squarely a data-engineering discipline: chunking that respects document structure, metadata that lets you filter before you rank, indexing that holds up as the corpus grows, and evaluation that scores retrieval on its own terms. That's the core of what sits inside generative AI development services — the retrieval layer treated as its own deliverable, not a library call bolted onto a model. If the harder problem is the corpus itself — scattered across silos, inconsistently tagged, or too large to reindex cheaply — that's a question for AI data infrastructure services before it's a question about which model to call.