Where the supporting passage sat

An answer is being served, and the passage that is supposed to support it was inside the prompt, or you cannot tell. Tonight, in this order. Open the prompt that was sent for that answer. Find the passage. Write where it sat: its index among the passages, or how far it sat from the start and from the end. If you cannot see the prompt, or you cannot find the passage in it, stop serving the sentence as grounded in that passage. Do not ask the model which paragraph it used. Asking the model replaces a position you have not read with a claim you have not read.

This is the position of a passage that was already in the prompt. It is not whether the page at a link supports the sentence. Open the page the answer cited is that check. A page can support a sentence that the model did not use, because the passage sat where the model does not reliably look. It is not the age of the passage. When the retrieved page is old compares the passage's date with the live page. A current passage can sit in the middle and still be skipped. It is not two sources that disagree. When two sources disagree is that fight. One passage, in one prompt, can be ignored without a second source. It is not two completions of one question. When one question gets two answers stops until one sentence can be shown twice. Here the question is where the support sat, not whether the next draw matches. The five checks that stop at the first broken stage of a pipeline are what to check when a production RAG answer is wrong. This page starts after the passage was present in the prompt.

The one-paragraph version of the position effect is already on RAG is a data problem: a U-shaped curve, highest at the edges, and a drop of more than 20% for one model when the passage sat in the middle. That paragraph is the reason to care. It is not the position of the prompt you sent tonight. The paper behind it is Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, and Liang, "Lost in the Middle: How Language Models Use Long Contexts," Transactions of the Association for Computational Linguistics, 23 February 2024, received 1 August 2023, revised 1 October 2023. The TACL page names Stanford University for Nelson F. Liu, and Samaya AI in the UK for Michele Bevilacqua and Fabio Petroni (Liu et al., Lost in the Middle). Do not promote the data-problem article's affiliation sentence into a second census of the author list.

The multi-document task is built so the position is the only thing that moves. They take NaturalQuestions-Open and keep the 2,655 queries whose annotated long answer is a paragraph, rather than a list or a table. Documents are Wikipedia passages of at most 100 tokens. Exactly one document contains the annotated answer. The other documents are distractors: the most relevant Wikipedia chunks, from Contriever fine-tuned on MS-MARCO, that do not contain any of the annotated answers, presented in decreasing relevance. Accuracy means any annotated correct answer appears in the output. They use input contexts of 10, 20, and 30 documents. In the closed-book setting the model gets no documents. In the oracle setting it gets only the document that contains the answer. For GPT-3.5-Turbo, closed-book accuracy is 56.1% and oracle accuracy is 88.3%. For GPT-3.5-Turbo with the 16K context window, the same two cells are 56.0% and 88.6%. 56.1 is not 88.3. 56.1 is not 56.0. 88.3 is not 88.6. Closed-book is parametric memory. Oracle is one passage and no distractors. Neither number is a position.

The position result is a shape, and the prose gives two quantities rather than a full table of cells. Performance is often highest when the relevant document is at the very beginning or the very end, and it degrades when the model must use the middle. They call the ends primacy and recency. GPT-3.5-Turbo's multi-document accuracy "can drop by more than 20%." In the worst case, performance in the 20-document and 30-document settings is lower than the closed-book 56.1%. "More than 20%" is not an exact drop, and the paper's prose does not print the worst-case percentage as its own cell. Do not invent one. "Lower than 56.1%" means the model did worse with the answer present, in the middle of a stack, than with no documents at all. It does not mean the middle is 56.1, and it does not mean every position in a 20-document prompt is below the closed-book line. The 10-document and 20-document settings fit inside both the 4K and the 16K windows, and performance between GPT-3.5-Turbo and the 16K variant is nearly identical, nearly superimposed. A longer context window, on that comparison, is not better use of the middle. They used the June 2023 OpenAI snapshots, the 0613 versions. A current model name is not that snapshot.

A second task in the same paper is easy to borrow and easy to mix. Key-value retrieval gives the model a JSON object of unique random identifiers and asks for the value of one key. Some models are perfect. The paper names Claude-1.3 and Claude-1.3 with the 100K window as perfect on this synthetic task. Other models show the same U-shape and struggle when the pair sits in the middle. Perfect recall of an identifier is not the 88.3% oracle on the question-answering task, and it is not the "more than 20%" drop. When they repeat the key-value task with the query placed before the JSON as well as after it, which they call query-aware contextualization, all of the models they discuss reach near-perfect accuracy at 75, 140, and 300 pairs. GPT-3.5-Turbo (16K) is perfect at 300 pairs under that arrangement. That is a different prompt. It is not evidence that the multi-document questions became insensitive to position. Instruction tuning is a third cut. MPT-30B and MPT-30B-Instruct both show the U-shape. Tuning reduces the worst-case gap between the best and worst positions from nearly 10% on the base model to around 4% on the instruct model. Nearly 10 and around 4 are their phrases for that pair of models. They are not the "more than 20%." On Llama-2, the 7B models in their appendix are only recency-biased, while the 13B and 70B models show the U-shape. Additional supervised fine-tuning and human-feedback training slightly reduce the bias at 13B and minimally change it at 70B. They do not print a percentage for that Llama pair. Do not supply one.

The open-domain case study is a fourth number, about how many documents you add, not about where one document sat. A retriever returns the top passages from Wikipedia for NaturalQuestions-Open queries whose long answer is a paragraph. Reader accuracy saturates before retriever recall does. Using more than 20 retrieved documents "only marginally improves" reader performance, about 1.5% for GPT-3.5-Turbo and about 1% for Claude-1.3, while the prompt gets longer. About 1.5 is not about 1. Neither is the "more than 20%" positional drop, and neither is the closed-book 56.1. Adding documents is not the same experiment as moving the one relevant document. A note in the paper says a small number of distractors may contain a reasonable answer because NaturalQuestions is ambiguous. On a subset they treated as unambiguous, the conclusions were similar. Similar is not a second table. The queries are historical Google queries, answered from Wikipedia, in English. They are not your tickets, and they are not a probability for the next prompt your product sends. GPT-4 was run on a subset, with similar trends and higher absolute accuracy. The note says a full run of the multi-document and key-value experiments would have cost upwards of $6,000. That is their cost comment. It is not an accuracy.

A later benchmark measures something people confuse with position: whether a claimed context length survives harder tasks. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg, at NVIDIA, introduce RULER, arXiv:2404.06654. They evaluate 17 models on 13 tasks, with 500 examples at each length in the series 4K, 8K, 16K, 32K, 64K, and 128K. Decoding is greedy. Each score in the main table is the average of the 13 tasks, not the accuracy of one task. The threshold they grade against is Llama-2-7B at 4K, 85.6%. Effective length is the longest tested length that still exceeds that threshold. Every model they include claims a context of 32K tokens or more. Their sentence is that only half maintain satisfactory performance at 32K, and that almost all fall below the threshold before the length they claim (Hsieh et al., RULER). "Only half" is their summary. Do not recount their table into a different fraction. NVIDIA built the benchmark. It is not a neutral census of the model your vendor named, and the authors say the synthetic tasks should not be preferred to realistic evaluations.

Two rows are enough to see a claim separate from a cell, and a cell separate from an average. GPT-4 is listed with a claimed length of 128K and an effective length of 64K. The task-average scores are 96.6 at 4K, 96.3 at 8K, 95.2 at 16K, 93.2 at 32K, 87.0 at 64K, and 81.2 at 128K. The unweighted average across those lengths is 91.6. 87.0 is above the 85.6 threshold, which is why 64K still counts as effective. 81.2 is below it, which is why the claimed 128K does not. 81.2 is not 87.0, and 91.6 is not the 128K cell. Gemini-1.5-Pro is listed with a claimed length of 1M and an effective length above 128K. Its 128K score is 94.4. They did not print a score at 1M. Above 128K is not 1M. Llama-3.1 at 70B is also claimed at 128K and effective at 64K, and its 128K score is 66.6. 66.6 is not GPT-4's 81.2. A different model, Yi, in a different analysis, copies the one-shot example for over 80% of its outputs on one aggregation task at 128K, and the copying is absent on short sequences. Over 80 is not 81.2, and it is not Liu's "more than 20%." A drop of about 40 points for Yi at 256K, in an extreme distractor setting, is a third experiment. 256K is not a length in the main table.

RULER's own limitation is the reason it cannot replace the position proof. The benchmark reports one number per input length. It does not report how deep in the prompt the relevant item sat. The authors say that depth is exactly what makes the lost-in-the-middle result visible, and that position control was not in the benchmark yet. A model that passes a needle placed where the test happens to put it, or a model whose average across 13 tasks clears 85.6 at 32K, has not shown you the middle of your prompt. Nearly perfect accuracy on the plain needle test, which they report as a contrast, is a simpler retrieval. It is not the 13-task average, and it is not Liu's question-answering curve. Treat a vendor's "128K context" the way you treat GPT-4's claimed 128K next to its effective 64K: write both numbers down, and do not let the claim stand in for the cell.

Do the proof on one answer you are still willing to show a customer. Export the prompt that was sent, not a reconstruction from memory and not a new call. Mark the passage that is supposed to support the sentence. Record the index and the two distances. If the passage is absent, you are not on this page. You are on the checklist's earlier failure, the one where retrieval did not put the passage in the prompt, and you stop for that reason instead. If the passage is present, you still do not ask the model to confirm it used the middle. The Liu result is that the same passage, moved, changes the accuracy, including to below the no-document line. A model agreeing that it "used document 7" is another completion. The record you keep so a later complaint can be rebuilt is what to keep after a wrong AI answer, if you have it. A trace that stores the prompt is how you read the position. A trace that stores only the final sentence cannot answer this page. The suite that should fail a release when the position effect shows up on your own questions is the evaluation suite. A suite that only checks the final sentence, and never moves the supporting passage, will stay green while the middle stays unread.

What a grounded answer looks like, as a product, is narrower than these papers. The support agent we shipped for a DTC brand answers order status, returns, and product questions from order data, policies, and the catalog, and it can look up an order, start a return, or change an address behind a permission check, with escalation to a person. It is not a position study. Nothing in that description is a headcount, a duration, or a result figure. If the sentence a customer sees is supposed to come from one policy passage, the prompt that carried that passage is the object you open. The papers' percentages will not appear on that screen, and they should not be spoken as if they had. A session you end for one person, and a secret still sitting in a commit, are different stops: end the session they still hold and name the secret the history holds. Do not record either one as a passage index.

There is a version of this where hiring the position check is the wrong purchase. Nothing is in production, so there is no prompt that was sent. The passage was never retrieved, because the index is empty or the retriever returned nothing. That is a retrieval failure, and this page will not fix it by rearranging a prompt you do not have. The two sentences a customer compared were two different questions. A customer has already seen another customer's data. The wrong value was a write the agent already made. No metric was ever named, so there is nothing for a rescue to miss. The audit that says whether any of that is true is the first one or two weeks of a rescue. The build, if there is one, is scoped after that audit, runs 8–12 weeks, and is unpaid if the metric you agreed is missed. Rollout stays behind a human-review gate.

When the prompt is real and nobody can say where the passage sat, the missing object is specific. The prompt, stored, and a line for the index. An AI pilot failed to reach production for this reason when the demo was a short prompt and production is a stack of passages nobody re-reads. AI evaluation and observability is the practice of keeping that line, not a leaderboard of context lengths. If the first slice is one path, an AI integration sprint can require the position to be recorded on the answers the sprint is allowed to send. If you do not yet know whether the prompts you send can even be opened, a two-week AI readiness assessment maps that. The findings are yours. They are not Liu's 56.1, and they are not RULER's 85.6.

Before you send the next sentence as grounded, open the prompt and write the position. If you cannot, stop. Do not ask the model which paragraph it used, and do not let a claimed context length stand in for the middle of the prompt you sent. The edges are where the paper's curve is highest. The model and the length of the vector that retrieved the passage are a different record: which embedding did that passage use?. A position is not a dimension. The line you write down is the page you can defend.