When the retrieved page is old

An answer is live, and the passage it used is older than the page a customer would be shown today, or the passage has no date. Tonight, in this order. Read the date on the passage that was retrieved. Read the date on the live page, or on the record that is supposed to win. If you cannot tell which is newer, stop serving that sentence as current. Write the pair down — the sentence, both dates or the fact that a date is missing — and do not ask the model which copy wins. Asking the model replaces an age you have not checked with a preference you have not checked.

This is the age of one passage. It is not a fight between two passages that could both be current. When two sources disagree is that fight: write both, or stop, and do not let the model pick a rule the sources do not share. Two pages can disagree and both be new. One page can be old and still be the only page you retrieved. Age is not contradiction, and contradiction is not age. It is not whether the page supports the sentence. Open the page the answer cited is that check. A page can fully support a sentence that stopped being true on Monday. Support is not currency. It is not the five checks that decide which stage of a pipeline failed. What to check when a production RAG answer is wrong stops at the first failure, and it already says to put an effective date in front of the ranker. This page is what you do when that filter was missing and a customer has already been shown the old sentence. The quiet version of the same failure — a document that was right six months ago and is now the best semantic match — is described in RAG is a data problem. The description is not the proof. The proof is the two dates.

It is also not the calendar on which a model is retired. Model drift is that calendar: a model identifier stops accepting requests on a published date. The weights can be current and the index can be old. Swapping the model before you have the dates replaces an old page with a newer voice reading the same old page.

The measurements that separate a fresh fact from a fluent one come from a benchmark built because models are trained once. Vu, Iyyer, Wang, Constant, Wei, Wei, Tar, Sung, Zhou, Le, and Luong, at Google, the University of Massachusetts Amherst, and OpenAI, introduce FreshQA: 600 questions, divided into never-changing, slow-changing (the answer typically changes over several years), fast-changing (within a year or less), and false-premise. The test set is 125 questions in each of the four types, 500 in total. A development set of 25 per type, 100 questions, "is reserved for future studies and not used in this paper." Fifteen further examples were held out for demonstrations, and the remaining data were discarded. An accuracy printed for this paper is not a percentage of 600 (Vu et al., FreshLLMs, arXiv:2310.03214, Findings of ACL 2024).

They score each answer twice, and the two scores are not synonyms. Relaxed asks whether the primary answer is correct, and it accepts hallucinated or outdated information that does not significantly affect that primary answer. Strict asks whether all of the claims are factual and up to date. Outdated information fails Strict even when Relaxed would pass. Two authors scored 100 answers in both modes and agreed 99% of the time on Relaxed and 96% on Strict. An automatic judge, FreshEval, matched humans at an average of 96.5% on Relaxed and 96% on Strict. That pair is the judge. It is not the accuracy of any model. The human judgments are more than 50,000. The models were benchmarked on 26 April 2023.

Overall accuracy, across the models they report, ranges from 0.8% to 32.0% under Strict and from 0.8% to 46.4% under Relaxed. Those are ranges across models, not a rate for one product, and not a rate for one category of question. Fast-changing questions are the category that matches an old page. The paper says GPT-4 generally obtains the highest accuracy on questions that need up-to-date knowledge, "but it never exceeds 15% across both evaluation modes." 15% is a ceiling they state for that model on that category, in that April 2023 run. It is not the overall 46.4, and it is not a promise that a newer model cleared it. GPT-4 refused to answer fast-changing questions 60% of the time. ChatGPT refused 16% of the time. A refusal is not an old fact stated confidently. A product that must answer, and answers from a stale chunk, is not the system that refused.

False premises are a different failure, and the numbers that get quoted from the abstract often belong to them. For GPT-3.5, ChatGPT, and GPT-4, false-premise accuracy sits between 25.8% and 42.7% under Strict and between 32.3% and 66.9% under Relaxed. ChatGPT is best under Strict at 42.7%. GPT-4 is best under Relaxed at 66.9%. The same sentence places an accuracy of 83.9% on questions about knowledge before 2022. 83.9 is not 66.9, and 66.9 is not the overall Relaxed maximum of 46.4, because a category can outrun the average of all categories. Do not import 66.9 into a paragraph about fast-changing questions. Do not import 15 into a paragraph about false premises. The paper also says larger models do not reliably fix fast-changing questions or false premises. A bigger model reading an old chunk is still reading the old chunk.

Their remedy is fresh evidence at question time, and the paper prints two different sizes for the gain. The introduction says the best GPT-4 plus FreshPrompt variant improves 32.6 points under Relaxed and 49.0 points under Strict over vanilla GPT-4. A later paragraph says GPT-4 plus FreshPrompt achieves absolute improvements of 47 points under Strict and 31.4 points under Relaxed. 49.0 is not 47. 32.6 is not 31.4. The order of the modes is swapped as well as the tenths. This article does not decide which pair is the one the authors would stand behind, and it does not subtract either pair from the best-variant scores to invent a baseline. The best variant they single out, with 15 retrieved evidences, reaches 77.6% under Strict and 79.0% under Relaxed. Those two are close to each other, which is the paper's point about hallucination shrinking when the evidence is current. They are not your index. FreshPrompt retrieves from a search engine when the question is asked. A private corpus you embedded last quarter does not become that search engine because you call the pipeline RAG. Production does not have Google's index of your policy PDF unless you put the current PDF in the index yourself.

A second benchmark asks the question on a clock, and it shows what an old store does to the answer. Kasai, Sakaguchi, Takahashi, Le Bras, Asai, Yu, Radev, Smith, Choi, and Inui announce RealTime QA weekly from news quizzes. The paper's year of results runs from 17 June 2022 through 2 June 2023, 1,470 questions. The results prose says GPT-3 with Google custom search reaches 34.6 generation exact-match, against 15.3 for closed-book GPT-3 and 13.3 for GPT-3 retrieving from a dense index of a December 2018 Wikipedia dump. Read the order. The 2018 dump is older than every question in the window. On that column it does not beat the model's memory. 13.3 is below 15.3. An old retrieved document is not a neutral extra. It can be worse than answering with no document. The same prose puts original multiple-choice accuracy for GPT-3 with Google custom search at 66.5, and the "none of the above" variant at 58.4. 66.5 is multiple choice with the real options. 34.6 is generation with no options. Quoting 66.5 as "GPT-3 gets two thirds of fresh questions right" has changed the task (Kasai et al., RealTime QA, arXiv:2207.13332, revised 28 February 2024).

The error split is a smaller window, and it does not add up. For the first six weeks, on the best generation model, they print correct 52%, retrieval error 34%, and reading-comprehension error 13%. 52, 34, and 13 add to 99. This article does not invent the missing point, and it does not call 34 a stale-index rate. Retrieval error, in their words, is the bucket. A temporal gap is one way a retrieval fails to contain the answer, which their examples separate from a document that is simply unrelated. Humans, the authors, answered 300 multiple-choice questions sampled from ten weeks between 17 June 2022 and 13 January 2023, using Google, at 96.7%. That is the authors with a search box. It is not the model, and it is not the USA Today top scorers the footnote says record perfect scores, who are a different population on a public quiz. The paper's own limits: English, topics that tend to be U.S. and U.K., questions announced weekly rather than at the moment the fact changes. One passage says about 10% of questions were not strictly time-sensitive. A later table note says about 12%. Those are two figures for a related cut. Neither is a rate of stale chunks in a company corpus. The abstract's finding to keep is narrower than any of the percentages: GPT-3 tends to return outdated answers when the retrieved documents do not contain enough to answer. The model does not announce the gap. It fills it.

Write the pair so a person who was not on the call can repeat it. The sentence, copied from what the customer was shown. The passage identifier, and the date on that passage: a header, a filename, an effective-date field, or the words "no date." The live object, and its date, or the words "no date" if the system of record has none. Which one is newer, or the line that you cannot tell. If you cannot tell, the sentence is not current, because currency you cannot show is a claim you are still making. Store the pair where the complaint is handled. Do not paste the customer's account into a chat tool to debate the dates. The dates are the evidence.

Do not send the model back in to choose the newer page. "Which of these is current?" produces another answer with the same defect, and it spends a call on a comparison a person can do with two timestamps. The stop, while you compare, is the serving of that sentence as if it were the rule today. When to turn a live AI feature off is the wider rule for a feature whose next answer can hurt someone. This is the narrower one: the old page is the claim you are making about the present. Leave the rest of the product up if it does not depend on that sentence. Re-embedding the same old file, or raising the number of chunks, retrieves the old file more thoroughly. It does not change the date.

What an answer that can name its date looks like, as a product shape, is narrower than a news benchmark. The support agent we shipped for a DTC brand answers order status, returns, and product questions from order data, policies, and the catalog, and it takes actions — lookups, return initiation, address changes — behind permission guardrails, with escalation to a person. It is not a white-label engagement, and it is not a freshness study. Nothing in that description is a headcount, a duration, or a result figure. The shape is an answer a customer reads that is supposed to come from a source you can date. If the source is an order, the order has a time. If the source is a policy, the policy has a version. The FreshQA ceiling and the 2018 dump will not appear on that screen, and they should not be spoken as if they had.

There is a version of this where a rescue is the wrong purchase. Nothing is in production, so no customer has been shown a sentence, and the dates you would compare do not exist yet. The index is empty. That is ingestion, not staleness. The corpus is still being designed, and two people cannot produce the same current file. A retriever cannot mint the missing date. A customer has already seen another customer's data. An old policy does not close a leak. The wrong value was a write into a system of record, and the sentence was never the point. That hour is when the agent already wrote the row. Or nobody ever named a metric, so "current" has no meaning you could miss. Name the metric before you buy a rebuild.

When the sentence is already in front of a customer and you cannot show that the passage is the current one, the work that remains is specific. An audit of what is salvageable, in the first one to two weeks, including the date field the ranker should have had. One baseline the engagement has to move, agreed before the rebuild. That is what an AI pilot failed to reach production engagement is for: finished in 8–12 weeks, scoped after the audit, and if the agreed metric does not move you do not pay for the build. The pair you wrote tonight — sentence, two dates, or a missing date — is the kind of case the harness has to include, or the next release will sound just as current. The harness is AI evaluation and observability. If you do not yet know whether the pilot can be saved, a two-week AI readiness assessment hands you the findings either way. The free production-readiness diagnostic is the short way to see whether you are looking at an old page, a missing document, or a system that has not reached a customer. Those are the terms on the site. They are not a promise that every stale chunk is a rescue, and they are not a rate. 15% is one model's ceiling on fast-changing questions in April 2023, under a paper whose overall range is a different number. 13.3 is generation exact-match from a 2018 dump, on a year of news quizzes, and it lost to no retrieval at all. Neither is your staleness rate, and neither belongs on the invoice.

Before you tell anyone the answer was current, read both dates. Stop the sentence when the retrieved page is older, or when you cannot tell. Keep the pair. Leave the model out of the comparison. A fluent answer from an old page is still an old answer.