Open the page the answer cited

An answer is live, and it cites a page. Tonight, in this order. Open that page yourself. Decide whether the page supports the sentence the customer was shown: fully, in part, or not at all. If you cannot open the page, or the page does not support the sentence, stop serving that sentence as cited. Write the pair down — the sentence, the URL, and what you found — and do not ask the model whether the page agrees. Swapping the model before that pair exists replaces an answer you have not checked with another answer you have not checked.

This is the page a person can click. It is not the five checks that decide which stage of a retrieval pipeline failed. Those checks — indexed, returned, present in the prompt, used, and the right shape — are what to check when a production RAG answer is wrong, and they stop at the first failure. A pipeline can pass all five and still show the customer a citation whose page does not contain the sentence, if the citation was attached after the fact or points at a different URL from the passage that was sent. It is also not the record you keep so a later complaint can be reconstructed. What to keep after a wrong AI answer is that record: the input, the output, the model, the tool result. A complete trace tells you what was sent. It does not, by itself, tell you that the page at the link supports the sentence. You still have to open the page.

The measurements that separate those two questions come from a human audit of systems that already showed citations. Liu, Zhang, and Liang, Findings of EMNLP 2023, define verifiability as two rates. Citation recall: generated statements about the external world are fully supported by their citations. Citation precision: each citation supports its associated statement. They audited four generative search engines — Bing Chat, NeevaAI, perplexity.ai, and YouChat — with human annotators. The abstract's averages: "a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence" (Liu, Zhang, and Liang, Evaluating Verifiability in Generative Search Engines). 51.5% is not 74.5%. Recall is a property of sentences. Precision is a property of citations. A response can cite accurately on the sentences it cites and still leave other sentences with no support, which is how recall can sit far below precision. Quoting one of them as "about half of citations are wrong" has mixed the two columns.

The setup is what keeps the averages from becoming your product's rate. Each system was evaluated on 1,450 queries. Responses were scraped between late February and late March 2023. The paper thanks 34 annotators. A statement is fully supported, under the attributable-to-identified-sources test they use, when a generic hearer would affirm "According to the cited webpages, the statement," in the context of the query and the response. Precision distinguishes full support, partial support, and no support. Partial support counts toward precision only in the case they define: the statement is fully supported by the union of its citations, and no single citation fully supports it alone. NeevaAI abstains on 22.7% of the 1,450 queries, because its answer is built to sit inside a conventional results page. Bing Chat, perplexity.ai, and YouChat abstain on less than 0.5%. A system that refuses a hard query is not the same object as a system that answers it. The paper also prints an inverse correlation between citation precision and perceived utility, r = −0.96: the responses that seemed more helpful were often the ones with inaccurate citations. That coefficient is their correlation on this audit. It is not a law that a useful answer is a badly cited one, and it is not a number you paste onto a support bot.

What you are allowed to copy is the split, and the action that makes the split checkable. Open the page. Mark the sentence. Those four products, in early 2023, are not the model you are serving, and the queries were essay questions, debate prompts, explain-like-I'm-five posts, how-to queries, and Natural Questions slices, not your customer's order. Fluency is a different score. The paper says responses often have high fluency and perceived utility while citation recall and precision stay low. A fluent answer is not evidence the page agrees. An average across four engines is not any one engine's rate, and it is not a rate for a system that has no citations at all. If your answer does not cite a page, this audit does not apply, and the pipeline checklist is the article that does.

A page that supports the sentence can still be the wrong kind of support. Wallat, Heuss, de Rijke, and Anand separate citation correctness from citation faithfulness. Correctness is whether the cited document supports the statement. Faithfulness is whether the model actually relied on that document, rather than answering from what it already "knew" and then attaching a source that happens to match. They call the second move post-rationalization. The experiment is on Command-R+, a model trained for retrieval-augmented generation. They generated attributed answers, then appended short statements — on average 2 to 4 tokens — into other documents and asked again. Recovering the old statement in the new answer worked in 63–70% of cases, which is the filter for whether the adversarial document could have been cited for that statement. Injected into random documents, the model cited those documents in 12% (116/936) of cases. The share was 57% when the forged documents had been relevant but uncited, and 55% when they had been cited for a different reason. The abstract's "up to 57% of the citations" is that condition, not a rate for citations in the wild (Wallat et al., Correctness is not Faithfulness in RAG Attributions).

The paper's own limit is the one to keep. The datasets for the three forgeries are 1,344 QA pairs (random), 702 (relevant but not cited), and 829 (cited for other reasons). The 116/936 figure is the random condition after the statement had to reappear. The authors write that the empirical analysis is limited and that future work has to test a broader set of models. One model, short inserted strings, a constructed set of documents. 57% is not your queue, and it is not the Liu average wearing a different name. Correctness and faithfulness are different failures. Tonight you can check correctness, because the page is something you can open. Faithfulness needs the trace of what was sent to the model. If that trace does not exist, say so, and do not pretend the open page proved the model read it. The keep-the-trace article is how that record gets decided before the complaint. This article does not reconstruct a prompt you never stored.

Write the pair so a person who was not on the call can repeat it. The sentence, copied from what the customer was shown, not paraphrased after the argument. The URL that was actually linked, which may differ from the title the answer printed. Whether you could open it: yes, a login wall, a 404, a different document than the title. And the mark: full support, partial, or none, using the same plain test. All of the sentence is in the page. Some of it is. None of it is. If the answer has three citations, you have three pairs. A citation that partially supports one clause does not support the sentence, unless you have checked the union the way the Liu definition does, and you have written that you did. Store the pair where the complaint is handled. Do not paste the customer's whole account into a chat tool in order to discuss the page. The page is the evidence. The account is a different disclosure.

Do not send the model back in to grade its own citation. Asking "does this URL support the sentence?" produces another answer with the same defect, and it spends another call on a question a person can settle by reading. The stop, while you read, is the serving of that sentence as if it were cited. When to turn a live AI feature off is the wider rule for a feature whose next answer can hurt someone. This is the narrower one: the citation is the claim you are making to the customer, and a claim you have not opened is a claim you are still making. Leave the rest of the product up if it does not depend on that sentence. Turning off search, checkout, and the phone line because one answer cited the wrong help page is a bigger incident than the one you have.

What an answer that names its source looks like, as a product shape, is narrower than a research audit. The support agent we shipped for a DTC brand answers order status, returns, and product questions from order data, policies, and the catalog, and it takes actions — lookups, return initiation, address changes — behind permission guardrails, with escalation to a person. It is not a white-label engagement, and it is not a citation study. Nothing in that description is a headcount, a duration, or a result figure. The shape is an answer a customer reads that is supposed to come from a source you can open. If the source is an order record, open the order record. If the source is a policy page, open the policy page. The Liu averages will not appear on that screen, and they should not be spoken as if they had.

There is a version of this where a rescue is the wrong purchase. The answer cites nothing, and the failure is that the right document was never indexed. Run the five checks. You do not yet have a production answer, so there is no page to open and no sentence a customer saw. A customer has already seen another customer's data. A bad citation does not close a leak, and the first hour of that incident is a different article. The value that was written into a system of record was wrong, and the citation was never the point. That hour is when the agent already wrote the row. Or nobody ever named a metric, so "fixed" has no meaning you could miss. Name the metric before you buy a rebuild.

When the sentence is already in front of a customer and the page does not support it, the work that remains is specific. An audit of what is salvageable, in the first one to two weeks. An evaluation harness. Guardrails before anything acts on its own. One baseline the engagement has to move, agreed before the rebuild. That is what an AI pilot failed to reach production engagement is for: finished in 8–12 weeks, scoped after the audit, unpaid if the agreed metric does not move, with rollout behind a human-review gate. The pair you wrote tonight — sentence, URL, support — is the kind of case the harness has to include, or the next release will cite just as fluently. The harness is AI evaluation and observability. If you do not yet know whether the pilot can be saved, a two-week AI readiness assessment hands you the findings either way. The free production-readiness diagnostic is the short way to see whether you are looking at a citation, a missing document, or a system that has not reached a customer. Those are the terms on the site. They are not a promise that every bad link is a rescue, and they are not a rate. 51.5% and 74.5% are one paper's averages across four 2023 search engines. 57% is one condition in a different paper, on one model. None of them is your citation rate, and none of them belongs on the invoice.

Before you tell anyone the answer was grounded, open the page. Mark the sentence. Stop serving the sentence as cited when the page does not support it, or when you could not open the page. Keep the pair. Leave the model out of the grading. A link is not support until someone who can read has used it.