When two sources disagree

Two passages you retrieved disagree on the sentence you are about to show a customer as a rule. Tonight, do not show that sentence. Write both passages, with where each one came from, or stop the answer. Do not ask the model to pick. If you cannot tell whether they disagree, you are in the same place as if they do: you do not have a single rule, and a single confident sentence is how a disagreement leaves the building. Count how many answers today stated one rule while two incompatible passages were in the prompt, or write that you cannot count.

Opening the page an answer already cited is a different check. Open the page the answer cited asks whether that page supports the sentence the customer was shown. Here you already hold two passages, and they do not say the same thing. A sentence you selected from the page that agrees with you will pass a support check against that page. That is how a choice gets laundered into a citation. The five-step order for a wrong retrieval answer is the production RAG failure checklist. Its fourth check is whether the model used a passage that was actually in the prompt, including a prompt that contradicts itself. This page is the rule for that contradiction, before anyone tunes the generator. Keeping enough of a bad answer to reconstruct it is what to keep after a wrong AI answer. The trace is what was sent. This is what you refuse to send. An error you treated as permission is when the guard fails open. A row the agent already wrote is a reversal. Tonight's change is the sentence that has not gone out.

The figure people will want is a share of queries whose sources disagree, and the paper's own 65% is a wider bucket than disagreement. Cattan, Jacovi, Ram, Herzig, Aharoni, Goldshtein, Ofek, Szpektor, and Caciularu, at Google Research and Bar-Ilan University, built a benchmark they call Conflicts (arXiv:2506.08500, version 2). The statistics section says the final set contains 458 instances after filtering out 18 with no relevant sources. There are on average 9.2 search results per query. Queries were curated from FreshQA, SituatedQA, QACC, and ConflictingQA, then retrieved with Google Search. The paper says the distribution "is a result of our curated query selection and do not necessarily reflect the natural distribution of conflicts in search engines." Read that grammar as theirs. The point is the one they state: this is not how often the web disagrees, and it is not your policy corpus.

Table 2 is the split that keeps 65% from becoming your incident rate. No conflict: 161. Complementary information: 115. Conflicting opinions: 115. Outdated information: 62. Misinformation: 5. The total is 458. 115 plus 115 plus 62 plus 5 is 297. The paper says 65% of the instances (297 of 458) were flagged as conflicting, and it defines that flag as complementary information, conflicting opinions, freshness, and misinformation together. Complementary, in their taxonomy, means the answers "refer to different real-world concepts but are mutually compatible," so that "a single person can reasonably agree with all of them simultaneously." That is not a contradiction. Conflicting opinions are the category where the answers "are not mutually compatible": 115 of 458, not 65%. Misinformation is 5 cases. They say genuine misinformation was rare in the top results, and that most earlier work simulated it by perturbing text. Two annotators with linguistic expertise labelled each instance, then reconciled, then a third expert reviewed. To help them, the authors generated a per-source hint with Gemini Pro 1.5, and told the annotators to treat the hints as optional and to disregard them if they looked like hallucinations. The hints are not the labels. No instance was labelled "other," which they take as a sign the taxonomy covered the set. That is a statement about their set.

The behaviour they want from a model depends on which of those categories you are in, and a policy bot is narrower than web question-answering. No conflict: a direct answer, without invented alternatives. Complementary: consolidate the partial answers, and do not frame them as a debate. Conflicting opinions: "explicitly reflect the debate" and neutrally summarize the viewpoints. Freshness: prioritize the up-to-date information, and you may mention that older sources exist. Misinformation: disregard the inaccurate source and answer from what is reliable. Their example of a false disagreement is the useful one to steal. Snippets for "when did Toyota first come" mention 1933, 1937, and 1955. The full pages show those dates belong to different events in the company's history. A model that picks one year has manufactured a conflict the pages did not contain. Their example of no-conflict noise is a peach question, where one page names an origin and another names a favourite farm. The farm is not an answer. A customer-facing policy is a different object again. If the two passages are the return rule you published and a draft of that rule, "a balanced summary of the debate" is the wrong style. The live rule is the rule. The draft is not a viewpoint. Say which document is in force. If you cannot, stop.

Models in the paper are often loyal to the sources and still answer in the wrong style, which is why you do not hand them the choice. On the response task, vanilla expected-behaviour accuracy runs from 59.4 for Gemma 3 27B to 68.3 for Gemini 2.5 Flash with thinking. GPT-4o vanilla is 67.2 on that behaviour score, 87.3 on answer recall, and 91.2 on factual grounding. High grounding with a middling behaviour score means the reply can stick to the pages and still be the wrong kind of reply. The hardest category they single out is conflicting opinions: Gemini 2.5 Flash with thinking reaches only 36.2% expected behaviour under the vanilla prompt. A manual read of 40 non-adherent outputs from that model found the usual misses. On complementary questions, the reply kept one correct answer and dropped the rest. On conflicting opinions, it presented a single viewpoint, or several viewpoints with a bias. An oracle prompt that inserts the gold conflict type gains 24 points on average over vanilla. The examples they print are +28.2 for Gemma, +21.4 for GPT-4o, and +21.0 for Gemini 2.5 Flash with thinking. You do not have the gold label at the moment a customer is waiting. The oracle is an upper bound they use to show the model can comply when told the type. It is not a prompt you paste into production. Pipeline and taxonomy-aware prompts, which ask the model to reason about the conflict, gain 9 and 5.5 points on average. An improvement on a benchmark is not a control. Their automatic judge matched human judgements at 0.89 accuracy on 100 examples. That 0.89 is the judge. It is not your product (Cattan et al., DRAGged into Conflicts).

A second paper measures a different fight, and its rate does not belong next to 36.2 or 65. Wu, Wu, and Zou, at Stanford, built ClashEval: "over 1200 questions" across six domains, including drug dosages, Olympic records, and locations, with the retrieved answer perturbed from subtle errors to blatant ones. They benchmarked six models, including GPT-4o. The abstract's finding: models are "susceptible to adopting incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time." The more unrealistic the retrieved error, the less often the model adopts it (Wu, Wu, and Zou, ClashEval). That 60% is one wrong document against the model's own memory. It is not a share of policy pages that contradict each other, and it is not the Conflicts behaviour score. The paper mentions public headlines, glue on pizza and eating rocks, as illustrations of bad pages being repeated. Those headlines are not a rate. Do not "repair" a disagreement by trusting the model's prior. ClashEval is the measurement that says the prior loses to a wrong page. Do not repair it by trusting whichever page the model preferred, when a second page disagrees. A person reads both.

Do the check before the sentence ships, on the passages that were actually retrieved, not on a demo query. Open the two. Mark them as the same claim, as compatible extras, or as not compatible. Same claim, including a date that is only more specific: answer, and do not invent a dispute. Compatible extras, in the Conflicts sense: the customer may need both, and stating one while dropping the other is the under-specification they saw in the 40 outputs. Not compatible, and both are being offered as the rule a customer would rely on: do not show one of them as the rule. If one passage has a date and you can show that the later one is the document you published, the freshness decision is that date, not a feeling that the newer page "looks right." If one passage is a draft that was indexed beside the live rule, take the draft out of what retrieval can return, and do not answer from it while you do. Write the pair: the two sources, the sentence you refused, the time. Then the count of answers that already stated a single rule with both passages in the prompt. If the log cannot support the count, write that the count does not exist. The first refusal is still correct. The Conflicts annotators were told the model's hints were optional. You are the annotator. You do not send the model back in to grade the pair it is about to speak.

Three moves that feel like a resolution and ship the disagreement anyway. Swapping the model, or raising the amount of context, because the checklist's coverage numbers moved and the answer barely did. The generator is the wrong module while you have not marked the pair. Asking for a "balanced" answer when one document is the policy you issued and the other is a draft, a joke, or a page about a different product. Deleting the log so the pair is gone. The log is how the next person sees that you stopped. A citation glued onto the sentence you preferred will pass the page-opening check for the page you chose, and it will hide the page you did not. Keep both identifiers in the note, including the one you refused to quote.

The shape that makes the stop obvious is a product that answers from more than one policy and can decline to act. An AI support agent for a DTC ecommerce brand answers from order data and policies, and it puts lookups, returns, and address changes behind permission guardrails, with escalation to a person when confidence is low. Nothing in that description is a headcount, a duration, or a result figure, and it is not a conflict study. The shape is the test: if two policy pages disagree on whether a return is allowed, the return does not start, and the customer is not told a single rule. A design that must always produce a sentence, and that treats "I have two pages" as a prompt to pick the confident one, cannot make that split.

There is a version of this night where hiring a production engagement is the wrong purchase. Nothing is in production. The failure in front of you is an empty index, which is the first check on the retrieval checklist, not a disagreement. You are still designing the corpus, and you can write the rule now: two incompatible passages do not become one sentence. You do not need a rescue to type that. A customer has already seen another customer's data. The first hour is the tenant boundary. The wrong value was a write the agent already committed, not a sentence you can still hold. Or no metric was ever named, so you cannot say what "a correct rule" was supposed to move, and a tighter refusal will not invent the metric.

Where the feature is already live, two retrieved passages are not compatible, and the product is stating one of them as the rule, the work is the stop, the note, and the count, then the evaluation of whether later answers still collapse a pair. That sequence is what an AI pilot failed to reach production engagement owes you when the pilot's answer was whichever passage came back first: an audit in weeks one and two, then a path that cannot state a rule the sources do not share, measured against a number you agreed. If the agreed metric does not move, you do not pay for the build. The findings of a two-week AI readiness assessment are yours whether or not that build happens. AI evaluation and observability is how you learn, afterwards, whether the behaviour score is moving. It is not a substitute for refusing the sentence tonight. The 36.2% was a laboratory category on web opinions. Your number is the count you wrote down, or the admission that you could not.

On the night the prompt contains two passages and the draft reply states one rule, stop the sentence before you change the model. Write both sources. Say they are not compatible, or say you cannot tell. Refuse the single rule. Keep the note. Do not ask the model which page it prefers, and do not trust its prior to break the tie. A pair you have not marked is not a citation. It is two stories, and the customer was about to hear only the one that fit.