When one question gets two answers

The same question, asked twice, came back as two sentences a customer could be shown as the answer. Nothing you changed in the sources explains it. Stop. Do not serve either sentence as the answer until you can show one of them twice, or until you write that you cannot. Do not ask the model which sentence to keep. Read the temperature and the seed you actually sent, and do not treat a zero as a proof that the next call will match. If any of that requires the person who last edited the prompt, you have not decided what the product says. You have two drafts and a preference.

This is one question with two outputs, and no source that changed between them. It is not two retrieved passages that disagree. When two sources disagree is that case: you do not show one sentence as a rule the passages do not share. Here the sources may be the same, or there may be no passage in dispute, and the sentences still differ. It is not the check that a cited page supports the sentence. Open the page the answer cited assumes there is a sentence and a page. Two sentences are two checks, and neither is yet the answer. It is not how long the customer waited. Time the answer the customer waited is a duration and a timeout. A fast wrong pair is still a pair. It is not an empty retriever. If the retriever returned nothing, say that, and do not bury it inside this article. And it is not a model that moved under you across months. Model drift: what changes underneath your AI system is that slower change. Two calls in one hour can diverge with the model id unchanged.

Temperature zero is not the proof people think it is, and one lab measured the gap on a single prompt. Horace He, with others at Thinking Machines Lab, 10 September 2025, sampled 1,000 completions at temperature 0 from `Qwen/Qwen3-235B-A22B-Instruct-2507`, non-thinking mode, prompt "Tell me about Richard Feynman," 1,000 tokens each. They report 80 of 1000 completions as unique. The most common occurred 78 times. The completions were identical for the first 102 tokens. The first divergence was at the 103rd token. Every completion produced "Feynman was born on May 11, 1918, in." 992 then produced "Queens, New York." 8 produced "New York City." 992 plus 8 is 1,000. That split is inside this run. It is not a rate for your product. When they enabled the batch-invariant kernels they describe, all 1,000 completions were identical (Thinking Machines Lab, Defeating nondeterminism in LLM inference).

Keep the other experiments in that post in their own boxes. A matrix multiply of the same tensors, repeated 1,000 times, matched bitwise. That is run-to-run determinism of one kernel on one machine. It is not the 80. The post's point is that a kernel can be run-to-run deterministic and still not be batch-invariant: the result for one request depends on how many other requests shared the batch, because the reduction order changes with the batch. From the caller's side, the other callers are not an input. They are load. A separate timing run, one GPU, Qwen-3-8B, 1,000 sequences with output lengths between 90 and 110, took 26 seconds on default vLLM, 55 seconds on an unoptimized deterministic path, and 42 seconds with an improved attention kernel. Those are durations. They are not a count of unique answers, and they are not the 235B run. Do not average 80 with 26. The post is one vendor's stack and one prompt about a physicist. It is not a support queue, not a policy corpus, and not a probability that your next pair of calls will split 992 to 8.

A model vendor's own API docs say the same thing in different words, about a different system, and the numbers do not transfer. OpenAI's chat reference describes `seed` as beta: if specified, the system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. "Determinism is not guaranteed," and the caller should refer to `system_fingerprint` to monitor backend changes. The temperature description says higher values such as 0.8 make the output more random, while lower values such as 0.2 make it more focused and deterministic. "More deterministic" is their phrase. It is not "identical" (OpenAI, Chat API reference). Their cookbook on the seed parameter repeats the best-effort line and adds a case the reference leaves implicit: there is a small chance responses differ even when the request parameters and the `system_fingerprint` match, which they attribute to inherent non-determinism. The fingerprint, in that cookbook, changes when you change request parameters, or when OpenAI updates numerical configuration of the infrastructure, which they say may happen a few times a year (OpenAI, Reproducible outputs with the seed parameter). "A few times a year" is their expectation for that configuration change. It is not a count of your incidents. "A small chance" is not 80 out of 1,000, and it is not a license to ignore a pair you have already seen.

Read the two populations apart. The Thinking Machines run is one open model, temperature 0, a fixed prompt, 1,000 tokens, on their inference path, and it still produced 80 completions. The OpenAI sentences are a promise about their API: best effort, not guaranteed, and a residual difference even when the fingerprint matches. Your product may be on neither stack. Quoting 80 as your error rate, or quoting "small chance" as a reason the two sentences in front of you are fine, uses one lab's result as the other's alibi. Neither number is this customer's pair.

What you stop is the sentence, not the research. Write down both outputs, the time of each call, the model id, the temperature, the seed if you sent one, and the fingerprint if the API returned one. If you did not send a seed, write that you did not. If the two fingerprints differ, you are allowed to say the backend may have changed between the calls. You are not allowed to pick the sentence you prefer and serve it. If the fingerprints match, the cookbook's own residual still applies on that API, and the Qwen run shows a split at temperature 0 with no fingerprint in the story at all. Matching fingerprints are not a referee. Then try to show one sentence twice, on purpose, with the same parameters, and write whether you could. If you cannot, stop serving an answer to that question until you can, or until a person is the one who sends it. Asking the model "which of these is right" produces a third sentence. It does not adjudicate the first two.

A score on a set is a different object, and it will not break the tie. The evaluation suite is a dataset, a gate, and a score. A green gate says the set passed. It does not say these two calls returned one sentence. You can fail the gate and still have a deterministic wrong answer, and you can pass the gate and still serve a pair. Do not quote the pass rate to the customer who held both sentences. The record of what was sent on one wrong answer is what to keep after a wrong AI answer. Keep both, if you have two. A checklist for a retrieval pipeline is the production RAG failure checklist. Use it when the failure is the pipeline. Use this page when the pipeline is not what changed and the sentences still differ.

The support case is a shape, not a study of duplicates. An AI support agent for a DTC brand answers from order data and policies, escalates to a person when confidence is low, and is scored for resolution accuracy before a release. That score is accuracy against a set. It is not a demonstration that one order question returns one sentence every time. Nothing in that description is a headcount, a duration, or a result figure. If the only control you have is a confidence threshold, a pair of confident and contradictory sentences both clear it. The threshold is not a digest of the output.

There is a version of this incident where hiring someone to build the path is the wrong purchase. Nothing is in production, so there is no pair a customer saw. The two sentences came from two different questions, and you have called them one. A customer has already seen another customer's data. Identical answers would not close that leak. The wrong value was a write the agent already committed. Two phrasings are not the problem; the row is. Or no metric was ever named, so there is nothing a rescue could be measured against. Why AI pilots fail to reach production is the longer account of stalls. This page is one stall: you cannot say what the product will answer.

When the pair is real, the missing object is specific. A way to show the same answer twice for the question a customer was asked, or a person in the path until you can. The price someone says they were shown is a different record, show the price they accepted. The commit that is serving the page is a different record, name the commit they are running. Either can be true while the two sentences still disagree, and neither picks a sentence for you. Rescue, where an AI pilot failed to reach production, starts with an audit in weeks 1–2 and runs 8–12 weeks, scoped after the audit rather than before it. The build is unpaid if the agreed metric is missed. Rollout sits behind a human-review gate. Those facts are the offer. They are not a temperature, and they are not a promise that two calls will match. If the first slice is the ability to refuse a question you cannot reproduce, an AI integration sprint is the shape where that refusal is the acceptance sentence. AI evaluation and observability is the score you still want, beside the reproduction, not instead of it. If you do not yet have a live question to reproduce, a two-week AI readiness assessment hands you the findings either way. The findings are yours. They are not 80, and they are not a seed.

Before you send either sentence, write the pair down. Same question, two outputs, the temperature, the seed or its absence, the fingerprint or its absence. Try to produce one of them again. If you cannot, do not serve one. Do not ask the model to vote. A zero on the temperature dial is a setting. The second call is the proof.