Designing an interface for a system that is sometimes wrong

Every convention in interface design assumes the system is either right or unavailable. A form validates or it does not. A model is neither: it is confidently approximate, and its characteristic failure state is one conventional UI has no vocabulary for — almost right. That leaves six design decisions nobody had to make before. How to show confidence without a percentage nobody can interpret. What happens to a correction after a user makes one. What a degraded state looks like when a system is partly capable rather than down. Where a human confirmation step earns its friction and where it is theatre. How the provenance of an answer is made inspectable. And whether a chat box is the right surface at all — it usually is not; it is the laziest available one.

Almost-right is the hard case, and every other decision follows from it. In Stack Overflow's 2025 Developer Survey, the biggest single frustration reported by developers — the people best equipped to check these systems — was cited by 66%: "AI solutions that are almost right, but not quite." The same survey found 46% actively distrust the accuracy of AI tools against 33% who trust it, and 45.2% naming debugging AI-generated code as more time-consuming. Asked when they would go and ask another person instead of the tool, 75.3% selected "When I don't trust AI's answers" (Stack Overflow, 2025 Developer Survey: AI). Wrong announces itself; almost right does not, which is exactly why it costs more — somebody acts on it. Users already route around uncertainty, unaided, doing detection work the interface was supposed to do.

Showing confidence, without a percentage. NIST wrote the useful version of this down before the current wave of products existed. Among its four principles of explainable AI is one called Knowledge Limits: "A system only operates under conditions for which it was designed and when it reaches sufficient confidence in its output." Systems should "identify cases in which they were not designed or approved to operate, or in cases for which their answers are not reliable," because declaring those limits "safeguards answers so that a judgment is not provided when it may be inappropriate to do so" — a practice that "can increase trust in a system by preventing misleading, dangerous, or unjust outputs" (NIST, Four Principles of Explainable Artificial Intelligence, NISTIR 8312, September 2021).

That section separates two failure states most interfaces collapse into one: a system can be asked something outside its domain, or be inside its domain and unsure. NIST's illustration is a bird classifier shown a picture of an apple, where the honest output is that it found no bird — "both an answer and an explanation." The second case is a bird too blurry to identify: "I found a bird in the image, but the image quality is too low to identify it." Notice what is missing. No percentage, no gauge, no amber badge. It states the boundary in the vocabulary of the task and names what would fix it — which doubles as the test. A confidence display that does not tell a user what to do differently has not earned its pixels.

Percentages fail for a duller reason than being hard to calibrate: a number has no reference class. Told an answer is 82% confident, a user cannot say whether that is unusually good for this system on this question, so it becomes a feeling and is ignored by the third screen. NIST's review of the human baseline points at where the signal lives — while people predict their accuracy poorly in the abstract, "when asked to assess their confidence for a given decision vs. this explicit judgment, people can gauge their accuracy at levels above chance." Per decision, in context. The interface equivalent is a marker on the specific claim that is soft — this date came from a document, that total was inferred — not one aggregate score that applies to everything and so to nothing.

A correction is a decision about destination, not about editing. Every team gets the edit affordance right; almost none answer the question the user asks silently every time they use it: where did that go? There are three honest answers. It applies to this output only. It applies to everything this user sees from now on. Or it applies to everyone. Most products implement the first and, by saying nothing, imply the third — so the user corrects on Tuesday what they corrected on Monday, concludes the system does not learn, and stops correcting. That is worse than never offering the affordance, because corrections were the most valuable data the product was producing.

Design the destination explicitly and corrections become a stream of labelled failures — the material an evaluation set is built from, arriving free, from people who know what a right answer looks like. That is the join between the interface and AI evaluation and observability: a correction captured with the input that produced it, the output that was wrong and the fix is a test case; one that only patches the screen is a rounding error. Turning them into a suite that gates releases is the evaluation suite.

The degraded state is where the design language runs out entirely. Conventional software has two states everybody knows how to draw: working, and down. An AI feature has a third that is more common than either — retrieval is stale, the model is rate-limited, one tool in a chain is failing. Hide the feature and the user learns nothing and assumes it is broken. Warn on everything and you build the problem NIST raises in its discussion of alerts, where more information may be needed "to address non-compliance in responding to weather alerts (e.g., 'cry-wolf' effects)" — a warning that fires constantly is a warning nobody reads. The honest option is to narrow the promise: keep the feature, name the capability that is missing, let the user judge whether the reduced version still helps. That requires the system to know at runtime which capability degraded, an architecture decision taken months before anyone opens a design file.

A human confirmation step is worth its friction under three conditions and is theatre otherwise. The person must be able to evaluate the thing. They must have, on the screen in front of them, whatever evaluating it requires. And the action must be expensive to reverse. Fail the first and you have asked a supervisor to approve a forecast they cannot check. Fail the second — the common case — and you have shown someone a decision without the evidence behind it and asked them to press Approve, which adds no oversight; it moves liability from the system to the user. Fail the third and the step is pure cost: if the action is cheaply undoable, ship it and design the undo instead.

Reversibility is the axis that actually governs this, and it sits under a designer's control far more often than the model's confidence does. The version that meets all three conditions is worth building: in a support agent for a DTC ecommerce brand, escalation is triggered by confidence and the human receives the full conversation context with it. The context is the load-bearing half. An escalation arriving without it is not oversight, it is a queue.

Provenance has to be mechanical, not narrated. The Government Accountability Office puts the problem plainly in its accountability framework for AI, organised around governance, data, performance and monitoring: "AI systems pose unique challenges to such oversight because their inputs and operations are not always visible" (US Government Accountability Office, Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities, GAO-21-519SP, June 2021). The tempting fix is to ask the model to explain itself, and NIST separates the two things that then look identical on screen: "Explanation accuracy is a distinct concept from decision accuracy. Decision accuracy refers to whether the system's judgment is correct or incorrect. Regardless of the system's decision accuracy, the corresponding explanation may or may not accurately describe how the system came to its conclusion."

A generated rationale is a second output with its own error rate, usually more fluent than the answer it explains — more persuasive, no more true. What is checkable is mechanical: which document, which field, which record, read at what time, through which tool call. Show that, one click away, and a user verifies in four seconds what no self-explanation could establish. The same report adds two constraints: an explanation risks exposing proprietary detail, so what is shown must be scoped rather than dumped; and one that is not meaningful to its audience is "in jeopardy of being ignored or not recognized as an explanation."

Which brings us to the chat box, and why it is usually the wrong answer offered confidently. NIST sorts explanation styles by degree of interaction: declarative, where the system explains and nothing further happens; one-way, where the user can query; and two-way, in which "the person can probe further, and the machine can probe back, ask clarifying questions, or provide new avenues of exploration." Of that last category the 2021 report observed: "To our knowledge, two-way interactions do not yet exist." Models have moved a long way since. Most products shipping a chat box have not — they ship the two-way affordance with none of the two-way behaviour, and a text input that accepts anything is an implicit promise to handle anything.

That is a design cost, not a technical one. A blank input removes the enumeration of what the system can do, so the user finds the boundary by hitting it, and failures read as their fault rather than the system's limit. It also removes every place an interface would otherwise have put uncertainty: no field to mark as inferred, no source to hang beside a number, no structure to degrade gracefully, because there is no structure.

Chat is right when intent genuinely cannot be enumerated, and even then it is not the cheap option. A multi-assistant AI platform we built for Inflectiv treats conversation as the product, and still needed specialised assistants per knowledge domain, history and settings across web and mobile, and prompt monitoring with dashboards and alerts underneath. The conversation is the visible tenth; the rest is what stops an open-ended surface from becoming an unbounded promise.

The test for all six decisions is behavioural and takes an afternoon. Give someone a task where the system will be almost right — not wrong, almost right — and watch whether they catch it; if they do not, your confidence display is decoration. Ask where their last correction went and compare the answer to the truth. Turn retrieval off mid-session and see whether the interface says what it lost or just gets quietly worse. Remove the evidence from behind a confirmation step and see whether approval rates move; if they do not, the step was already theatre.

That is what AI product design services have to mean now, and why the four patterns that page names — progressive trust, visible reasoning, graceful uncertainty and human handoff — are engineering constraints as much as design ones. Each needs something the system knows at runtime: which capability degraded, which claim came from where, which output fell below threshold. A design that assumes those signals exist, on a system that never exposed them, produces a beautiful prototype and an unbuildable specification.

Where this one stops, what breaks when an AI-built prototype meets real users picks up the operational half — and its sixth break governs everything above. Until somebody has written down what a wrong answer is, no interface can tell a user which outputs deserve a second look. A fixed-fee AI readiness assessment is the shortest route to that written answer.