When to leave a wrong AI feature on

Leave a live AI feature on, behind a person, only when three things are true together. A named person can disregard, override, or reverse the output before a customer is stuck with it. That same person can stop the system into a safe state, rather than file a ticket after the fact. And you have watched them reject a confident wrong answer, on a case where their own answer was right before the model spoke. If any one of the three is missing, turn the feature off. A reviewer who has never been shown a wrong answer they already knew was wrong is not oversight. They are a signature on the model's mistake.

The third test exists because wrong advice moves people who were already correct. Goddard, Roudsari and Wyatt reviewed automation bias — over-reliance on automation — with the evidence centred on clinical decision support (Goddard, Roudsari & Wyatt, Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association, 2012). From 13,821 papers they kept 74. Four of those, similar in method, compared decisions made after incorrect advice with a control group that received none. Pooled, the risk ratio was 1.26 (95% confidence interval 1.11 to 1.44). The authors' reading of that ratio is that erroneous advice increased the risk of an incorrect decision by 26%. They call the pool a small, indicative meta-analysis. The systems were non-interruptive, the advice was text, and the errors counted were commission errors: acting on bad advice, which they treat as the clearer form of the bias. This is not the error rate of a chatbot in 2026. It is the pull, measured on clinicians, of a wrong suggestion on a person who might otherwise have stayed right.

They also publish a cleaner count than the pooled ratio. A negative consultation is a decision that was correct before the system spoke and wrong after it. Across four prospective studies with similar designs, that share ran from 6% to 11% of cases. Friedman and colleagues, one study inside that range, found the other column too: the correct diagnosis appeared after consultation in 12% of cases, and a correct unaided answer was abandoned in 6%, a net gain of 6%. Oversight can be worth having. In that one study the net is positive. A team that counts only the saves will leave the feature on and never record the share it made worse. If you have no drill that can show you a negative consultation, you are reading one column of a two-column result.

The statute that writes the same duties down is narrower than a sales deck implies. Article 14 of Regulation (EU) 2024/1689, the Artificial Intelligence Act, applies to high-risk AI systems. It does not, by its own scope, apply to a typical support assistant sold to a US company. Nothing here determines whether your feature is high-risk, and nothing here is legal advice. Where the article does apply, it is the clearest public statement of what a person in the loop has to be able to do. High-risk systems must be built so they "can be effectively overseen by natural persons during the period in which they are in use" — the "they" in that sentence is the system. That person must be able "to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)," and the regulation names the case: systems "used to provide information or recommendations for decisions to be taken by natural persons." They must be able "to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output." And they must be able "to intervene in the operation of the high-risk AI system or interrupt the system through a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state" (European Union, Regulation (EU) 2024/1689, Article 14(1) and Article 14(4)(b), (d) and (e); adopted 13 June 2024, published in the Official Journal on 12 July 2024). Override and stop are different controls. Override fixes one answer. Stop halts the system. A review queue that can edit a draft and cannot halt the feature fails the moment the failure is no longer a single answer.

Do not import the two-person rule into a product it does not cover. Article 14(5) requires separate verification by at least two competent people for high-risk biometric identification under Annex III, point 1(a), and the regulation itself sets that requirement aside for specified law-enforcement, migration, border-control, and asylum uses where the law treats two-person verification as disproportionate. A support agent drafting a refund is not that system. Adding a second reviewer because a statute you are not under required two people, on a different product, turns oversight into delay. One named person who can reverse and stop, and who has passed the drill, is the bar this article will defend. Two people who both defer to the model are a slower way to ship the same wrong answer.

Reverse has to happen before the effect, or the word is a courtesy. If the model has already emailed the customer, posted the reply, or moved the money, the reviewer is not overriding an output. They are apologising for one. That fails the "in any particular situation" duty, because the decision was no longer available when they saw it. The safe state is the state in which the feature does nothing on the next request, not the state in which it acts and a person is told afterwards. Designing an interface for a system that is sometimes wrong is where a confirmation step earns its friction or wastes it: the person can evaluate the thing, the evidence is on the screen, and the action is expensive to undo. This article is the decision that comes first. If you cannot put the evidence on the screen, and you cannot undo the action, do not draw a confirmation step. Turn the feature off until both are true.

The drill is small, and it is the whole of the third test. Take a case a competent person on your team gets right with the model disconnected. Then show them the model's confident wrong answer, in the interface a customer would cause, and ask them to accept, edit, or reject. Write down which they did. Ten cases will show you a pattern. Ten cases will not support a rate you can put next to 6%, so do not publish one. You are looking for one failure: they accept because the model was fluent. If that happens, you do not have a person in the loop. You have the negative consultation, on your own product, and the feature comes off until the interface or the prompt changes and you run the ten again. The evaluation suite scores the model. It does not score the person. Both have to pass. A gate that is only a model score, with a human rubber stamp, is the configuration the 26% describes. A gate that errors and then allows the action is a different failure: when the guard fails open. A person cannot reject an action the catch block already took.

The shape that can pass is ordinary, and it is visible in work we can name. A support agent for a DTC ecommerce brand escalates on confidence and hands the person the full conversation, not a one-line summary, and changes are scored for resolution accuracy before they reach a customer. The context is what makes a refusal possible. An escalation that arrives as "the model said refund" is not a decision the person can evaluate. That case is not one of the four clinical studies, and this paragraph is not claiming Goddard's percentages for it. Healthcare advice and a commerce support draft are different jobs. The order transfers: measure the model, then let a person who can see the case refuse the answer, and do not treat the refusal as optional decoration on a feature you were always going to leave on.

Three further cases where the feature comes off even if a person can click reject. The failure is on the write path and the reviewer sits downstream of it, so reject does not unsend. You cannot name the safe state: what the product does, on the next request, once the feature is stopped. "We will look at it in the morning" is not a safe state. And the only person who can reject is the person who built the feature, which means the gate is open whenever they are in a meeting. A gate with one holder is a gate with office hours. Name a second holder who has also passed the drill, or turn the feature off. That is staffing. It is not the biometric two-person rule, and it should not be dressed up as one.

Do not buy a production engagement to avoid making this call. If you have not started a pilot, there is nothing to leave on. The fitting purchase is a two-week AI readiness assessment, and the findings are yours whether or not a line of the roadmap is built with us. If the goal is undefined, a reviewer cannot know a wrong answer, and no gate repairs an undefined goal. If the feature has no users yet, turning it off costs an afternoon, and a production engagement does not. Where the feature is already live, customers can already see the wrong answer, and the three tests fail, the work is to build a gate that meets them or to remove the feature, against one number you agreed before the work started. That is what an AI pilot failed to reach production engagement is: an audit in weeks one and two, then evaluations, guardrails, and a rollout behind a human-review gate. If the agreed metric does not move, you do not pay for the build. The gate in that sentence is the one defined above. A queue that cannot reverse, cannot stop, and has never been drilled is a sentence in a proposal, and it will not move the metric.

There is a version of "leave it on" that is honest and narrow. The three tests pass. You have seen at least one negative consultation in the drill and you changed the screen until that person stayed with their own correct answer. The safe state is written down: the next request gets the old path, a human, or a refusal, and you can demonstrate it. The model is scored on a set you control, and the person is scored on the drill, and neither score is allowed to stand in for the other. In that window the Friedman's-study shape is the relevant one: a real save column, a real harm column, and a net you are willing to own. Outside that window, another week of "a human checks it" is how the harm column grows while everyone reports the saves.

Run the three questions before the next release, on a Monday, and write the answers where the on-call can find them. Can this person undo what the model just did, before the customer has it? Can they stop the feature into a state you can describe without inventing it? Have they rejected a confident wrong answer in the last month, on a case they got right alone? Three yeses, and the feature may stay on. A missing yes is a reason to turn it off this week, before the next customer becomes the drill you skipped. The free production-readiness diagnostic will name likely failure modes from a description of what you shipped, and it will say where it is inferring. It will not sit the drill. AI evaluation and observability is the model's half of the same gate. The person's half is an afternoon with ten cases and a wrong answer you already know is wrong.