Stop the retry that landed twice

The same customer action has landed twice. Tonight, in this order. Stop the caller that can retry: the button, the queue, the scheduled job, and the agent tool, whichever of them can fire the action again. Keep both results, and keep the key that was supposed to make them one result, or write down that there was no key. Have a person reverse the result the customer did not ask for, in the system that took the money or wrote the row, using that system's own correction. Then count whether a second customer was hit by the same loop. If you cannot tell the two results apart, you are not reversing a duplicate. You are deleting at random, and asking the model to "refund the extra one" is how you get a third.

This is the hour after a correct action happened twice. It is not the hour after a wrong value was written. A wrong address, a wrong amount, a row that should not exist at all, is a different incident, and the first move there is to stop the tool that can write. A retry that refunds twice is the case that incident deliberately leaves to another night: the value may have been right, and the failure is that it was applied again. It is not the design review where you wish the write had carried a key. The key is a decision you wanted before launch. Tonight the second charge, the second return, or the second email has already left. Teams lose the night by holding one call that tries to be the design review, the wrong-row reversal, and this count. A bad sentence in a chat window does not mean a second charge. A second charge does not wait for you to decide the prompt was the cause.

The property you are missing has a definition, and it comes from a company that runs the retries itself. Malcolm Featonby, in the Amazon Builders' Library, defines an idempotent operation as "one where a request can be retransmitted or retried with no additional side effects." The dilemma he uses is a customer who wanted "at most one" EC2 instance. The call times out. "Simply retrying the request could result in multiple workloads." Amazon's preferred contract is "a unique caller-provided client request identifier." "Requests from the same caller with the same client request identifier can be considered duplicate requests." Recording that identifier and "all mutating operations related to servicing the request" has to be "an atomic, consistent, isolated, and durable (ACID) operation," an "all or nothing" process, so you do not record the token and fail to create the resource, or create the resource and fail to record the token. He also separates a response that is merely free of a second side effect from a response the caller can treat as the same outcome. Returning "the resource already exists" can be side-effect free for the service and still change what the caller does next, which he calls a side effect from the client's perspective (Featonby, Making retries safe with idempotent APIs, Amazon Builders' Library).

Use that as a definition, and stop before you use it as a comfort. The article is about AWS API design, with EC2 and a client token as the worked example. It is not a description of your refund route, your shopping cart, or the tool your agent called. A token you have not implemented is not present in the log because the essay says it should be. "At most one" is the customer's intent in his scenario. Your customer may have clicked twice on purpose, or a queue may have delivered one message twice, and those are different facts you still have to separate tonight. The ACID sentence is the reason a repair that writes a new row "so we remember the duplicate" can itself be a third mutation if it is not the correction the product already has. Do not add a table in the middle of the incident in order to feel closer to the essay. Keep the two results you already have. The essay's useful order is the one you can apply without deploying anything: same caller, same identifier, no additional side effect. If you have no identifier, you do not get to skip the rest of the night. You get a harder comparison, and you say so.

A second primary source is a simulation of retries under contention, and its author tells you it is a simulation. Marc Brooker, on the AWS Architecture Blog, modeled optimistic concurrency with "a small simulator," "a network with delay (and variance in delay), against a remote database," "delay with a mean of 10ms and variance of 4ms." With N clients contending, "the total amount of work done by the system increases with N2," because every client competes in every round. Adding jitter to the backoff: "In the case with 100 contending clients, we've reduced our call count by more than half." The post does not print a tighter percentage than "more than half," which is more than 50% of the call count in that one simulated case. "All of the graphs and numbers from this post were generated using a simple simulation of OCC behavior" (Brooker, Exponential Backoff and Jitter, 4 March 2015). A note added to the same page in May 2023 says "Most AWS SDKs now support exponential backoff and jitter as part of their retry behavior when using standard or adaptive modes." One hundred clients, a 10 millisecond mean delay, and a cut of more than half are properties of that simulator. They are not a measurement of payment retries, they are not your error rate, and they do not make a charge idempotent. Jitter spreads calls out. It does not decide which of two charges the customer asked for. A team that adds randomness to the backoff and then goes to sleep has improved a graph from 2015 and left the second charge in place.

Stop the caller that retries. Disable the job, the queue consumer, the webhook replay, and the agent tool that can invoke the action, before you touch the customer's record. A retry that is still armed will treat your reversal as a failure and try again, which is how a careful person creates the third effect the opening paragraph warns you about. If you cannot stop the retry without stopping the whole product, stop the whole product for the path that acts, and leave a read up only when the read cannot call the write. When to turn a live AI feature off is the rule for a feature whose next answer can hurt someone. Use it if the next retry is that answer. Do not use it as a substitute for naming the two results. A feature can stay up, serving reads, while the action path is off. If your system has no such split, tonight's honest move is the wider off switch, and the lack of a split is a fact you write down rather than a reason to leave the retry running.

Keep both results before anyone reverses one. The two identifiers in the system of record, the amounts, the times, the customer, and the idempotency key if a key was sent. If the key was not sent, write "no key" in the same note. Editing one of the results first destroys the comparison. What you keep after a wrong answer, when the problem is the answer a person saw, is a different list. What to keep after a wrong AI answer is that list. Use it if a customer was also shown a false sentence. Do not let it delay the note that names the two charges. A screenshot of one receipt is not a count. The log of the caller, across the window, is how you tell one duplicate from a loop that touched a hundred orders. Look at the log before you announce that the incident is a single customer.

How you tell a retry from a second real request, with the evidence you actually have. Same caller and same client request identifier, in Featonby's sense, is the clean case: the second delivery is the same intent, and reversing one of the two effects puts you back to a single effect. No identifier, and you are inferring. Same customer, same order, same amount, overlapping times, and no second gesture you can see in the product, is a timeout or a redelivery until a person finds a gesture that contradicts it. Two clicks, seconds apart, both intentional, can still be one charge if the product promised one, and the note should say you reversed a double submit rather than a lost response. Those notes are different because the fix is different. A double submit wants the button disabled after the first success. A lost response wants the key. Writing "duplicate" on both trains the next incident to apply the wrong repair. If you cannot decide, say you cannot decide, and do not reverse a result you cannot tell from the one the customer wanted. Waiting with both results visible is better than a confident reversal of the wrong one.

A person reverses one result. Open the system that took the money or created the record, as a user who is allowed to refund, void, or cancel, and undo the extra effect. Keep the void. Do not paste the two receipts into a prompt and ask the model to pick one and call the refund tool. That request is another call, and if the tool is still armed it is another side effect, which is the failure Featonby defined as the thing idempotency exists to prevent. The design you wanted, a key derived from the intent so the second delivery is recognized, is what breaks when AI meets a system of record. That article is the review you hold before the tool exists. Tonight the second effect exists. A design discussion does not refund it, and a second agent call is not a compensating transaction. It is another try. The wrong-value version of this hour, where the row should not have been written at all, is when the agent already wrote the row. If the amount itself is wrong, you are in that article. If the amount is right and present twice, you are in this one. Write down which one you are in before you change a record.

Then count, and do not widen the credential to go faster. One customer is an incident you can finish. A loop is the same caller delivering the same class of action across many customers after you thought you had stopped it. The count comes from the log of that caller, not from the person who wrote in. If the count is larger than the one complaint, the reversal is a list, and the list is work for a person with a query. If you cannot produce the list, say the incident is not contained. Contained means you know which effects, not that the chat session feels over. The identity that can refund every charge in the system is a wider grant than the reversal needs. Handing it to the agent "just for the cleanup" repeats the pattern that let an unattended caller act in the first place. A person with a scoped login is slower. Slow is how you avoid a fourth effect.

What an action you can stop looks like, when one already exists, is a write a person can still refuse. A support agent for a DTC ecommerce brand can look up an order, start a return, or change an address, and those actions sit behind permission guardrails, with a person receiving the conversation when confidence is low. That case is not a story about a double charge. Nothing in the description is a headcount, a duration, or a result figure. The shape is the point: there is a switch on the action that is not the whole product, so "stop the retry" has somewhere to land. If the only credential that can refund is the one the agent holds, tonight's honest move is to take that credential away, not to hope the next prompt notices it has already refunded.

There is a version of this night where calling in a rescue team is the wrong purchase. The second effect has not happened. You are looking at a design, a demo, or a log from a laptop, and the question is whether the write should have carried a key. That is the design review, and it belongs before launch. Nothing is in production, so there is no charge to reverse and no queue to disarm. The value that landed once was wrong. That is the wrong-row incident, and reversing "the duplicate" will not fix a value that was bad on the first write. A customer has already seen another customer's data. A retry count does not close a leak. You never named the metric the feature had to move, so "it charged twice" is the only fact you have. A rescue priced against a metric you have not named is a template, and you should not buy one.

When the second effect is real and the caller can still do it again, the engagement that fits starts by saying what is salvageable. An AI pilot failed to reach production is scoped after an audit, not before it, and the audit is the first one to two weeks of an engagement that runs eight to twelve. The build is unpaid if the metric you agreed does not move. Rollout, when it happens, sits behind a human-review gate. If you do not yet know whether the pilot can be saved, a two-week AI readiness assessment hands you the findings either way. The free production-readiness diagnostic is the short way to see whether you are looking at a duplicate effect, a wrong value, or a system that has not reached a customer. Those are the terms. They are not a promise that every double click is a rescue, and they are not a rate. Brooker's more than 50% is a simulated call count. It is not your duplicate rate, and it will not appear on the invoice for the engagement.

The number the rebuild has to move, once you are past the alarm, is the same discipline as AI evaluation and observability: a metric you can miss, checked on a cadence a customer will notice. "Charges that landed twice" can be that metric only after you can count them from a log rather than from a complaint. Before midnight, do the four steps in order and write down which one you reached. Stop the retry. Keep both results. Have a person reverse the extra one. Count. Do not ask the model to pick the receipt. The two results are how you notice. They are not the repair, and a second call is not a key you forgot to send the first time.