What makes an AI feature expensive to run

An AI feature's running cost is set by five decisions taken at design time, and the price per token is not one of them. Where the stable part of the prompt sits, because everything after the first thing that changes has to be paid for again. Whether the user has to wait, which is a product decision worth half the bill. Whether context is carried or retrieved, because carried context is re-sent and re-billed on every turn. Which model runs which step, rather than which model runs the product. And how many turns a task takes, which nothing in the code caps unless someone decides to. Cheaper tokens do not produce a cheaper invoice, because all five of these multiply token count, and only one of them is visible on a pricing page.

The reason this feels wrong to everybody who has priced software before is structural. An AI feature's cost scales with engagement rather than seats, which inverts the assumption every SaaS pricing model is built on: that a heavy user and a light user cost roughly the same to serve, so usage is pure margin. Here the enthusiastic customer is the expensive one. The size of that bill is a separate question — what it costs to run an AI agent in production, per month covers the ranges, and outcome-based AI pricing covers the commercial structures that survive it. This piece is about the engineering decisions that set the multiplier before either question can be asked sensibly.

Lever one: where the stable part of the prompt sits. Caching is usually described as an optimisation and is really an ordering constraint. Anthropic's documentation is explicit that "prompt caching references the entire prompt — tools, system, and messages (in that order) up to and including the block designated with cache_control," and that the cache follows that hierarchy, where "changes at each level invalidate that level and all subsequent levels" (Anthropic, Prompt caching). Read that as a design rule rather than an API detail: one dynamic value placed early — a timestamp, the user's name, today's date at the top of the system prompt — invalidates every token after it, on every request, forever.

The economics attached to that ordering are published as multipliers. Anthropic lists five-minute cache writes at 1.25 times the base input token price, one-hour writes at 2 times, and cache reads at 0.1 times, with the cache refreshed at no additional cost each time the cached content is used. There is also a floor: the minimum cacheable prefix runs from 512 to 4,096 tokens depending on the model, so a prompt assembled from a scatter of small, individually varying fragments cannot be cached at all. The documentation's own instruction is the one to hand an engineer: "Place cache_control on the last block whose prefix is identical across the requests you want to share a cache." Doing that is an afternoon. Discovering it after launch means re-architecting how the prompt is assembled, which is not.

Lever two: does the user actually have to wait? This is the largest single saving available and it is a product decision, not a technical one. Anthropic's Message Batches API processes requests asynchronously "with most batches finishing in less than 1 hour while reducing costs by 50% and increasing throughput," with results available "when all messages have completed or after 24 hours, whichever comes first," and a batch "limited to either 100,000 Message requests or 256 MB in size, whichever is reached first" (Anthropic, Batch processing). Batches expire if processing does not complete inside 24 hours.

The engineering question is therefore narrow and answerable: for each AI call in the product, is a human sitting there? For a great many of them nobody is. Classification, enrichment, summarising records that will be read tomorrow, scoring a backlog, generating the copy for pages nobody has requested yet — all of it was built synchronously because that was the shape of the first prototype, and it stays synchronous because no one revisited it. A generative catalog-enrichment pipeline we built for a marketplace operator is the type case: seller listings are enriched per category with structured output validation, low-confidence items queue for human review and everything above threshold is approved automatically. No seller is watching a spinner while that happens, and a feature nobody is watching does not need a synchronous price.

Lever three: carried context versus retrieved context. The cheapest change a developer can make is to paste more into the prompt, and it is the most expensive one to run, because it is paid on every request rather than once. Anthropic's documentation on context windows is blunt about the accounting: "Everything in the request counts toward the context window: the system prompt, every message in messages (including tool results, images, and documents), and your tool definitions," and across turns "each user message and assistant response accumulates within the context window, and previous turns are preserved completely" (Anthropic, Context windows). A twenty-turn conversation does not cost twenty times a single turn. It costs the running sum.

Two lines from the same page kill the two most common consolations. The first: "Cached prompt prefixes still occupy the context window — prompt caching changes what you pay for those tokens, not whether they count." Caching is a discount, not a capacity increase. The second is the one that matters more, because it turns a cost argument into a quality argument: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot. This makes curating what's in context just as important as how much space is available." Stuffing the window is not a cheap way to buy accuracy that happens to cost money. Past a point it buys less accuracy and costs more. Retrieval — fetching the three relevant records rather than carrying four hundred — is cheaper and better at the same time, which is a rare combination and the reason RAG vs. fine-tuning vs. prompt engineering is a design decision rather than a preference.

Lever four: which model runs which step, not which model runs the product. Most AI features are pipelines wearing a single name. A document workflow typically has one step that needs genuine judgement and four that need extraction, classification, validation and formatting. Choosing one model for all five is a procurement decision dressed as an architecture one, and it prices every cheap step at the rate of the expensive one. Routing per step is straightforward to build and has one hard prerequisite: you need a scored way to tell whether the cheaper model is good enough at that specific step, which means an evaluation set per step rather than a general impression of the model. Without it, routing is guessing with extra deployment risk — which is why AI evaluation and observability is the thing that makes this lever available rather than a separate line item you can defer.

Lever five: how many turns the task takes. Nothing caps this unless somebody decides to. An agent that retries a failed tool call, re-reads its own output, or asks itself whether it is confident, pays for the whole accumulated context again on every pass — and by lever three, that accumulated context is at its largest exactly when the task is going worst. Three controls are ordinary engineering and each takes about a day: an explicit ceiling on turns per task, a token ceiling per request, and a stop condition that is written down rather than emergent. The related decision — where the confidence threshold sits that sends work to a human instead of another model pass — is covered in the run-cost piece above, and it is the same dial seen from the finance end.

None of the five can be managed without attribution, and attribution is the part almost everyone skips. The data exists from the first request: every response reports what it consumed, and when caching is in use the input count is split into ordinary input tokens, cache reads and cache writes, so the cache hit rate is directly observable rather than inferred. Almost nobody records those numbers against a tenant, a feature or a completed task. The result is a monthly invoice with one number on it and no way to answer which customer, which capability, or which code path produced it. Cost per completed task, tracked per capability, is the metric that makes every lever above testable, and it belongs in the same telemetry as latency and error rate rather than in a spreadsheet somebody rebuilds each quarter. That plumbing is ordinary AI data infrastructure work, and it is cheap to build first and painful to retrofit.

So: who pays for it? Internally, the answer has to be a named person before launch rather than a discovery afterwards, because the cost of an engagement-scaled feature is not a finance problem that arrives at month end — it is a product decision about what a single user is allowed to consume. Two things need answering while the feature is still being designed. What is the ceiling for one account, in tasks or tokens rather than in currency? And what happens when it is reached: degrade to a cheaper path, queue the work, or refuse and say so? A product with no answer to the second question has chosen the fourth option, which is to keep serving and find out later.

The commercial half of the question — what a customer is charged, and against what definition of a delivered outcome — is genuinely a different discipline, and it is treated properly in the outcome-based pricing piece linked above. What is worth saying here is only that the two are coupled: a price per resolved ticket is unquotable until somebody can say what a resolved ticket costs to produce, including the retries and the escalations. Firms that publish outcome pricing confidently have usually done the attribution work first. It is not the other way round.

None of this is exotic. Ordering a prompt so its stable half comes first, moving unattended work off the synchronous path, retrieving instead of carrying, routing per step, capping the loop, and recording consumption per task are the same class of decision as indexing a database or putting a queue in front of a slow dependency — ordinary AI development services engineering, taken early because each one is an afternoon at design time and a rebuild afterwards. The reason they get skipped is not difficulty. It is that none of them makes the demo better, and the invoice that would have argued for them does not exist yet.

If a vendor is quoting you for an AI feature, the useful question is not what it will cost to run. It is which of these five decisions they have already made, and whether they can show you the per-task consumption number from something they have already shipped. That is the same discipline behind our own AI development pricing: a build cost and a run cost quoted separately, because the second one is set by architecture and the first one is not.