What AI changes for an internal platform team

An internal platform team has always had two kinds of consumer: people, and the build system. An agent is a third, and it violates four assumptions the platform was built on. It is not one caller — a single user action fans out into a loop of reads, retries and tool calls nobody declared in advance. Its requests are not the same size: one costs a cache hit, the next reads a hundred pages. It cannot be rate-limited per seat, because the scarce resource sits upstream and is shared across every team. And it needs an identity per task rather than per service, because the authority it should carry belongs to the person who asked, not to the deployment it runs in. None of the four is an AI problem. All four are platform problems, and they arrive whether or not the platform team was consulted.

That last part is the organisational shape of it. The AI work is commissioned elsewhere — a product team, an innovation function, a vendor — and the platform team finds out when an unfamiliar service account starts issuing a thousand requests a minute against an internal API that has served forty. The useful move is to treat the agent as a consumer class with a written contract, the way the platform already treats a mobile client or a batch job, before the first incident writes that contract for you.

Start with the resource, because it cannot be engineered around. Anthropic's published rate limits are a fair picture of how every serious provider meters this, and the shape is unfamiliar to anyone used to per-user quotas. Limits are measured "in requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM) for each model class," the documentation states plainly that "Limits are set at the organization level," and replenishment is continuous rather than periodic: "your capacity is continuously replenished up to your maximum limit, rather than being reset at fixed intervals." Exceed one and "you will get a 429 error describing which rate limit was exceeded, along with a retry-after header indicating how long to wait" (Anthropic, Rate limits, Claude developer documentation).

Read that as a platform engineer and the consequence is immediate: the scarcest resource in your AI estate is organisation-wide and unallocated. Three teams with three agents on one account share one bucket, and whoever ships a background summarisation job on a Thursday can starve the customer-facing assistant without touching its code. There is no per-team fairness in the upstream limit; if you want it, the platform provides it. The same documentation points at the mechanism, noting that per-workspace limits protect other workspaces "from potential overuse" and ensure "a more equitable distribution of resources across your Organization."

The other half of that lever is one no single agent team can pull well, because its value rises with how much context is shared. On most Claude models "only uncached input tokens count toward your ITPM rate limits," and the documentation gives the arithmetic: "With a 2,000,000 ITPM limit and an 80% cache hit rate, you could effectively process 10,000,000 total input tokens per minute (2M uncached + 8M cached), because cached tokens don't count toward your rate limit." A cache hit rate is therefore a capacity decision rather than an optimisation — and system instructions, tool definitions and the reference documents several agents read are exactly what a platform can standardise and one team cannot. Owning the shared prefix does not look like AI work at all.

The next assumption to retire is that request count models load. Google's site reliability engineering practice arrived at this the hard way for ordinary services, and an agent makes it worse by an order of magnitude. "Different queries can have vastly different resource requirements," the SRE book observes, and it is blunt about the consequence: "We learned this lesson the hard way: modeling capacity as 'queries per second' or using static features of the requests that are believed to be a proxy for the resources they consume (e.g., 'how many keys are the requests reading') often makes for a poor metric" (Google, Handling Overload, Site Reliability Engineering). Between an agent request served from a warm cache and one that pulls a hundred documents into context, the cost ratio is not two or three. Agent capacity has to be modelled in the resource actually consumed; a dashboard counting calls per minute will look calm through the whole of the first outage.

The same chapter supplies the control that matters most and that almost no internal platform has: a declared criticality on every request. Google attaches one of four values, from CRITICAL_PLUS — "Reserved for the most critical requests, those that will result in serious user-visible impact if they fail" — down to SHEDDABLE, "Traffic for which frequent partial unavailability and occasional full unavailability is expected," and reports that "We found that four values were sufficiently robust to model almost every service." Agent traffic sorts into those four unusually cleanly, which is the good news buried in this problem. A user waiting on an answer is critical; the same agent's speculative pre-fetch, overnight re-indexing and evaluation runs are not, and they are usually most of the volume. A platform that cannot tell them apart protects the worst case by over-provisioning for all of it.

Then there is the failure mode that turns a slow afternoon into an outage, and agents are close to purpose-built for it. The SRE book's account of retry amplification is worth quoting because the arithmetic is the argument: "avoid amplifying retries by issuing retries at multiple levels: a single request at the highest layer may produce a number of attempts as large as the product of the number of attempts at each layer to the lowest layer. If the database can't service requests because it's overloaded, and the backend, frontend, and JavaScript layers all issue 3 retries (4 attempts), then a single user action may create 64 attempts (4^3) on the database" (Google, Addressing Cascading Failures, Site Reliability Engineering).

An agent inserts at least two more layers into that product, and both retry by default. The provider SDK retries a 429. The agent loop retries a tool call that errored, often by reasoning its way into calling it again with a slightly different argument, which no retry counter recognises as a retry. Add the platform's client library and the service's own retries and the exponent grows rather than the base. The remedies in the same chapter are unglamorous and they work: "Always use randomized exponential backoff when scheduling retries," "Limit retries per request," and — the one belonging in the platform rather than in each agent — "Consider having a server-wide retry budget. For example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don't retry; just fail the request." Client-side throttling is the other half: clients keep issuing requests "until requests is K times as large as accepts," and Google reports that "We generally prefer the 2x multiplier." Ship both in the platform's SDK and every agent inherits them; leave them to each team and you get five implementations, three of them a bare loop.

There is a subtler version of the retry problem that no backoff fixes. A retried write is a duplicated write unless the endpoint says otherwise. Human clients retry rarely and a person usually notices the second invoice; an agent retries confidently and does not. Every write path an agent can reach therefore needs an idempotency key, and the platform is the right place to make that mandatory — a header the gateway requires, not a convention each service half-implements.

The fourth assumption is the identity one, and it is where the platform's existing model is most likely to be quietly wrong. The usual arrangement gives a workload one service account with a fixed role, which was fine while the workload did the same thing for everybody. An agent does a different thing for each requester, and the authority it needs is the requester's. NIST states the underlying shift precisely: "A key paradigm shift in ZTAs is the change in focus from security controls based on segmentation and isolation using network parameters (e.g., Internet Protocol (IP) addresses, subnets, perimeter) to identities," and from an application security point of view "this requires authentication and authorization policies based on application and service identities in addition to the underlying network parameters and user identities." It is explicit that this takes infrastructure rather than discipline, naming "a platform that consists of Application Programming Interface (API) gateways, sidecar proxies, and application identity infrastructures (e.g., Secure Production Identity Framework for Everyone [SPIFFE])" (NIST, A Zero Trust Architecture Model for Access Control in Cloud-Native Applications in Multi-Cloud Environments, SP 800-207A, September 2023).

That is the platform team's job description for agents, written by a standards body before the question became urgent. The agent needs a workload identity so the platform knows which agent is calling, and it must carry the requesting user's authority so the record system can apply the permissions it already has. Collapse the two into one broad service account and the agent's reach becomes the union of every user's — the condition under which a prompt injection stops being an embarrassment and becomes a breach. What breaks when AI meets a system of record is the same territory from the record system's side.

One more thing changes: the observability contract. A platform's logs answer which service called, with what arguments, and what came back. For an agent that reconstructs nothing, because two identical requests can come from two prompt versions against two model versions and behave differently. Four fields have to travel with the call — the agent name, the prompt or configuration version, the model identifier, and the identity of the human it acted for. They are cheap to add at the gateway on day one and archaeologically expensive after the first incident. Who owns an AI system after it ships covers who reads those signals once they exist.

None of this is a large programme. It fits on one page: a declared criticality on every agent request, a published per-team allocation of the shared model quota, a retry budget and client-side throttling shipped in the SDK rather than left to callers, idempotency keys required on every write path an agent can reach, workload identity plus delegated user authority instead of one broad service account, and four extra fields in the request log. Six items the team already knows how to build, none of which requires anybody to become an AI engineer.

What it does require is being asked in time. The work above has to exist before the second agent, because the first survives on goodwill and manual attention and the second is where the shared quota, the retry product and the service account stop being theoretical. What makes an AI feature expensive to run is the cost half of this conversation; the reliability half is here. A platform consolidation we ran for Cove, a US home-security device ecosystem, is the ordinary version of the discipline — three disconnected systems rebuilt into one operator experience on Kubernetes with blue/green deploys and full observability — and none of it was added for AI. It is what a platform needs to absorb a new class of consumer without renegotiating everything.

The framing that helps most is the least dramatic. The agent is not a new kind of software to be governed. It is a new kind of customer: one that never sleeps, never reads the documentation, retries without being asked, and cannot file a ticket when something is wrong. Platform teams have onboarded difficult customers before, and the tools are already on the shelf — quotas, criticality, budgets, identities and logs. The enterprise AI solutions that scale past their first workflow are the ones where the platform team met that customer early enough to write the contract, and AI evaluation and observability is the part of it that stays true after launch.