What breaks when an AI-built prototype meets real users

Six things break when an AI-built prototype meets real users, and code quality is not one of them. Authorisation — permissions live in the interface, not behind the endpoint. Permission inheritance — nobody decided what a manager, a contractor or an agency user inherits. Data migration — the schema that was free to change now holds customer records. Error paths — the happy path was described, so the happy path is what exists. Concurrency and cost — both were invisible at one user and both are linear or worse in traffic. And the one with no answer at all: nobody has written down what a wrong answer is, so nothing can be tested. AI closed the build gap. It left the operate gap exactly where it was.

Be precise about that, because the popular version is both wrong and insulting to anyone who has just shipped something. AI-generated code is not uniquely bad; your prototype works, which is why there are users on it. And the people who use these tools hardest are explicit about where the line actually falls. Stack Overflow's 2025 Developer Survey found 84% of respondents using or planning to use AI tools and 51% of professional developers using them daily — and in the same survey, the tasks they most refuse to hand over are the operational ones: 76% do not plan to use AI for deployment and monitoring, and 58.7% do not plan to use it for committing and reviewing code. Trust runs the same direction: 46% actively distrust AI accuracy against 33% who trust it, and 66% name "AI solutions that are almost right, but not quite" as their biggest frustration (Stack Overflow, 2025 Developer Survey: AI). That is a map, not a verdict — the heaviest users have already sorted software into a half they delegate and a half they do not, and the half they do not is the operate half.

Authorisation is the first break, and it is two breaks wearing one name. The first is that authentication got built and authorisation did not: there is a login, the sidebar hides the admin link from non-admins, and the endpoint behind that link will answer anyone who calls it directly. That is the oldest shortcut in web development rather than an AI artefact, but a generated interface makes it easier than ever to ship, because the visible thing is the thing that was described. The test takes two minutes — sign in as your least-privileged user, open the network tab, re-issue an administrative request with that session. If it succeeds, your permissions are a stylesheet.

The second and harder break is permission inheritance, and almost no prototype has an answer, because with one user the question never came up. Real organisations arrive with structure: a manager who sees their team's records but not their peer's, a contractor scoped to one project, an agency user acting for a client who must not see the agency's other clients, a support agent who can read the order but not the card. Each is a rule about what a role inherits, and inheritance rules compose in ways nobody holds in their head. The failure is not a hole you can see; it appears where two rules, each individually correct, meet.

The discipline for this predates all of it. NIST's Zero Trust Architecture defines zero trust as concepts designed to "minimize uncertainty in enforcing accurate, least privilege per-request access decisions," and its third tenet speaks directly to inheritance: access to individual resources "is granted on a per-session basis," should be "granted with the least privileges needed to complete the task," and "authentication and authorization to one resource will not automatically grant access to a different resource." A fourth adds that "least privilege principles are applied to restrict both visibility and accessibility" (NIST, SP 800-207, Zero Trust Architecture, August 2020). Out of federal language: authorise per request, at the resource, and never let a grant in one place imply a grant in another.

Data migration turns a design decision into a project. With no real data the schema is free: rename a column, drop a table, and the cost is a re-seed. Once customer records exist the same change acquires a migration, a backfill, a rollback plan and a window, plus the risk of doing arithmetic wrong on live balances. Two decisions are worth settling now, because neither is ever cheap to reverse: how tenancy is enforced — a filter clause developers must remember, or a constraint they cannot bypass — and whether anything once true is still recoverable, or whether an update overwrote history in place.

Error paths are the largest volume of missing work and the least visible. A prototype is a specification of the happy path, because the happy path is what got described. Missing is what happens when the payment provider times out after charging the card, the mail service accepts a message and never sends it, a webhook is delivered twice, an upload is half-written. Stripe's API documentation is the clearest short statement of the problem a founder can read: idempotency exists "for safely retrying requests without accidentally performing the same operation twice," and works "by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails" (Stripe, Idempotent requests). Every clause there is a decision your code has to make, and most prototypes have made none of them — invisible until the first double charge.

Concurrency only exists once there are strangers. One person tests one thing at a time; a hundred do not. The read-modify-write pattern that is perfectly correct in a single session — fetch the row, compute the new value in application code, write it back — silently loses one of two simultaneous updates and leaves no trace. Same for anything that checks then acts: two requests both see the last seat, both book it. Stripe names the case in passing, noting that when "the request conflicts with another request that's executing concurrently" the idempotent result is not saved at all. Generated code is prone here not through carelessness but because the answer depends on context no prompt contained: which of your operations must be serialised, and against what.

Cost under load arrives as an invoice rather than an incident, which is how it survives launch reviews. At one user an N+1 query is imperceptible and a model call per keystroke costs pennies; both are linear or worse in traffic, and the AI one is steepest because it bills per token rather than per CPU second you already pay for. Derive two figures before launch — the cost of one user's typical session and of one user's worst session, since the second is what a looping or abusive client produces at volume — then answer what stops it. Per-tenant rate limits, a token ceiling per request and a spend alarm are each an afternoon of work.

And the one nobody has an answer for: nobody has written down what a wrong answer is. If there is any AI inside the product this makes every other break harder, because it removes the possibility of testing. A test needs an expected result, and "summarise this contract" has none until somebody states what an unacceptable summary looks like — omits a termination clause, invents a number, names the wrong party. Until that exists there is nothing to run and no way to know whether last week's prompt change improved the product. Teams here do not have a failing test suite; they have no test suite and no way to build one, and they find out when a customer disputes an output.

NIST's AI Risk Management Framework is honest that it cannot do this part for you. It "does not prescribe risk tolerance," because "risk tolerance and the level of risk that is acceptable to organizations or society are highly contextual and application and use-case specific"; where established guidelines do not exist, "organizations should define reasonable risk tolerance," and only "once tolerance is defined" can the framework manage risk at all (NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023). It observes that "senior leadership sets the tone for risk management within an organization." In a two-person company, senior leadership is you: no engineer and no model can decide on your behalf that a wrong medication dosage and a clumsy email subject line belong in different categories.

The diagnostic sequence, ordered to find the most in the least time. One: sign in as your weakest user and replay a privileged request from the network tab. Two: create two accounts in two tenants and read the first tenant's record with the second tenant's session, by ID, straight against the API. Three: ask the database what enforces isolation — a clause someone must remember, or a constraint they cannot bypass. Four: take last week's real bug report and find the exact request behind it; how long that takes is your incident response time. Five: kill your least important dependency in staging and see whether the product degrades or stops. Six: write the ten worst outputs your AI feature could produce and check whether any test would catch them. Seven: cost one heavy session and multiply by the traffic you want.

What that usually reveals is reassuring in an irritating way: the gaps are ordinary and finite — tenant isolation moved into the database, idempotency keys on the write paths, structured logs with a request ID, a rate limit, a spend ceiling, and an evaluation set built from ten written-down failure definitions. None of it is research; all of it is work somebody has to schedule, and it goes unscheduled because none of it makes the demo better. If you would rather not run the seven steps yourself, our free production-readiness diagnostic reads a description of what you built and returns the failure modes most likely to bite first, in order, and says where it is inferring rather than being told.

The wider version of this argument is why a startup software development company is worth hiring after the prototype exists rather than before: the build is done, and what remains is the operate half. The verification layer specifically is AI evaluation and observability work, and a multi-tenant agency operating system we built needed subscription billing, role-based access across agencies, admins and team members, and an admin-trained AI assistant designed together rather than bolted on, because the permission model and the AI layer constrain each other. Two related pieces go further: what breaks when AI meets a system of record on identity, write semantics and reconciliation, and what a non-technical founder should own, and what to delegate on which of the decisions above are yours rather than an engineer's.