How to evaluate an AI development company

Six questions separate an AI engineering practice from a web shop that added a model call last quarter, and each has an answer you can check rather than believe. How do you evaluate output quality, and what was the score at your last two releases? What happens when the model changes underneath you — which version is pinned, and who pays for the migration? Whose identity does the system act under when it writes to one of my records? What exactly do you hand over? Who owns the prompts, the evaluation data, and anything fine-tuned on my data? And which party is responsible for each practice, in writing? A firm that has run a system in production answers all six from memory. A firm that has only demoed one answers in adjectives.

The category makes this hard because the label is free. Any shop can describe itself as offering AI development services, and from outside, a competent build and a wrapper look identical: both demo well, both call the same model. Nor does tool use distinguish anyone now — Stack Overflow's 2025 Developer Survey found 84% of respondents using or planning to use AI tools in their development process, up from 76% the year before (Stack Overflow, 2025 Developer Survey: AI). When four in five developers use the technology, saying you use it carries no information. What separates vendors is everything after the demo works.

One: how do you know an output is good? This filters hardest, because a real answer has artefacts attached and a bluff cannot produce them. Ask for four: the golden dataset — real cases, drawn from data like yours, with the correct answer recorded for each; the scoring method per case type, and who decided what counts as correct; the score at the last two releases; and the rule that blocks a deploy. If there is no evaluation harness there is no engineering, only a demo with a subscription. Three answers should worry you: "we review outputs manually before release", "we use the strongest available model" and "the client tells us if something looks wrong" — each describes a process that learns about failure from a complaint.

Manual review fails on precisely the failure mode this technology has. In the same survey, the biggest single frustration, cited by 66% of developers, was "AI solutions that are almost right, but not quite," and 45.2% said debugging AI-generated code takes longer than writing it. Almost-right is invisible to spot-checking, because a spot check looks for something obviously broken. It shows up only against a recorded expectation, which is what a golden dataset is — the substance behind AI evaluation and observability. The harness does not exist to prove the system works; it exists to detect the day it quietly stops.

So ask the follow-up too: who defines a wrong answer, and where is that definition written down? A wrong answer in your business is a business judgement — a refund that should have escalated, a record that must not surface in a search result — and a vendor who never makes you write it down will supply their own by accident, in code. In one production support agent we built, the evaluation suite measured resolution accuracy before and after every release and escalation to a human was confidence-based rather than keyword-based (AI support agent).

Two: what happens when the model changes underneath you? Models are retired on published dates; this is a scheduling fact, not a vague risk. Anthropic publishes a four-state lifecycle — Active, Legacy, Deprecated, Retired — states that "requests to retired models will fail," and commits to "at least 60 days' notice before model retirement for publicly released models." The dates are recent and real: Claude Opus 4.1 was deprecated on 5 June 2026 and retired on 5 August 2026; Claude Sonnet 3.7 was deprecated on 28 October 2025 and retired on 19 February 2026 (Anthropic, Model deprecations, Claude Platform documentation).

Two details on that page are what a good vendor raises before you do. Retirement dates differ by platform, because partner-operated platforms such as Amazon Bedrock and Google Cloud "set their own retirement schedules." And parameters deprecate too: temperature, top_p and top_k now return a 400 error when set to a non-default value on Claude Opus 4.7 and later, so a migration undertaken to escape a retirement can itself break on a parameter. Ask where the model identifier lives, whether it is pinned to a dated version, whether the evaluation suite is re-run against the replacement before switching, and whether that work is in the agreement or a change order. Model drift has the mechanics.

Three: whose identity does the system act under? Once an AI system can write to a CRM, an ERP or a ledger, the interesting questions stop being about the model. Whose credential does the write happen under, and can you prove on whose behalf it acted? Does retrieval filter by the requester's entitlements at query time, or has the index flattened permissions the source system was enforcing? NIST's generative-AI profile of its secure development framework makes the logging expectation explicit, recommending that developers "log, monitor, and analyze all inputs and outputs for AI models to detect possible security and performance issues" (NIST, SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models, July 2024). Ask to see the permission model and the audit log, not to hear them described; what breaks when AI meets a system of record is the long version.

Four: what do you actually hand over? NIST's Secure Software Development Framework, published in February 2022, is useful here because it was written with buyers in view: the framework "provides a common vocabulary for secure software development" so that "software purchasers and consumers can also use it to foster communications with suppliers in acquisition processes" (NIST, SP 800-218, Secure Software Development Framework version 1.1, February 2022). Two of its practices belong verbatim in a handover clause: PS.3.1, "securely archive the necessary files and supporting data (e.g., integrity verification information, provenance data) to be retained for each software release," and PS.3.2, "collect, safeguard, maintain, and share provenance data for all components of each software release (e.g., in a software bill of materials [SBOM])."

In purchase-order terms: the repository with its history, not a zip of the final state; an environment a new engineer can reproduce from the repository alone; every credential and cloud account in your name; the prompts under version control; the evaluation dataset and the harness that runs it; a runbook for the failure modes they have actually hit; and a component inventory you can check against a vulnerability feed. Ask for that list before signing — the handover: what you must receive from a development partner sets out the same checklist for a whole engagement.

Five: who owns the prompts, the evaluation data, and anything trained on your data? This is a contract question hiding inside a technical one, and the standards body says so: NIST notes that the framework and its AI profile "only address cybersecurity risk management," and that "there are many other types of risks to AI systems (e.g., data privacy, intellectual property, and bias)." Nobody gets an IP position as a by-product of good security practice. Ask whether the prompts and golden dataset transfer on termination, whether your data may improve anything of theirs, and who holds the embeddings and the index. The same profile asks producers to track the provenance of training, testing and fine-tuning data "and document which data do not have known provenance" — and that second half is the realistic test. A vendor claiming to know every origin has not checked.

Six: which party is responsible for each practice, in writing? The same profile describes the buyer-side move plainly: an acquirer "can establish an agreement with an AI system producer and/or AI model producer that specifies which party is responsible for each practice and task and how each party will attest to its conformance." In practice that is a one-page split — who monitors the system, who holds the provider account and absorbs token spend, who re-runs the evaluation suite after a model change, who is reachable when the pipeline fails on a Saturday, and who patches a dependency found vulnerable. Every one of those has bitten a buyer who assumed it was included.

The phase buyers forget to budget is the one after launch, and the industry has not automated it either: in the Stack Overflow data, 76% of developers said they do not plan to use AI for deployment and monitoring, the most resisted task in the entire workflow, against 69% for project planning. Whatever else AI development services include, running the system is staffed human work, so a proposal that ends at "launch" has a hole in it. Ask what the first ninety days after go-live look like and who does that work.

Three answers should end a conversation. "The model handles that" — offered against any question about permissions, correctness or cost, it means the vendor has not separated what the model does from what the system must guarantee. "We will add evaluations in phase two" — a definition of correct that nobody wrote at the time cannot be retrofitted later. "You will get the code at the end" — handover is a design constraint, not a delivery event. Two quieter signals deserve equal weight: a demo run on their data rather than a sample of yours, and a fixed price quoted without written acceptance criteria.

It would be dishonest to publish this without naming the conflict: we are one of the firms these questions get asked of, so run them on us. Our first engagement is a two-week AI and data readiness assessment at one fixed fee agreed before it starts, and the roadmap is yours whether or not you build with us. We publish no rate card on pricing, deliberately. The criterion we would score lowest on is public reference calls: much of our production work sits under NDA, so against a firm with a wall of public logos we are the weaker answer on that one.

Used well, the six questions take one call. A vendor who has shipped production AI answers faster than you can ask, because each answer is an artefact they already maintain — a dataset, a pinned version, a permission model, a handover package, a responsibility split. A vendor who has not converts every question into a promise about the future. Our AI development services page maps what each part of that work involves, and how to choose an AI agent development company covers the commercial questions that sit alongside the engineering ones.