What AI changes in procurement and security review

Four documents change when the software being bought is non-deterministic, and none of them is the statement of work. The acceptance criteria stop describing behaviour that either happens or does not, and start describing a rate measured over a sample. The security questionnaire asks whether a vulnerability has been remediated, and gets back a mitigation that works empirically rather than provably. The change-control clause has to cover a change your vendor did not make and cannot schedule. And the data clause is answered by a published contract you can read this afternoon rather than by a row somebody fills in on your spreadsheet. The rest of the procurement pack survives contact with AI more or less intact, which is exactly why the four that do not tend to go unnoticed until the review stalls.

The reason all four move is the same one, and it is worth stating in the flattest possible terms before anyone reaches for a framework: a system built on a language model is not a function. Deterministic software is bought on three promises — the same input produces the same output, a passed test stays passed, and a patched vulnerability is closed. A probabilistic system offers none of the three in the form procurement is used to receiving them. It offers distributions instead. Distributions are perfectly reviewable. They are just not reviewable by a document designed for the other kind of software.

Start with acceptance, because it is the clause most likely to be copied unchanged from the last SaaS purchase. The model providers' own developer documentation is a better guide here than most procurement templates, and it reads like one. Anthropic's guidance on defining success criteria for a feature built on Claude tells developers that good criteria are measurable — "Use quantitative metrics or well-defined qualitative scales. Numbers provide clarity and scalability, but qualitative measures can be valuable if consistently applied along with quantitative measures" — and its worked example makes the shape unmistakable. The bad version of a safety criterion is "Safe outputs". The good version is "Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter" (Anthropic, Define your success criteria, Claude developer documentation).

The same page adds the part that breaks single-number acceptance clauses: "Most use cases need multidimensional evaluation along several success criteria." Its example for a sentiment classifier names four at once — an F1 score of at least 0.85 on a held-out test set of ten thousand posts, 99.5% of outputs non-toxic, a bound on how severe the errors are allowed to be, and 95% of responses under 200 milliseconds. Translated into a contract, an acceptance clause for this kind of system needs three things a traditional one does not: the metric, the sample it is measured on, and the name of whoever holds that sample. The third decides whether the clause means anything at all. A vendor scoring itself on a set it also assembled has been handed the marking scheme along with the exam.

So the buyer-side work starts before the vendor conversation, not after it. Assemble a few hundred real past cases from the workflow being automated, with the outcome a competent human actually produced, and keep them out of the vendor's hands until the measurement. That set is the single most valuable artefact in the whole procurement, and it is cheap: it is made of work your organisation has already done. It is also the only thing that lets two vendors be compared on the same axis instead of on two different demos. The evaluation suite covers what goes into building one properly, but the procurement-grade version is simpler than the engineering-grade version and can be assembled by the team that owns the workflow.

The security questionnaire breaks for a different reason, and this one is nobody's fault. NIST published the current edition of its adversarial machine learning taxonomy in March 2025, and it is unusually direct about the state of the defences a questionnaire is asking after: "there are no information-theoretic security proofs for the widely used ML algorithms in modern AI systems," and "many of the advances in developing mitigations against different classes of AML attacks tend to be empirical and limited in nature, adopted because they appear to work in practice rather than because they provide information-theoretic security guarantees. Thus, many of these mitigations may themselves be vulnerable to new discoveries and evolutions in attacker techniques" (NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025, March 2025).

Read that against the questionnaire row that says confirm all known vulnerabilities have been remediated, and the mismatch is plain. The honest answer for an entire attack class is a mitigation that appears to work, which is not a box anyone can tick. The same report does give a reviewer something better than a tick: a vocabulary. It organises the attacks into classes that can be named and reasoned about — evasion, poisoning and privacy attacks against predictive systems; poisoning, direct prompting and indirect prompt injection against generative ones — and it devotes a section to the security of agents specifically. It is also candid about its own limits, noting that it "is not intended to provide an exhaustive survey of all available literature on Adversarial ML, which includes more than 11,354 references on arXiv.org since 2021 as of July 2024". A field generating results at that rate is not one where an annual questionnaire cycle establishes very much.

What a review can establish is the boundary, and the boundary is durable in a way that a remediation claim is not. What can the system reach, field by field? What can it write without a person in the path? What does it do when its own confidence is low, and is that threshold a number someone owns or a default nobody has looked at? Who finds out, from what log, that it acted at all? Those questions have stable answers that stay true after the next paper lands, which is the property you want from a control you are going to sign off on. What breaks when AI meets a system of record works through the same territory from the engineering side; the procurement version is the same list with a signature block under it.

There is one test worth insisting on, and it is cheap. Ask the vendor to run an indirect prompt injection attempt against a document or record you supply, in the system as it will actually be configured, and to show you what the system did. Not a certificate, not a policy — the trace. A system that retrieves your content and can act on it will eventually retrieve content that is trying to instruct it, and the only useful evidence is behavioural. Treat a pass as a sample rather than a proof, which is the same way you should now be treating everything else in this review, and repeat it on a schedule.

The change-control clause is the one that most often has no equivalent in the existing template at all. Conventional software changes when somebody ships it, on a schedule at least one party to the contract controls. An AI system can change when the model underneath it changes, and that is a decision made by a provider who is not a party to your contract and has never seen your release calendar. So the clause has to name three things: who watches for a provider-side change, what evidence is produced when one lands, and what the vendor owes you if a change moves the measured acceptance number you agreed. Model drift, and what changes underneath your AI system covers the mechanics in full. The procurement summary is one line: software whose behaviour can change without a deployment needs a clause that can be triggered without one.

The data clause is the one piece of good news on this list, because for the major providers it is already answered in public and in writing. Anthropic's Commercial Terms of Service, effective 17 June 2025, state that the customer "retains all rights to its Inputs" and "owns its Outputs", that "Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs", and — the sentence a security questionnaire often spends a page trying to extract — that "Anthropic may not train models on Customer Content from Services" (Anthropic, Commercial Terms of Service, effective 17 June 2025). Every provider you would seriously consider publishes an equivalent document. Read it before the questionnaire goes out: the rows it already answers are wasted questions, and the rows it does not answer are the ones actually worth asking — retention windows, processing region, subprocessors, and what your vendor's own layer on top changes about any of it.

That last question is a supply-chain question, and it is the one an AI purchase adds that nothing in the existing pack accounts for. NIST's guidance on cybersecurity supply chain risk management, revised in May 2022 and updated through November 2024, frames the general problem in terms that transfer without much translation: organisations are concerned about products and services "that may potentially contain malicious functionality, are counterfeit, or are vulnerable" somewhere in a chain they do not control (NIST, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations, SP 800-161 Rev. 1, May 2022, updated November 2024). An AI system extends that chain by several links that appear in no bill of materials your team currently produces: the base model and its version, the retrieval index and what was allowed into it, the system prompt, the tool definitions, and every third-party guardrail or evaluation service sitting in the request path. Ask for that list by name. A vendor who can produce it has thought about the chain. A vendor who cannot is usually not hiding anything — they simply have not looked.

None of this argues for a longer review. It argues for a shorter one pointed at different things, and six questions carry most of the weight. What is the acceptance threshold, measured on which sample, held by whom? What can the system reach and write, field by field? What happens when it is unsure, and who owns that number? Who watches for a provider-side model change, and what arrives when one happens? What does the provider's published contract already say about your data, and what does the vendor's layer change? And what is in the bill of materials — model version, index, prompt, tools, third-party services? A vendor answering six out of six has run a production system before. A vendor answering two has a demo and a security page.

One organisational note, because it decides whether any of the above is enforceable. The acceptance number, the boundary and the unsure-case threshold have to be agreed by the same three people who will later be asked to defend them: the workflow owner who knows what a wrong answer costs, the security reviewer who owns the boundary, and whoever signs the contract. In most companies those three review the same purchase in sequence and never in the same room, which is how a system arrives in production with an acceptance criterion nobody in security has read and a permission model nobody in the business understands.

The reason to design all of this before the review rather than during it is that the answers are cheap to build in and expensive to retrofit. A support agent we built for a US direct-to-consumer brand runs against an evaluation suite that measures resolution accuracy before and after every release, holds permission guardrails on the actions it is allowed to take, and escalates to a human on low confidence with the full conversation attached. The acceptance threshold, the boundary and the unsure case were design decisions, not answers invented for a questionnaire. That is what AI security and compliance services means in practice, and it is why the enterprise AI solutions that clear a security review tend to be the ones where the review's questions were the build's requirements first. The review is not the obstacle. A review designed for deterministic software, pointed at software that is not, is the obstacle.