A subcontractor says the evaluation passed. You may repeat one kind of number to the client, and you must refuse three others. You may repeat a number you produced, on a set you froze before they could tune to it, with the worst case written beside the average. You refuse a score computed on a set you have not opened. You refuse a score you cannot reproduce tonight with them out of the room. You refuse an average that hides the case you sold — the refusal, the empty file, the second language, the answer that has to be wrong. If the only figure in the draft update is one of those three, delete the figure. Send the sentence instead: we have not produced a number we will stand behind.
This is the number in the client update. It is not the hour before they watch a screen, and it is not the inventory of what you still have to own when the work is AI. The screen can be perfect while the percentage in your email is theirs. The organisation, the keys, and the golden set can sit in your name while you still forward a score you did not run. Whether they should already know the other firm's name is a different decision. What the brief had to contain so the work could read as yours is a different document. Both can be in order on the day you are about to type "94 percent."
A high score on the wrong set is a known failure, and it has been measured. Ribeiro, Wu, Guestrin, and Singh tested commercial sentiment models and research models with behavioral checks rather than with another accuracy number (Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh, Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, Proceedings of ACL 2020). Two research models, BERT-base and RoBERTa-base, finetuned on the SST-2 dataset, are reported at 92.7% and 94.8% accuracy. Those two percentages are not the commercial APIs, and they are not a 2026 support agent. They are the bait. On a minimum-functionality test whose label is "negated negative should be positive or neutral," with the example "The food is not poor," the paper reports a 54.2% failure for Google Cloud Natural Language and a 29.4% failure for Amazon Comprehend. The failure rate is "near 100% for all commercial models" when the negation comes at the end of the sentence, as in "I thought the plane would be awful, but it wasn't," or when neutral content sits between the negation and the sentiment word. The commercial systems were Microsoft Text Analytics, Google Cloud Natural Language, and Amazon Comprehend, called through paid APIs, from November 2019, with similar results obtained in April 2020.
Hold the two kinds of number apart, because mixing them is how an agency writes a careless update. The 92.7% and 94.8% are held-out accuracy on a benchmark, for research models. The 54.2% and 29.4% are failure rates on one linguistic check, for two commercial APIs, on sentiment, in 2019 and 2020. "Near 100%" is the authors' description of a further negation pattern, not a third decimal you should promote into a slide. None of these is a rate you may quote about your client's model. The shape is what transfers. A model can carry a high accuracy on the set everyone already knows and still fail the sentence a customer actually writes. If your update contains only the first kind of number, you are sending the bait.
The same paper's user study is the reason "our engineers tested it" is not a number either. Eighteen participants with at least intermediate NLP experience — eight from industry, ten from academia — were given two hours, including instructions, to test a model on the QQP task. They were split equally across three conditions, so six people each. Table 4 prints 5.8 ± 1.1 tests created in the unaided condition and 13.5 ± 3.4 in the condition given capabilities plus templates, and 2.2 ± 1.2 bugs at severity 3 or higher unaided against 6.2 ± 0.9 with the templates. The abstract's summary of that comparison is that practitioners with CheckList "created twice as many tests, and found almost three times as many bugs." Twice and almost-three-times are the authors' rounding of those means, not a separate experiment, and the paper prints the ± rather than a claim that every person doubled. One participant in the unaided group had prior experience with the task. The study is two hours on a research model, not a quarter on your client's corpus. What it supports is narrower and still useful: people who only stare at a benchmark write fewer tests and find fewer serious bugs than people who are handed the behaviors to check. Your subcontractor's green suite, if you have not seen the behaviors in it, is the unaided condition with a logo.
The limit has to be written next to the number. NIST's AI Risk Management Framework states subcategory MEASURE 2.5 as: "The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the conditions under which the technology was developed are documented" (NIST, AI RMF 1.0, NIST AI 100-1, January 2023). Demonstrated, and the limitation documented, are one subcategory. A percentage with no sentence about where it stops applying has met neither half. The same publication is explicit that its suggested actions "do not constitute a checklist, nor are they necessarily an ordered set of steps," so do not treat 2.5 as a form you can tick because a dashboard is green. Treat it as the rule for the email: the number, and the condition it does not cover.
That condition is the case you sold. If the client bought a refusal — the assistant must not answer a medical question, must not invent an order status, must not proceed when the file is empty — then a score that never includes a refusal is a score about a different product. External validity, in the ordinary sense 2.5 is pointing at, is whether the result travels past the conditions of the test. A set of clean paragraphs does not travel to a scanned invoice. Write the worst case in the same sentence as the average: "On the 40 cases we froze, 36 matched the accepted answer; the four misses were empty files, and empty files are in the scope you sold." A client can use that sentence. A client cannot use "we are at 90 percent."
The set itself has to be frozen before the score, and it has to be a set you can open. If the subcontractor can read the cases on Monday and report the score on Tuesday, the number is a rehearsal. Freeze means a copy they cannot edit, dated, with a count you can check by opening the file. Owning the golden set, the model-vendor organisation, and a run you can execute without them is what to keep when you subcontract AI. This rule assumes that inventory is already yours. If it is not, you do not have a score problem. You have an ownership problem, and forwarding their percentage makes it worse, because the client will treat the percentage as yours.
You produce the number with them out of the room. Open the frozen set. Run it on the build you are about to describe. Write down the average and the misses yourself. If the run requires their notebook, their login, or their narration, you have not produced a number. You have attended one. The hour where you open the product on data shaped like the client's, including the wrong answer, with only your brand on the screen, is check the demo before the client does. The demo is the screen. This is the figure in the email that follows the screen. A client who watched a good demo and then receives a percentage you did not run has been shown two different products.
How a serious evaluation is built — what belongs in the set, how a human scores a fuzzy answer, when a number is allowed to move a release — is the evaluation suite. Do not restage that design here. The agency question is smaller and commercial. Which of the numbers already computed may leave your firm. Design without that gate produces a beautiful harness whose output you still should not forward.
There is a version of the week where hiring a different partner is the wrong purchase. The work is not sold, so there is no client to tell and no case that was bought. The client has asked to meet the people writing the code, and your obligation is the disclosure decision, not a percentage. The room only has a slide, and nobody can run the set tonight — do not invent a number to fill the slide. You cannot name the case the client bought, because the brief never held an acceptance sentence. That is a brief problem. Nothing is in production and the buyer only wants a walkthrough of a fixture. Say that the fixture passed, if you ran it, and do not call the fixture an evaluation of the system they will get.
What a score looks like when it is tied to a release, rather than to a pitch, is visible in the shape of a system that already has one. A support agent for a DTC ecommerce brand measures resolution accuracy with an evaluation suite before and after every release, and escalates on confidence with the conversation attached. That engagement is not a white-label subcontract, and nothing in this sentence is a pass rate, a headcount, or a duration. The shape is the point: the number is produced against a suite at release time, not narrated after the fact. If your subcontractor cannot show you that shape on your frozen set, you do not yet have a number to send.
Before the update goes out, delete every percentage you did not produce. Keep the one you ran, with the misses named, on the set you froze. If that leaves the email without a percentage, send it anyway. A white label software development partner that expects this will hand you the frozen set and the command that reruns it, because the number is the thing your client will repeat, and a number you cannot defend becomes your number the moment you type it. The work of building the harness that makes the rerun possible sits with AI evaluation and observability. The screen they will watch, which is a different gate from the figure, remains the demo you run alone.
Find this useful? Tell Google to show you more of it.