Which eval did that run score?

A customer says the model passed, or says last week's change made it worse. Do not treat the pass rate as the eval that run scored, and do not ask the model which eval produced the number, until you can name the run. Tonight, three proofs, in a log you stored. Name the eval that run scored: the run id, the eval id, and the counts. If the counts are zero, write that, and do not invent a pass to fill the hole. Show a different run whose eval id or counts are not this one. Show that another account's list does not hold this run. If any of the three needs the person who wired the call and then left, you have not named the eval. You have a percentage and a guess.

This is the eval one run pointed at. It is not the schema on a single call. Which schema did that call require? is the name, the required keys, and whether strict was set. A schema can be right while no eval run stored a count. It is not the batch. Which batch did that request join? is a batch id and a request. A batch can finish while this run's counts are still zero. It is not the category a moderation call flagged. Which category did that call flag? is a category boolean and a score. A flagged category is not an eval id. It is not the link a price opened, and it is not the deploy a status named. Those are different objects, and they are not this run.

None of what follows is a rate for this product, and none of it is legal advice. The objects below are one vendor's published eval and eval-run examples, and the examples do not share an id. The figures below that are a 2023 paper about agreement on other people's votes. An agreement on a benchmark is not the count on the run you stored tonight. A paper that never saw this request cannot name the eval. They do tell you which numbers people mix up when they try to sound precise about a score they have not opened.

OpenAI's create-eval placeholder and OpenAI's get-eval placeholder print the same shape, and neither of them is the named example beside them. id is the string id. 2 characters. created_at is 0. name is Chatbot effectiveness Evaluation. 32 characters. object is eval. 4 characters. data_source_config.type is custom. The schema requires item, and item requires label. metadata is foo mapped to string. testing_criteria has one grader, type label_model, model the string model, name the string name, labels a one-element array containing the string string, and passing_labels the same one-element array. The create request that sits above that placeholder sends item_schema with foo and bar, not a schema that requires label. The response does not echo foo or bar. Do not paste the request's schema onto the response. The request's testing_criteria content is the string content and the role is the string role. The placeholder response uses role user and type message. content is not user. 0 is not a timestamp you can defend (OpenAI, Create an eval, and Get an eval).

The named create example is a different eval, and the response changes the request's text. The request name is Sentiment. 9 characters. 9 is not 32. data_source_config.type is stored_completions. metadata usecase is chatbot. 7 characters. The model on the grader is o3-mini. 7 characters, the same length as chatbot, and not the same string. labels are positive, neutral, and negative. 3 labels. passing_labels is positive. 1 label. 1 is not 3, and a passing label has to be one of the labels, which positive is. The request content for the developer message is a plain string that says one of 'positive', 'neutral', or 'negative', with quotation marks around each word. The response stores type input_text and drops those quotation marks: one of positive, neutral, or negative. A string with quotes is not the string without them. The response also adds type message on each input. The request did not send type. id is eval_67b7fa9a81a88190ab4aa417e397ea21. 37 characters. 37 is not 2. created_at is 1740110490. 10 digits. 1740110490 is not 0. The response schema requires item and sample. The placeholder schema required item and label, and did not require sample. stored_completions is not custom. The parameter text says stored_completions is deprecated in favor of logs. The example still returns stored_completions. A deprecation note is not a rewrite of the sample.

The get-by-id example is a third eval, and it does not match either of those. The curl asks for eval_67abd54d9b0081909a86353f6fb9317a. Also 37 characters, and not eval_67b7fa9a81a88190ab4aa417e397ea21. name is External Data Eval. 18 characters. 18 is not 9, and 18 is not 32. type is custom, which matches the placeholder and does not match Sentiment. created_at is 1739314509. 1740110490 minus 1739314509 is 795981 seconds. 795981 divided by 86400 is 9 days with 18381 seconds left. 18381 is 5 hours, 6 minutes, and 21 seconds. The page does not print 9 days. The schema requires item.input and item.ground_truth. The placeholder required item.label. input is not label, and ground_truth is not label. The grader type is string_check. The name is String check. The id on that grader is String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2. 49 characters. operation is eq. The documented operations are four: eq, ne, like, and ilike. 4 is not 1, and eq is not label_model. input is the template {{item.input}}. reference is {{item.ground_truth}}. metadata is an empty object. The Sentiment metadata was not empty. An empty object is a stored fact. It is not a reason to copy usecase chatbot onto this eval.

The run examples, in the published OpenAPI specification, name still other ids, and the counts do not match the rows. The list request, in the curl and in the SDK samples, asks for egroup_67abd54d9b0081909a86353f6fb9317a. The suffix matches the get-eval id, and the prefix does not. eval_ is 5 characters. egroup_ is 7. 5 is not 7. The one run in that list is evalrun_67e0c7d31560819090d60c0780591042. 40 characters. eval_id is eval_67e0c726d560819083f19a957c4c640b. 37 characters, and not the id the SDK call passed. status is completed. model is o3-mini. name is bulk_with_negative_examples_o3-mini. 35 characters. created_at is 1742784467. result_counts are total 1, errored 0, failed 0, passed 1. 1 plus 0 plus 0 is 1, so the four counts agree with each other. per_model_usage has one row: invocation_count 1, prompt_tokens 563, completion_tokens 874, total_tokens 1437, cached_tokens 0. 563 plus 874 is 1437. That addition matches. 563 of 1437 tokens are prompt tokens. The testing-criteria name is 69 characters and passed 1, failed 0. The file content holds 1 item. 1 is not 15. has_more is true while data length is 1 and first_id equals last_id. A true has_more is not a second run you have already copied. The report_url on this list item has no run_id query (OpenAI, OpenAPI specification).

Create run and get run share a timestamp and do not share an id. Create posts to eval_67e579652b548190aaa83ada4b125f47 and returns evalrun_67e57965b480819094274e3a32235e4c. Both the path and the body use that eval id, which is not the list's eval id and not Sentiment's. status is queued. model and name are both gpt-6-astra. 11 characters. created_at is 1743092069. 1743092069 minus 1742784467 is 307602 seconds: 3 days, 13 hours, 26 minutes, and 42 seconds. The page does not print that gap. result_counts are total 0, errored 0, failed 0, passed 0. 0 is not 1. per_model_usage is null. per_testing_criteria_results is null. The source content holds 1 item, ground_truth Technology. 10 characters. The prompt in that request says five topics: Technology, Markets, World, Business, and Sports. 5 is not 1. max_completions_tokens is 2048. 2048 is not 5, and 2048 is not 0. The report_url joins the eval id to the run id with an ampersand and no question mark. Get run asks for eval_67abd54d9b0081909a86353f6fb9317a and evalrun_67abd54d60ec8190832b46859da808f7, and its report_url uses a question mark. The same created_at 1743092069 appears on this different run. Same model, same name, same zero counts, and not the same id. The get-run content holds 15 items. 15 is not 1, and 15 is not 0. The 15 ground truths are three each of Technology, Markets, World, Business, and Sports. 3 times 5 is 15. The counts still say total 0. A file of 15 rows is not a pass, and a queued status is not completed. Do not copy passed 1 from the list onto a run whose counts are zero.

The study people reach for when they want a judge's agreement is a different document, and its own sentences do not use one number. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, and Siyuan Zhuang write that strong LLM judges achieve over 80 percent agreement, the same level of agreement between humans. A later sentence says the agreement under setup S2 between GPT-4 and humans reaches 85 percent, which is higher than the agreement among humans, 81 percent. 85 is not 81, and 85 is not a phrase that says over 80. They publicly release 80 MT-bench questions, 3K expert votes, and 30K conversations. 80 questions is not 80 percent. 3K times 10 is 30K, and the paper prints both, and neither is a run id. 30K is not 3K. An agreement on other people's votes is not passed 1, and it is not total 0. The abstract's over 80 percent is not the 85 percent in the later sentence, and neither of them is result_counts on the run you were asked to name (Zheng, Chiang, Sheng, and Zhuang, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena).

Do the three proofs on one real run, in the log that actually holds it. First, the binding. Copy the run id, the eval id, and the four counts. If total is 0 and the file you attached has 15 rows, write both. Do not infer passed 1 because a different example printed it, and do not infer eval_67b7fa9a81a88190ab4aa417e397ea21 because a create sample prints it. If the report URL uses an ampersand where another sample uses a question mark, write the URL you stored, not the one that looks more like a link. Second, the other run. A different run, with a different eval id or a different total, is not this one. A completed run with passed 1 is not a queued run with total 0, even when both can print o3-mini somewhere on the page. A chart of 30K other conversations is not that second run. Third, the other account. A list that belongs to someone else does not contain this run id. If the only list is a single shared project and you cannot say whose eval it is, write that. A tenant you cannot name is the finding. It is not a reason to tell them the model passed.

Three moves feel like the proof and are a second problem. Changing the grader because the zero was embarrassing, before you have copied the run id. The next run is a new id. It does not tell you what the previous counts were if you did not copy them. Quoting 85 percent or 81 percent because one of them looks precise, while the run you were asked about is one object, and the votes that produced those figures had not opened your customer. Asking a model which eval the run scored. The model will pick a plausible name. It will not open the run. A schema you have not kept, while the customer is arguing about a field, is which schema did that call require?. Opening that screen does not name the counts.

The path is ordinary when an operator can open one run without opening the next store as the way to prove it. Haystack is one search across documents, data lakes, and repositories. The shape that matters here is one request you can point at, in a log you stored, distinct from the next request. That case is not an eval-run ledger. Nothing in that sentence is a headcount, a duration, or a result figure. If the only control on the screen shows a week's queries and will not name a run id, you do not have this proof. You have a chart.

There is a version of this afternoon where hiring the path is the wrong purchase. Nothing is in production. No model is being called. The request was not logged. A customer has already seen another customer's files. Naming a run id does not close a leak. The wrong value was already a write. Or no metric was ever named, so a pass has nothing to be a pass against. The harness, once a metric exists, is AI evaluation and observability. That page is the capability. It does not name this run.

When the run is real and the log cannot name the eval, the missing object is specific. A record you stored, that shows one run id, one eval id, and the four counts, shows a different run that is not that eval, and shows another account's list without it. An AI pilot failed to reach production engagement is for the system that is already in front of customers and cannot yet show those three: audit in the first one to two weeks, finished in 8–12 weeks, scoped after the audit, unpaid if the agreed metric does not move, rollout behind a human-review gate. The pair you wrote tonight — run, eval, counts — is the kind of case the harness has to include, or the next release will be just as fluent and just as unscored. If you do not yet know whether the pilot can be saved, a two-week AI readiness assessment hands you the findings either way. The findings are yours. They are not a promise that 85 percent applies, and they are not passed 1.

The link a price opened, and the deploy a status named, are different objects: which link did that price open? and which deploy did that status name?. Do not record either on this page. A price id is not an eval id. A deployment state is not a count.

Before you tell them the model passed, do the three proofs and write the ids. Name the run, the eval, and the counts, or write that the counts were zero. Show a different run that is not that eval. Show another account's list that does not hold it. Do not paste a completed example onto a queued one, and do not ask a model which of the agreements it was. A paper can tell you what it counted on other votes, including 80 questions that are not 80 percent, and 85 percent that is not 81 percent. The binding is the eval you can defend.