Which batch did that request join?

Someone wants to know which request produced the sentence the customer saw, and the call was not a single turn. Do not treat the answer as something a batch produced, and do not ask the model which batch it was in, until you can name the batch id and the custom id, show a different request that is not in this batch, and show that you stored the batch rather than the model describing it. If the log does not keep the id, write that.

This is one Message Batch. It is not the token counts on one message. Name the tokens the response billed is usage on a response that already returned. A batch can hold many messages, and a message's input_tokens is not the batch id. It is not the thinking budget. Which budget did that thinking use? is a field on one request. A budget is not a processing status. It is not why the message stopped. Name the stop the response carried is stop_reason. end_turn on a result is not ended on a batch. It is not the temperature. Which temperature did that call use? is a sampling value. A number you sent for sampling is not a count of requests.

None of what follows is a price for this product, and none of it is a throughput you should quote a customer. The fields below are one vendor's current description of a Message Batch. The multiples below that are a 2024 research scheduler on open models. An open model on a lab GPU is not a Claude batch. A 2.6 in a paper is not 50 percent off a list price. Do not ask the model to reconcile them.

Anthropic's Message Batches API says a batch is limited to 100,000 Message requests or 256 MB, whichever is reached first. 100,000 requests is not 256 MB, and neither is a token count. Most batches complete within 1 hour. You can read results when every message has completed, or after 24 hours, whichever comes first. Batches expire if processing does not finish within 24 hours. 1 hour is not 24 hours. Results stay downloadable for 29 days after creation. After that you can still see the batch, and you cannot download the results. 29 days is not 24 hours. The guide says this cuts costs by 50 percent. It says that in the opening line, and it says it again as reducing costs by 50 percent. Two sentences, one figure. 50 percent is not 29, and it is not 24. A batched request must set max_tokens of at least 1. max_tokens 0, which is cache pre-warming, is not supported inside a batch, because a cache written during the batch would likely expire before a follow-up ran. 0 is not 1. A zero that means pre-warm is not a zero in a request count (Anthropic, Batch processing).

The sample batch is two requests, and the counts say so without a result yet. The id is msgbatch_01HkcTjaV5uDC8jWR4ZsDV8d. type is message_batch. processing_status is in_progress. request_counts prints processing 2, succeeded 0, errored 0, canceled 0, and expired 0. 2 plus 0 plus 0 plus 0 plus 0 is 2. The guide says the sum of those values always matches the number of requests in the batch. 2 is that match, on this example. It is not two customers. ended_at is null. results_url is null. cancel_initiated_at is null. created_at is 2024-09-24T18:37:24.100435Z. expires_at is 2024-09-25T18:37:24.100435Z. The dates are one day apart, at the same time of day. 2024-09-24 is not 2024-09-25. A null ended_at is not a status of ended. The two custom ids in the example are my-first-request and my-second-request. A custom id is not the batch id. The workspace example on the retrieve call is wrkspc_011CZkZaBF1tNoB5wlCeusgy. A workspace id is not a batch id. The list example asks for limit=20. 20 is a page size in a snippet. It is not 2 requests, and it is not 100,000.

processing_status and the result type are not the same list. The tracking section says the status starts as in_progress and becomes ended when every request has finished. The cancel section adds a third status: immediately after you cancel, processing_status is canceling, and a canceled batch ends as ended, with partial results for whatever was already processed. canceling is not canceled. ended after a cancel is still ended. It is not a fifth result type. The result types, once processing of the whole batch has ended, are succeeded, errored, canceled, and expired. Four names. processing is a count, not a result type. The guide says canceled, errored, and expired stay at zero until the entire batch has ended. It says the same about succeeded. It does not say that about processing. A 0 in succeeded, next to a 2 in processing, is the sample. It is not a failed batch. You are not billed for errored, canceled, or expired requests. The page does not say you are not billed for a request that succeeded. Do not extend the sentence.

The results file does not keep the order you sent. The example jsonl prints my-second-request first and my-first-request second. Order is not guaranteed. The second request's message id is msg_014VwiXbi91y3JMjcpyGBHX5, model claude-opus-5-5, stop_reason end_turn, input_tokens 11, output_tokens 36. The first request's message id is msg_01FqfsLoHwgeFbguDgpz48m7, the same model, the same stop_reason, input_tokens 10, output_tokens 34. 11 is not 10. 36 is not 34. Two succeeded results. Neither usage object is the batch's request_counts. 11 input tokens is not 2 requests. end_turn is not ended. stop_sequence is null on both. A null stop sequence is not a null ended_at. A message id that starts with msg_ is not a batch id that starts with msgbatch_. The result type on both lines is succeeded. A succeeded result is not processing_status ended, even though you only download results once the batch has ended. One is a line in the file. The other is the batch.

A research scheduler that batches requests on a GPU is not this API, and its own sentences do not use one multiple. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee introduce Sarathi-Serve. The abstract says it improves serving throughput, inside a latency target, by up to 2.6 times for Mistral-7B on a single A100, and up to 6.9 times for Falcon-180B on 8 A100 GPUs, over Orca and vLLM. The introduction adds a third multiple the abstract does not print: up to 2.8 times for Yi-34B. 2.8 is not 2.6. 2.6 is not 6.9. 8 A100s is not 1 A100. The same section places Yi-34B on 2 A100s with 2-way tensor parallelism, LLaMA2-70B on 8 A40s, and Falcon-180B with 2-way pipeline and 4-way tensor parallelism across those 8 A100s. 2 A100s is not 8 A100s. 8 A40s is not 8 A100s. 2-way is not 4-way. Figure 1 is Yi-34B on two A100s and 128 requests from one trace. 128 is not 2.8 (Agrawal et al., Sarathi-Serve).

The paper's own definitions of the same abbreviation do not match, and its prompt lengths do not match the API. One paragraph expands TTFT as time-to-first-byte. A later paragraph expands TTFT as time-to-first-token. Byte is not token. Do not pick the expansion you prefer and cite both sentences as if they were one. Figure 3 uses a prompt length of 1024 for Mistral-7B on one A100, and it says decode throughput rises almost linearly with batch size while prefill throughput nearly saturates at a single request. 1024 is a prompt length in that figure. It is not max_tokens of at least 1, and it is not 100,000 requests. A later caption puts a token budget of 512 on LLaMA2-70B. 512 is not 1024. The execution-time note says the curve is largely flat from 128 to 512 tokens. 128 there is a token count. It is not the 128 requests in Figure 1. Table 2's medians, as they quote them, are 1730 tokens for one dataset and 7059 for another. 1730 is not 7059. They also say LLaMA2-70B has an 8 times smaller key-value cache than LLaMA-65B. 8 times a cache is not 8 GPUs. None of these multiples is 50 percent off a price. A lab batch is a set of requests on a GPU. It does not return a msgbatch_ id.

Do the three proofs on one real batch, in the log that actually stored it. First, the binding. Copy the batch id, the processing_status, the custom id of the request you were asked about, and that line's result type. If the status is in_progress, write that the results are not the final counts. The guide says the succeeded, errored, canceled, and expired tallies stay at zero until the whole batch ends. Do not read a zero as a failure while processing is still 2. If the status is canceling, write canceling. Do not write canceled until the batch has ended. Second, the other request. A different custom id in the same file, with a different message id, is not this one. The example already prints two. If you cannot show a request that is not this one, you have a counter, not a record. Third, the other batch. A second batch id, or another workspace's list that does not contain this id, is the separation. If every request you stored shares one batch and you cannot say so, write that. A workspace you cannot name is the finding.

Three moves feel like the proof and are a second problem. Asking the model which batch it joined. Dividing a monthly invoice by 2 because a page said 50 percent, and calling the quotient this request. Treating 2.6 times throughput as proof that this batch finished in an hour. 2.6 is one model's capacity on one GPU in a paper. 1 hour is the guide's typical completion, not a stopwatch on your id. A usage object you have not stored is still name the tokens the response billed. Eleven input tokens on a sample line are not that object, and they are not this customer's bill.

The shape of a product that can name one conversation without naming the next is narrower than a scheduler. Inflectiv Helios runs more than one domain assistant, with conversation history and prompt monitoring. The shape that matters here is one assistant's turn you can point at, distinct from the next turn. That case is not a Message Batch ledger. Nothing in the description is a headcount, a duration, or a result figure. If the only control on the screen sends every assistant's prompts as one unlogged call, you do not have this proof. You have a chat window.

There is a version of this request where hiring the path is the wrong purchase. Nothing is in production, so there is no batch to open. No model is being called. The request was not logged. A customer has already seen another customer's data. Naming a batch id does not close a leak. The wrong value was already a write. No metric was ever named, so there is nothing for a rescue to miss. The audit that says whether any of that is true is the first one or two weeks of a rescue. The build, if there is one, is scoped after that audit, runs 8 to 12 weeks, and is unpaid if the metric you agreed is missed. Rollout stays behind a human-review gate.

When the call is real and the log does not store the batch, the remaining work is specific. An audit of what is salvageable, in the first one to two weeks. An engagement finished in eight to twelve weeks, scoped after that audit, not before it. One metric agreed up front. If the engagement does not move it, the build is on us and you do not pay for it. Rollout, when it happens, sits behind a human-review gate. That is what an AI pilot failed to reach production engagement is for. The harness that scores a stored batch id against the custom id you actually sent, on your examples, is AI evaluation and observability. If you do not yet know whether the pilot can be saved, a two-week AI readiness assessment hands you the findings either way. An AI integration sprint is the shape where the acceptance sentence can require the batch id on every stored result. Those are the terms. They are not a promise that a 2024 throughput multiple will recur, and they are not a rate. Agrawal's 2.6 is one model's cell. It will not appear on the invoice for the engagement, and it is not 50 percent.

The refund a charge got, and the review that blocked a merge, are different objects: which refund did that charge get? and which review blocked that merge?. Do not record either on this page. A refund id is not a batch id. A review state is not a processing status.

Before you tell them which request the sentence came from, do the three proofs and write the id. Name the batch, the status, the custom id, and the result type, or write that the batch was not stored. Show a different request that is not that one. Show another batch that does not hold it. Do not divide the invoice by two, and do not ask the model which batch it was. A paper can tell you what a scheduler did to tail latency on a public trace. The binding is the batch you can defend.