Skip to content
Insights · AI Agents

Which document did that citation name?

Name the document on that citation: the index, the title, and the span. A different citation names another document, and another request does not hold it.

Scroll
AI AgentsOct 30, 202612 min readBy Salman Naqvi, Founder & CEO
Which document did that citation name?

A customer says the answer cited the policy, or says it cited the wrong one. Do not treat the reply as something a citation produced, and do not ask the model which document it used, until you can name the citation that was stored: the type, the document index, the document title, and the span. Show a different citation whose index, or whose span, is not this one. Show that you stored the response, rather than asking the model to recall it. If any of the three needs the person who pasted the prompt, you have not shown the document. You have a fluent sentence and a guess.

This is the citation object on one response. It is not the page a person opens afterward. Open the page the answer cited is the sentence and the URL, and whether the page supports the sentence. A document index can be stored while the URL still fails, and a page can support a sentence the citation object never named. It is not the prompt. Which prompt did that request send? is the system text, the messages, and the tools. A document block can sit in that request while the citation on the response points at a different index. It is not the bill. Name the tokens the response billed is the usage object. cited_text is a field on a citation. It is not an output-token count. It is not the tool result. Which tool result did the model see? is a tool payload matched to an id. A tool id is not a document index.

None of what follows is a rate for this product, and none of it is a promise that a stored span is true. The fields below are one vendor's current description of citations on a document you passed in. The scores below that are a benchmark of generated answers that cite retrieved passages. A character index is not a census of your customers. A benchmark that never opened your response cannot name the document on the citation you stored tonight. They do tell you which numbers people mix up when they try to sound precise about a citation they have not opened.

Anthropic's plain-text example sends model claude-opus-5-5, max_tokens 1024, and anthropic-version 2023-06-01. The document source is the string The grass is green. The sky is blue. That string is 36 characters. The title on that request is My Document. The context is This is a trustworthy document. citations.enabled is true. The user text is What color is the grass and sky? 1024 is a token cap on the example. It is not a character index. 2023-06-01 is a header. It is not a span. 36 is the length of that one source string. It is not 1024. The page says all active models support citations. That sentence is not a count of models, and it does not name the model on your request (Anthropic, Citations).

The response example is a different block, and its strings do not match the request's title. The first citation is type char_location, cited_text The grass is green., document_index 0, document_title Example Document, start_char_index 0, end_char_index 20. The source slice from 0 inclusive to 20 exclusive is The grass is green. with a trailing space. That slice is 20 characters. The printed cited_text has no trailing space, and it is 19 characters. 19 is not 20. The response's visible text on that block is the grass is green, which is 18 characters, lowercase, and without the period. 18 is not 19, and it is not 20. The second citation prints cited_text The sky is blue., start 20, end 36. The source slice from 20 to 36 is The sky is blue. and it is 16 characters, which matches that cited_text. The visible text the sky is blue is 15 characters. 15 is not 16. 20 plus 16 is 36, which is the whole source. The two spans meet. The first cited_text still drops the space the first span includes. Do not repair the example by deleting the space, and do not call the visible sentence the cited_text.

Three citation shapes are on that response, and the indexes do not share a unit. char_location uses character indexes, 0-indexed, end exclusive. page_location uses page numbers, 1-indexed, end exclusive. content_block_location uses block indexes, 0-indexed, end exclusive. The page-location example prints document_index 1, title PDF Document, start_page_number 5, end_page_number 6, cited_text Water is essential for life. An exclusive end of 6 with a start of 5 is page 5, not pages 5 and 6. The visible text says page 5. The cited_text is a longer sentence than the visible words water is essential. document_index 1 is not document_index 0. The block example prints document_index 2, title Custom Content Document, start_block_index 0, end_block_index 1. End equals start plus one, so the span is one block. 1 is not 2. The visible words important findings are not the cited_text These are important findings. A generic example elsewhere on the page prints end_char_index 50 and cited_text The exact text being cited. 50 is not 20, and it is not 36, and that cited_text is not The grass is green.

What may be cited is narrower than what you sent. Text inside the document source can be cited. title and context are passed to the model and are not used toward cited content. The page says title is limited in length, and it does not print the limit. Do not invent one. My Document, Document Title, Example Document, PDF Document, and Custom Content Document are five titles on that page. Example Document is not My Document. Context This is a trustworthy document. is not the later sentence Context about the document that will not be cited from. A context string is not a citation. cited_text does not count toward output tokens. When you pass it back on a later turn, it does not count toward input tokens either. A field that is free is still a field you have to store. It is not evidence that the span is right. Plain text and PDF documents are chunked into sentences. Custom content is not chunked further. Citing images inside a PDF is not supported. A scan with no extractable text is not a citable PDF. A file_id is how you point at an uploaded file. A file_id is not a document_index.

The benchmark people quote when they want a citation rate did not read a character index. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen built ALCE on three datasets, and they randomly selected 1000 examples from the development set of each. Three selections of 1000. The page does not print 3000. Do not invent the product. ASQA uses a Wikipedia snapshot they size at 21 million passages. ELI5 uses Sphere, which they size at 899 million. 21 is not 899. QAMPARI is the third set, also on Wikipedia. 1000 is not 1024, and it is not 36. With a 4K window they say they can safely fit k equals 5 passages, and top-5 passages cover 56.8% of ASQA answers. 5 is not 56.8. Summaries reduce ASQA passage length by 6 times. 6 is not 5. Rerank draws n sample equals 4 responses. 4 is not 5. ChatGPT in the main experiments is a 4K window. LLaMA variants they tested have a 2K window. GPT-4 is an 8K window. ChatGPT-16K is 16K. 2 is not 4, 4 is not 8, and 8 is not 16 (Gao et al., Enabling Large Language Models to Generate Text with Citations).

The ELI5 table does not say one number for every column. The abstract says that on ELI5, around 50% of ChatGPT and GPT-4 generations are not fully supported by the cited passages. Table 6, Vanilla, five passages, ChatGPT: fluency 57.2, claim recall 12.0, citation recall 51.1, citation precision 50.0. 51.1 is not 50.0. Around 50% is not the cell 51.1, and it is not the claim-recall cell 12.0. Rerank on the same table is fluency 56.1, claim 11.4, citation recall 69.3, citation precision 67.8. Citation recall went up. Claim recall went down. 69.3 is not 11.4. Closed book is claim 18.6 and citation recall 15.5, with precision also 15.5. 18.6 is a higher claim score than 12.0 and a worse citation score than 51.1. GPT-4 with five passages is citation recall 44.0 and precision 50.1. 44.0 is not 50.1. GPT-4 with twenty passages is claim 18.3, citation recall 48.5, precision 53.4. 20 passages is not 5. 18.3 is not 12.0. A higher claim score is not a higher citation score.

The other two tables use different correctness units, and the human check is a fourth set of percents. ASQA Vanilla prints 66.6, 40.4, 73.6, and 72.5. QAMPARI Vanilla prints recall-at-five 20.8, precision 20.8, citation recall 20.5, and citation precision 20.9. 40.4 on ASQA is not 20.8 on QAMPARI, and neither is ELI5's 12.0. 20.8 is not 20.5. ASQA's citation recall 73.6 is not ELI5's 51.1. They say closed-book citation recall is lower than Vanilla by 47% on ASQA, and the ASQA closed-book citation cell is 26.7. 73.6 minus 26.7 is 46.9, not 47. Do not cite 47 as that subtraction. Against human labels, automatic citation recall agrees at 85.1% and citation precision at 77.6%. Cohen's kappa is 0.698 for recall and 0.525 for precision. 85.1 is not 77.6. 0.698 is not 0.525. For insufficient citations they print recall 82.3% and precision 84.2%. For irrelevant citations they print recall 75.6% and precision 66.1%. 82.3 is not 84.2. 75.6 is not 66.1. 66.1 is not 51.1. None of these is a document_index.

Do the three proofs on one real response, in the log that actually holds it. First, the binding. Copy the citation type, the document_index, the document_title, and the span: character indexes, or page numbers, or block indexes, matching the type. Copy cited_text. If the span's slice is not the cited_text, write both, the way the grass example's first span is 20 characters and the printed cited_text is 19. If title and context were sent and are not citable, do not record them as the citation. Second, the other citation. A different index, or a different span, is not this one. A benchmark percent is not that second citation. If you cannot show a citation that is not this one, you have a counter, not a record. Third, the stored response. The object has to be in a log you administer. A sentence the model produces when you ask it which document it used is a new answer. It is not the citation from the turn the customer saw.

Three moves feel like the proof and are a second problem. Opening the URL, or the PDF, and declaring the sentence supported, before you have copied the index and the span. That opening is the other article, and it still needs the pair. It does not replace the object. Quoting 51.1% or 85.1% because one of them looks precise, while the citation you were asked about is one span, and the benchmark had not opened your log. Asking the model whether the citation was right. The model will produce another sentence. It will not read the stored indexes. The page a person can click, if you still have not opened it, remains open the page the answer cited. A document_index of 0 is not that click.

The path is ordinary when an operator can open one assistant's conversation without opening the next assistant's conversation as the way to prove it. Inflectiv Helios is a platform of domain assistants, with the conversation and the prompt observable. The shape that matters here is one assistant's turn you can point at, distinct from the next assistant's turn. That case is not a citation ledger. Nothing in the description is a headcount, a duration, or a result figure. If the only control on the screen is a chat box with no stored citation, you do not have this proof. You have a reply.

There is a version of this where a rescue is the wrong purchase. Nothing is in production, so no customer saw a sentence. No model is being called. The request was not logged, so there is no citation object to name. A customer has already seen another customer's data. A document index does not close a leak. The wrong value was already a write in a system of record, and the citation was never the point. Or no metric was ever named, so fixed has no meaning you could miss. Name the metric before you buy a rebuild.

When the reply is already in front of a customer and the log cannot name the document, the missing object is specific. The index, the title, and the span on that citation, a different citation that is not that one, and a stored response rather than a recollection. An AI pilot failed to reach production engagement is for the system that is already in front of customers and cannot yet show those three: audit in the first one to two weeks, finished in 8 to 12 weeks, scoped after the audit, unpaid if the agreed metric does not move, rollout behind a human-review gate. The citation you wrote tonight is the kind of case the harness has to include, or the next release will cite just as fluently. The harness is AI evaluation and observability. An AI integration sprint is the shape where the acceptance sentence can require the citation object on every stored answer. If you do not yet know whether the pilot can be saved, a two-week AI readiness assessment hands you the findings either way. The free production-readiness diagnostic is the short way to see whether you are looking at a missing index, a page that does not support the sentence, or a system that has not reached a customer. Those are the terms. They are not 51.1%, and they are not 1024.

The dispute a charge opened, and the alert a dependency raised, are different objects: which dispute did that charge open? and which alert did that dependency raise?. Do not record either on this page. A dispute status is not a document index. An alert number is not a character span. The schema that call required is a different object: which schema did that call require?. Do not record it on this page. A schema name is not a document index.

Before you tell them the answer was grounded, do the three proofs and write the span. Name the document index, the title, and the span, or write that no citation was stored. Show a different citation that is not that one. Show that the response is in a log you hold. Do not ask the model which document it was, and do not treat a free cited_text field as a reason to skip storing it. A benchmark can tell you how often generated sentences lacked support on someone else's questions. The binding is the citation you can defend.

Questions

You stop treating the reply as something a citation produced, and you stop asking the model which document it used, until you can name the type, the document index, the title, and the span, show a different citation whose index or span is not this one, and show that you stored the response. If the log does not keep the citation, write that.

No. A percentage on a benchmark is a share of someone else's questions. A document you can defend is the index, the title, and the span on this response, or a written line that the call stored no citation. 51.1 on one table is not a document index of 0, and 1024 tokens on an example is not a character index.

When nothing is in production, when no model is being called, when the request was not logged, when a customer has already seen another customer's data, when the wrong value was already a write, or when no metric was ever named.

Find this useful? Tell Google to show you more of it.

Let's put AI to work in your business.

A 30-minute call. You bring the workflow or the roadmap — we'll tell you what's feasible, what it costs, and what we'd build first.

Book a call→