Does this engineer actually know Claude?

Two candidates answer "yes, I've worked with Claude" and mean completely different things. One has run an agent against Anthropic's API every working day for a year. The other read the documentation carefully last week and is a good enough engineer to sound like the first one for forty-five minutes. Nothing on either CV separates them, and a general technical interview will not either, because what separates them is not engineering ability — it is a short list of small, specific facts about how this API behaves, each stated plainly in the documentation and none of them remembered by anybody it has not already cost a day. What follows is that list, as questions you can ask, with the documented answer beside each so somebody who is not a specialist can still mark the paper.

Two things this is not. It is not a way of judging whether somebody is a good engineer at all — the four things a non-technical buyer can actually grade are in hiring your first engineer when you cannot judge engineers. Nor is it a description of the job, which is in what an AI agent engineer actually does day to day. It is the narrower question between those two: given that you already believe somebody is competent, is their competence with this model's tooling or with the general idea of models? Most of the skill is portable. A small, expensive part of it is not.

Start with the loop, because it is where reading and shipping diverge fastest. Ask what actually happens at the moment Claude decides to use one of your tools. The documented answer is unambiguous: "The response will have a stop_reason of tool_use and one or more tool_use content blocks", after which your own code has to "Run the actual tool in your codebase corresponding to that tool name, passing in the tool input" and continue the conversation with a tool_result block that carries the matching tool_use_id (Anthropic, Handle tool calls, Claude Platform documentation). The model proposes; your process disposes. Somebody who has built this says "we get a tool_use block back and our executor runs it" without being prompted, and usually volunteers what they do when a call arrives for a tool they have since removed. Somebody who has only read says "Claude calls our API," which describes a system nobody has ever debugged.

The follow-up separates the next tier: what does the reply message have to look like? The rule is specific and unforgiving: "In the user message containing tool results, the tool_result blocks must come FIRST in the content array. Any text must come AFTER all tool results," and tool results "must immediately follow their corresponding tool use blocks in the message history" (Anthropic, Handle tool calls). Get that wrong and the API returns a 400. Everyone who has shipped an agent has seen that error, usually while adding a well-meant line of narration in front of the results. Then ask the same question about failure. The documented mechanism is a tool_result carrying is_error set to true, and the advice on what to put in it is not obvious — "Instead of generic errors like 'failed', include what went wrong and what Claude should try next (for example, 'Rate limit exceeded. Retry after 60 seconds.')". An engineer who has internalised that treats error strings as part of the interface rather than as logging. The same page notes that when a tool call is malformed, "Claude will retry 2-3 times with corrections before apologizing to the user" — behaviour you want somebody to have measured rather than assumed.

Now the security question, which distinguishes a person who has taken an agent to production from one who has taken it to a demo. Where does content that arrived from outside your company go? Anthropic's own warning is explicit: "Tool results often carry content from sources outside your control: web pages, inbound email, user uploads, third-party APIs. Treat that content as untrusted: an attacker who can influence it may embed instructions that try to redirect Claude (indirect prompt injection). Keep untrusted content inside tool_result blocks rather than system prompts or plain user text blocks" (Anthropic, Handle tool calls). The wrong answer concatenates a retrieved document into the system prompt because it "gives Claude more context." The right answer names the boundary and says why it exists — and usually turns into questions back at you about which systems the agent may write to, which are worth more than the answer that prompted them.

Caching is the fourth probe and the most reliably diagnostic, because its failure mode is silence. Anthropic documents the minimum cacheable prompt length not as one number but as a per-model table — 512 tokens on the current top-end models, 1,024 on several Sonnet and Opus generations, 2,048 or 4,096 on others — then states the consequence: "Shorter prompts cannot be cached, even if marked with cache_control. Any requests to cache fewer than this number of tokens will be processed without caching, and no error is returned" (Anthropic, Prompt caching, Claude Platform documentation). Somebody who has shipped against this verified caching by reading the cache_creation_input_tokens and cache_read_input_tokens fields on the response rather than by believing the configuration, and knows the operating envelope: "You can define up to 4 cache breakpoints," and "By default, the cache has a 5-minute lifetime. The cache is refreshed for no additional cost each time the cached content is used." The sharper version of this question is a trap from a different page: Anthropic notes that "changes to the tool_choice parameter will invalidate cached message blocks. Tool definitions and system prompts remain cached, but message content must be reprocessed" (Anthropic, Define tools, Claude Platform documentation). Nobody learns that from reading about caching. You learn it by toggling forced tool use in production and watching the cache hit rate collapse.

The fifth probe is MCP, where vocabulary is cheapest. Almost every candidate says they have used the Model Context Protocol; far fewer have read the specification they are implementing against. The current version, 2026-07-28, is not a convention — it makes an MCP server an OAuth 2.1 resource server with hard obligations: "MCP servers MUST validate that access tokens were issued specifically for them as the intended audience," and, stated as plainly as a standards document ever states anything, "MCP servers MUST NOT accept or transit any other tokens" (Model Context Protocol, Authorization, specification 2026-07-28). The same section draws a line most implementers miss, that the rules differ by transport: "Implementations using an STDIO transport SHOULD NOT follow this specification, and instead retrieve credentials from the environment." Ask what stops their MCP server being used as a confused deputy for the internal API behind it. Somebody who has shipped one answers with audience validation and a separate upstream token. Somebody who has installed one describes what MCP is for.

The sixth probe is measurement, and it reveals seniority rather than familiarity. Ask how they know an agent works, and listen for whether the answer is a number over repeated trials or a story about a demo. The distinction is not taste. In the paper that introduced the τ-bench benchmark, Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan proposed a metric precisely for this, writing that they "propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials," and reported that at the time of publication "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)" (Yao, Shinn, Razavi and Narasimhan, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv, June 2024). Those figures are two years old and describe a model nobody would build on today, so they are not a current capability claim — the durable finding is the gap between succeeding once and succeeding eight times out of eight. A candidate who has felt that gap talks about golden datasets, regression gates and the score before and after a change, the loop described in our own evaluation methodology. One who has not describes the demo again, slightly louder.

The last probe is tool design, the closest thing to a portfolio review you can run in an hour. Ask them to walk through the tools they exposed to a model in a real system, and why that set rather than another. Anthropic is blunt about where tool performance comes from — "Provide extremely detailed descriptions. This is by far the most important factor in tool performance" — and about the shape of the surface: rather than a tool per action, "Consolidate related operations into fewer tools", because "Fewer, more capable tools reduce selection ambiguity and make your tool surface easier for Claude to navigate" (Anthropic, Define tools). Its engineering write-up goes further, advising that "tool implementations should take care to return only high signal information back to agents," that they should "prioritize contextual relevance over flexibility, and eschew low-level technical identifiers", and reporting the budget the company applies to its own product: "For Claude Code, we restrict tool responses to 25,000 tokens by default" (Anthropic, Writing effective tools for AI agents, 11 September 2025). The same piece warns against evaluating any of it in "overly simplistic or superficial 'sandbox' environments that don't stress-test your tools with sufficient complexity", noting that "Strong evaluation tasks might require multiple tool calls—potentially dozens." A candidate who has done this work describes the tool surface as a product with users, and can name a tool they deleted.

It is worth saying what not to test, because interviewers reach for these by default and they measure nothing. Do not ask for model names, context lengths or prices from memory: those change on a published schedule, and the engineer who has them memorised has usually just been reading. Do not ask for a prompt on a whiteboard, and do not set a take-home that builds a chatbot — the one shape of this work that has been genuinely commoditised. And be careful with certifications in either direction: they establish that somebody sat an exam, which is a real signal about diligence and no signal at all about whether they have ever watched an agent do something expensive at three in the morning.

One honest caveat. Most of what makes an AI system work is not Claude-specific: retrieval quality, permissions, integration with the systems that hold your data, the discipline of measuring before shipping. Choosing between a strong general engineer and a weaker one who happens to know this API, take the strong one — the provider-specific layer above is roughly a week of learning for somebody who already has the rest. These probes matter when the choice is between two strong engineers, or when the work is genuinely agent-shaped and starting from zero on tool-use design costs a month you do not have. That is when it is worth being deliberate about where you hire Claude engineers, and about what you asked them before you did.

For what the answers look like when the work is real rather than described, an AI support agent built on the Claude API is a reasonable reference shape: order and returns actions taken directly against the systems of record but held behind permission guardrails, confidence-based escalation to a human with the full conversation attached, and an evaluation suite measuring resolution accuracy before and after every release. Each is an answer to one of the probes above, given in code rather than in an interview. And if you are not yet sure the gap in your own project is Claude-specific at all — as opposed to retrieval, data access, or nobody having defined a wrong answer — an AI readiness assessment will tell you which hire to scope before you interview for the wrong one.