What Changes When An AI Agent Leaves The Demo For Production

What Should You Fix First When An Agent Misbehaves?

The tool descriptions are part of the prompt. Treat them that way.

That is Anthropic's framing, not a reading of it. Its tools guidance calls prompt engineering one of the most effective methods for improving tools, and says even small refinements to tool descriptions can yield dramatic improvements. The two are not a choice, they are the same surface. It recommends making tool usage obvious through descriptions, parameters and example usage, applying poka-yoke principles by changing arguments so that it is harder to make mistakes, and choosing formats close to what the model has seen naturally occurring in text on the internet. It also suggests running many example inputs in a workbench to see what mistakes the model makes, and iterating.

The point that follows is that this work is measurable rather than a matter of taste. Anthropic's tools guidance is explicit that building an evaluation allows you to systematically measure the performance of your tools.

Start With The Tool Interface

Anthropic frames this directly. Think about how much effort goes into human computer interfaces, it says, and plan to invest just as much effort in creating good agent computer interfaces.

That framing is useful because it changes what you review. A human computer interface gets scrutinised for whether the labels are clear, whether the wrong action is easy to take by accident, and what happens when someone gets it wrong. Apply the same three questions to a tool an agent calls.

Are the description and parameters clear enough that correct use is obvious? Is a mistake made harder by the shape of the arguments, rather than merely warned against? And when the call fails, does the error tell the caller what to do differently?

Anthropic is direct about the third. It says you can prompt engineer your error responses to clearly communicate specific and actionable improvements, rather than opaque error codes or tracebacks. In my experience that is the most neglected of the three: an error saying invalid request teaches an agent nothing, so it retries the same way.

Two more from the same guidance. Namespacing, grouping related tools under common prefixes, helps delineate boundaries between lots of tools. And on token efficiency it recommends pagination, range selection, filtering or truncation, with sensible default parameter values. My own reading is that both share the instinct of the rest: reduce the ways the caller can be wrong rather than instructing it not to be.

Stopping Conditions Are Not Optional

Anthropic's guidance names stopping conditions, such as a maximum number of iterations, to maintain control.

That sentence is short and easy to skim past, and it is the difference between a bad afternoon and a bad month. An agent that can loop can loop on your bill, your rate limits and your data.

Anthropic notes that the autonomous nature of agents means higher costs and the potential for compounding errors. My own view is that an iteration cap is the cheapest way to put a ceiling on that, though what a single iteration actually costs depends on which tools it calls.

I would treat that cap as a requirement with a figure attached, not a safety net someone adds later. Write down what the agent is allowed to spend on one task, in iterations, and enforce it in code.

Guardrails And The Sandbox

The guidance is specific: extensive testing in sandboxed environments, along with the appropriate guardrails.

The word sandboxed is doing real work there. A guardrail that has only ever been tested against inputs you chose is not a guardrail, it is an assumption. The value of a sandbox is that you can run the unpleasant inputs, the malformed ones and the adversarial ones without the agent reaching anything real.

Anthropic also recommends transparency, showing the agent's planning steps explicitly. That is a debugging affordance as much as a trust one. When something goes wrong at three in the afternoon, the difference between a legible plan and an opaque one is whether you can say what happened.

There is a related pattern worth knowing from Google Cloud's production guidance. Its agent design patterns cover checkpoint and resume mechanisms to recover from failures, and delegated approval workflows where agents pause for human review while consuming zero compute resources. Note the exact claim in the second half: a paused agent consumes zero compute resources. That is narrower than zero cost, and worth quoting accurately rather than rounding up.

Where A Person Signs Off

Anthropic describes agents that pause for human feedback at checkpoints or when they hit a blocker, and says human review remains crucial even where automated testing confirms the thing works.

The method I use is a short list rather than a policy. Write down every action the agent can take. Mark the ones that move money, contact a customer, or delete anything. Those are the checkpoint candidates. Everything else is a candidate for running unattended, once tested.

The reason to write it as a list of actions rather than a rule about risk is that a list can be reviewed by someone who is not technical. The person who owns the process can look at it and say that one needs me, and that is a better conversation than a debate about autonomy levels.

Google Cloud's governance material describes a five layer stack covering agent identity, a registry for tool governance, a gateway enforcing policies, behavioural anomaly detection and a security dashboard. Worth knowing it exists. My view is that most first builds need the checkpoint list and an iteration cap long before they need any of that.

What Compounding Errors Look Like

Anthropic names the risk plainly: the autonomous nature of agents means higher costs, and the potential for compounding errors.

The shape it takes in practice, in my experience, is not a crash. The agent takes a wrong step, receives a plausible result, and proceeds confidently. Each subsequent step is reasonable given the last one. The output is coherent and wrong, which is considerably harder to notice than an exception.

That is the argument for the transparency Anthropic recommends. A visible plan lets you find the step where it went sideways. Without it you are left comparing a wrong answer against a right one and guessing.

Example: An Inbox Triage Agent That Escalates Too Much

This is a constructed example, not a client engagement, assembled from the documented guidance above.

An agent triages an inbox. It reads a message, decides which of six categories it belongs to, looks up the sender in a CRM, and either replies from a template or escalates.

The demo works. In production it starts escalating things it should answer, and the reflex is to rewrite the prompt.

The tool is worth examining alongside the prompt. The CRM lookup returns an error body that says not found, with no distinction between the sender being absent and the query being malformed. The agent cannot tell those apart, so it does the safe thing and escalates.

The fix here is in the tool. Return distinct errors, name the field at fault, and the agent has something to act on. That matches Anthropic's guidance to prompt engineer error responses so they communicate specific and actionable improvements.

Two other things follow from the same guidance. An iteration cap, so a lookup that keeps failing stops rather than retrying indefinitely. And a checkpoint on the reply action, because replying contacts a customer, while categorising does not.

That ordering is my judgement. The sources support the tool design emphasis, the stopping condition and the checkpoint pattern; which to do first is a call about where your risk sits.

Signs Your Agent Is Not Ready For Production

Four signals, drawn from building and running agent systems, including an orchestration system of my own.

1. Your debugging loop is rewriting the prompt, and it has not converged.
2. You cannot say what the agent is allowed to spend on a single task.
3. Nobody has written down which actions need a person.
4. When it gets something wrong, you cannot reconstruct which step went wrong.

Signal four is the one I would fix first, because until you can see the plan you are guessing at the other three.

If any of these apply, you can Request The Free Scoping Audit. The Agentic AI and What We Build sections are on the main site.

Common Questions About Production AI Agents

Why Is My AI Agent Unreliable In Production?

Check the tool definitions, not just the prompt wording. Anthropic treats tool descriptions as prompt engineering in their own right, and says even small refinements to them can yield dramatic improvements. It also recommends prompt engineering your error responses to give specific and actionable feedback rather than opaque error codes.

What Is An Agent Computer Interface?

Anthropic's framing for the tools an agent uses, arguing you should invest as much effort in designing them as goes into human computer interfaces. In practice it means reviewing tool descriptions, parameter shapes and error messages with the same care you would give a form a person has to fill in.

Do AI Agents Need A Maximum Number Of Iterations?

Anthropic's guidance names stopping conditions, such as a maximum number of iterations, to maintain control. It also notes that autonomy brings higher costs and the potential for compounding errors. A cap bounds how long a single task can run before something stops it, though what each iteration costs depends on which tools it calls.

How Should Agents Be Tested Before Production?

Anthropic recommends extensive testing in sandboxed environments with appropriate guardrails, and running many example inputs to see what mistakes the model makes, then iterating. A sandbox matters because it lets you run malformed and adversarial inputs without the agent reaching anything real.

When Should A Human Approve An Agent's Action?

Anthropic describes agents pausing for human feedback at checkpoints or when they hit a blocker, and says human review remains crucial even when automated testing passes. Google Cloud describes delegated approval workflows where an agent waits for review while consuming zero compute resources.

Part of The Agentic AI Guide. See also How To Choose What To Automate First and The Build Or Buy Decision.