On this page
Why the demo is not the evidence
Most agent projects are approved on a demo: someone types a request, the agent calls the right tools, the result looks right. The demo is real. It is also one example, usually chosen by the person who built the agent.
Production is every other example: the ticket written in two languages, the invoice with a missing field, the request that should be refused. An agent that works on the demo and fails on a fifth of real traffic is not 80% done; it is a source of quiet errors someone will have to find by hand.
The fix is not a better demo. It is deciding, before you build, how you will know the agent works.
Define the job in one sentence
"Help the support team" cannot be evaluated. "Given an inbound refund request, either issue the refund through the payments API with the right order and amount, or route it to a person with a reason" can.
A good definition names the input, the allowed actions, and what counts as correct. If two people on the team would grade the same output differently, the definition is not done yet.
Build the evaluation set from real cases
Start with real examples of the work, taken from the system where it happens today and anonymised where they need to be.
- Cover the ordinary cases in proportion to how often they occur.
- Add the hard ones on purpose: missing data, conflicting data, ambiguous requests, cases that must be refused or escalated.
- Write the expected outcome for each, ideally with the person who does the job now.
- Keep some cases back and do not tune prompts against them, so you have an honest check at the end.
A few dozen well-chosen cases beat thousands of synthetic ones. The set grows over time: every failure found in production becomes a new case.
Check what can be checked exactly
Much of an agent's behaviour can be graded without judgement, and those checks should come first because they are cheap and unambiguous.
- Did it call the right tool?
- Do the arguments match the schema, and are the values correct?
- Did it refuse or escalate when the case required it?
- Is the output valid against the format the next system expects?
For tool-calling agents in particular, comparing the planned call against the expected call catches most real failures before any language is involved.
Grade the rest with care
Free-text outputs, such as a summary or a reply, need a rubric. A second model can apply it (often called "LLM as a judge"), and that scales well, but the judge is another system that can be wrong.
Calibrate it: grade a sample by hand, compare, and adjust the rubric until the judge agrees with people on the cases that matter. Watch for known biases, such as preferring longer answers. Treat the judge's score as a measurement with error bars, not a verdict.
Measure retrieval on its own
If the agent answers from your documents, split the evaluation in two. First, for each question, did retrieval return the passage that contains the answer? Then, given the right passage, did the model use it correctly?
When retrieval misses, no prompt will fix the answer. Measuring the two halves separately tells you which one to work on, instead of tuning prompts against a search problem.
Agree the threshold before you build
Pick the pass rate the agent must reach, and what happens to the cases it fails, before anyone sees results. Choosing the bar after seeing the score is how a 70% agent ships as "good enough".
Then make it a gate: run the evaluation on every change to prompts, tools or model, and fail the build when the score drops below the bar.
type Case = { input: string; expected: { tool: string; args: Record<string, unknown> } }
const THRESHOLD = 0.95 // example only: agree yours with the team before building
export async function gate(cases: Case[]) {
let passed = 0
for (const c of cases) {
const call = await agent.plan(c.input) // the tool call the agent intends to make
const ok = call.tool === c.expected.tool && isDeepEqual(call.args, c.expected.args)
if (ok) passed++
else console.log("FAIL", c.input, call)
}
const score = passed / cases.length
console.log(`score ${score.toFixed(3)} (threshold ${THRESHOLD})`)
if (score < THRESHOLD) process.exit(1) // the build fails, not the customer
}Make failure boring in production
Evaluation tells you how often the agent is right. Production engineering decides what happens when it is not.
- Retries with backoff for transient API errors, and idempotent actions so a retry cannot refund twice.
- A dead-letter path for cases that still fail, so nothing disappears silently.
- Alerts to a channel a person actually reads.
- A human on exceptions: the agent handles the clear cases and routes the rest with its reasoning attached.
- Logs of every tool call, so a wrong action can be traced to the input that caused it, and turned into a new evaluation case.
What this looks like as a project
This is how we scope One Workflow, Automated: a mapping session with the person who does the work today, one workflow built on your infrastructure and connected to the systems you already use, retries, a dead-letter path and alerts, and a runbook someone else can maintain it from. If it does not run end to end on your data by day ten, you do not pay the second half.
