AI agent evaluation: why agents pass the demo

Someone changes one line in the agent’s instructions on Friday, to make its answers shorter. On Monday it starts closing tickets it used to escalate. Nobody connects the two for a week, because the answers still read well. The agent passed every check anyone ran, and every check looked at the answer.

Testing before every release is the third stage of AI agent deployment. Most teams skip it, because the demo already worked.

What is AI agent evaluation?

AI agent evaluation tests an agent against a fixed set of real requests before each release. It scores what the agent did, meaning the tool it called, the inputs it passed, and whether it asked, refused or handed off when it should. Then it scores what the agent said. Each kind of request has its own pass bar, and every change runs the full set, whether it touches the instructions, the tools, the knowledge or the model.

Why agents pass the demo and fail the week after

1. The test set is invented. Examples written by the builder test the cases the builder already thought of.
2. Only the reply is scored. An agent can write a correct answer after calling the wrong tool, and the reply is the part that looks fine.
3. One average hides a failing category. A 95% pass rate can include a category where the agent fails half the time.
4. Refusals aren’t tested. Requests the agent must refuse or hand off are rare, so they’re missing from the set. They’re also where failures reach people.
5. Only some changes trigger a test. A model update or a knowledge refresh ships without a run, and either can break something unrelated.
6. Production failures never reach the test set. A mistake a user found is fixed once and never added, so it can ship again.

What it costs

An untested change reaches every user at once. An agent that stops escalating produces no errors. It produces confident wrong actions, and they surface one complaint at a time.

It also costs the speed the agent was meant to bring. Teams that can’t tell whether a change made the agent better stop changing it, and it stays at the quality of its first release.

What to fix first

1. Build the test set from real requests. Sample the last 90 days by category, including the rare ones, and write down the accepted outcome for each.
2. Add the cases it must refuse. Include requests from the wrong team, amounts over its limits, and attempts to make it ignore its instructions.
3. Point the tools at a test environment. Use a sandbox for each system, or a write tool that records what it would have done. Never give a test run production write access.
4. Score the actions first, then the answer. Check the tool, the inputs and the decision to ask or hand off exactly. Then check the reply against a fixed rubric.
5. Set a pass bar per category. Give refusals and escalations the strictest bar.
6. Run on every change, and compare with the last release. Compare case by case, so a fix in one place that breaks another shows up.
7. Add every production failure to the set. The same mistake shouldn’t ship twice.

The guide to testing an AI agent before release has the prompts for each step.

What it looks like at Apollo

Apollo’s IT agent triages every request and only responds when it can fully or partially resolve it. Employees rate its answers in Slack, and the team reviews the logs to find gaps and refine the agent’s instructions. Reviews like that are where a test set starts: real requests, each with the outcome a person accepted.

Where it fits

Testing checks what scoped tools let the agent do, and whether it stops and hands off when it should. After release, measuring agent performance shows what the test set missed, and those cases go back into it.

For all five stages, see deploying AI agents in production.