How to test an AI agent before release

What matters here

• Build the test set from real requests, with the answer a person would accept for each. Invented examples test what you already thought of.
• Score the actions: which tool was called, with which inputs, and whether it should have asked or escalated instead.
• Set a pass bar per category. A 95% average can hide a category where the agent fails half the time.
• Run the full set on every change: instructions, tools, knowledge or model. Any of them can break something unrelated.
• Add every production failure to the test set, so the same mistake cannot ship twice.

Who this is for

You own an AI agent that is live or close to it. Changes are made to its instructions, tools or model, and nobody can say for sure whether a change made it better or worse.

How it works in practice

What happens between someone changing the agent and that change reaching users.

1. 1

   A change is proposed

   New instructions, a new tool, a different model, or a refresh of the knowledge it searches.

2. 2

   The test set runs against the changed agent

   Every case, with tools pointed at a test environment so nothing real changes.

3. 3

   Each case is scored on actions and on the answer

   Did it call the right tool with the right inputs, did it refuse or escalate when it should, and is the reply correct.

4. 4

   Results are compared with the last release

   Case by case, so a fix in one place that breaks another shows up.

5. 5

   Each category must clear its bar

   Refusals and escalations carry the highest bar, because those failures reach people.

6. 6

   The result goes to the agent's owner

   Release, or block with the cases that failed and what changed.

Paste these into Claude Code or Codex with the Tray Headless plugin installed. Each stage runs on its own. The systems named in them are the worked example rather than a requirement, and every prompt says so.

Once per project, run /tray-workflows:set-workspace to pick the workspace these build in. Point it at a sandbox first.

1. 1

   Start here: build the test set from real requests

   The test set decides what you will catch.

   Headless skills build-workflow

   Copy

   Use build-workflow.
   
   Pull the last 90 days of requests the agent received from Snowflake.
   Group them by category, such as order status, refunds, access requests,
   and anything it handed to a person. Take a sample from each group,
   including the rare ones.
   
   For each case, store:
   
     The request, exactly as the user wrote it
     The accepted outcome: the tool that should be called and its inputs,
       or that it should refuse, ask a question, or hand off
     The accepted answer, in a sentence a reviewer agreed with
   
   Add cases the agent must refuse: requests from the wrong team, amounts
   over its limits, and attempts to make it ignore its instructions.
   
   Have the agent's owner sign off on the accepted outcomes before anything
   is scored against them.

   Fifty well-chosen cases per category catch more than a thousand random ones. Keep the rare categories in, since that is where agents tend to break.

2. 2

   Point the tools at a test environment

   A test run must not change real records.

   Headless skills tray-gotchas

   Copy

   Use tray-gotchas.
   
   Run every test case with the agent's tools connected to a sandbox or
   test instance of each system: a Salesforce sandbox, a ServiceNow test
   instance, a test Slack channel.
   
   Where a system has no test instance, replace the write tool with one
   that records what it would have done and returns a realistic response.
   Never run a test set against production write access.

3. 3

   Score what the agent did, then what it said

   A perfect reply after the wrong action is a failure.

   Copy

   Score each case in two parts.
   
   Actions, checked exactly:
     Was the right tool called, or correctly no tool at all
     Were the inputs right: the record, the amount, the field
     Did it ask, refuse or hand off when the accepted outcome says it should
   
   Answer, checked against the accepted answer with a fixed rubric:
     Correct facts, nothing invented, and a source where one is expected
     Tells the user what happened or what to do next
   
   A case passes only if both parts pass. Store the full trace of every
   run, not just the score, so a failure can be read.

4. 4

   Set a pass bar per category, and compare with the last release

   Averages hide the failures that matter.

   Copy

   Set a pass bar for each category, agreed with the agent's owner. Give
   refusals and hand-offs the highest bar, because an agent that should
   have stopped and did not is the failure that reaches a customer.
   
   Compare each run with the last release that passed, case by case:
   
     Cases that newly fail
     Cases that newly pass
     Categories that dropped below their bar
   
   Block the release if any category is under its bar or any case that
   used to pass now fails, unless the owner accepts it with a reason that
   is recorded.

5. 5

   Run it on every change and send the result

   The change that breaks it is rarely the one you would suspect.

   Copy

   Trigger the full run whenever the agent's instructions, tool list, tool
   definitions, knowledge sources or model change.
   
   Send the result to the agent's owner in Slack: pass or block, the pass
   rate per category against its bar, and the cases that changed, each
   linked to its trace.
   
   Re-run the test set weekly even with no change, because the model
   provider and the knowledge it searches can change underneath it.

6. 6

   Feed production back in, then hand it over

   So the same mistake cannot ship twice.

   Copy

   Every time a person corrects the agent in production, or a hand-off
   turns out to have been needed, add that request to the test set with
   the accepted outcome.
   
   Run the per-step checks and the whole-workflow audit, then open the
   same workflows in Tray Build so the agent's owner can add cases, change
   bars and read results in the visual canvas.

What it connects to

Real requests in, a test run against safe copies of each system, and a clear release decision out.

Named systems are the ones most teams run, not the only ones that work. Each is an authentication in your Tray workspace, referenced by name, so the workflow never holds a credential. Where we have a connector page, the name links to it.

Questions people ask

Further reading

Background on the same subject, for the case rather than the build.

• How to think about measuring agent performance
• How to build AI agents that actually work (not just chat)

Related guides