How to build an agent observability pipeline

What matters here

• The run is the unit, not the call. Nine healthy calls can compose into one wrong answer.
• Join each run to what it changed. Without that you have telemetry about an agent and no evidence about the work.
• The dangerous failure returns an answer. An error is visible; a confident wrong answer is not.
• Watch the shape of a run: tool call counts, retries and loops move before quality does.
• Retain the inputs and the tool calls, not just the final output. A complaint three weeks later is unanswerable without them.

Who this is for

You run AI platform or engineering. Agents are in production, the dashboard shows tokens and latency, and nobody can tell you which runs actually did the right thing.

How it works in practice

What has to happen between an agent run starting and somebody being able to say whether it worked.

1. 1

   A run is captured end to end

   Trigger, inputs, every tool call and result, retries, and the final output as one record.

2. 2

   The run is joined to what it changed

   The ticket it resolved, the record it updated, the message it sent.

3. 3

   Quality is scored on the outcome

   Did the ticket reopen, was the record corrected by a human, did the customer reply again.

4. 4

   Run shape is monitored

   Tool call counts, retry rates and loop detection, which move before quality does.

5. 5

   Silent failures raise an alert

   A confident wrong answer returns cleanly and is invisible to error monitoring.

6. 6

   Runs are retained with their inputs

   So a complaint three weeks later can be investigated, not guessed at.

Paste these into Claude Code or Codex with the Tray Headless plugin installed. Each stage runs on its own. The systems named in them are the worked example rather than a requirement, and every prompt says so.

Once per project, run /tray-workflows:set-workspace to pick the workspace these build in. Point it at a sandbox first.

1. 1

   Begin by capturing the whole run

   One call tells you nothing about the run it belonged to.

   Headless skills build-workflow tray-patterns

   Copy

   Use build-workflow. The systems in play are Snowflake, Datadog and the
   systems our agents act on, or whatever we run in those seats.
   
   Capture each agent run as one record: a run identifier, what triggered
   it, the inputs it received, every tool call with its arguments and
   result, every retry, the final output, and the wall clock duration.
   
   Nest tool calls under the run instead of storing them flat. The
   question people ask is what did this run do, and answering it by
   reassembling loose calls after the fact is how observability projects
   stall.
   
   Capture the arguments, not just the tool name. Which record it read is
   usually the thing that explains a wrong answer.

2. 2

   Join every run to what it changed

   Otherwise you have telemetry about an agent and no evidence about the work.

   Headless skills build-workflow

   Copy

   Use build-workflow. Write the run identifier back onto whatever the run
   touched, and store the reverse pointer on the run:
   
     The ticket it commented on or resolved
     The CRM record it updated
     The message it sent and to whom
     The approval it requested
   
   This join is the whole point. Without it you can report that four
   thousand runs completed and cannot answer whether any of them helped.
   
   Where a run touched nothing, record that explicitly. A run that read
   five systems and changed nothing is either correct restraint or a
   failure to act, and the two are worth telling apart.

   Write the run identifier onto the target record. Anything reconstructed later by matching timestamps will be wrong exactly when volume is high, which is when you need it.

3. 3

   Score on outcomes, not on errors

   Whether the run errored is the least interesting thing about it.

   Copy

   Score runs on what happened afterwards:
   
     A resolved ticket that reopened within seven days
     A field the agent set that a human changed within a day
     A customer who replied again after a supposedly complete answer
     An approval request that was rejected
     An action reversed by anybody
   
   Each is a signal that the run completed and was wrong. None of them
   appear in error monitoring, because the run returned cleanly.
   
   Aggregate by agent, by tool, and by the kind of input. Quality rarely
   degrades uniformly, and knowing that one input class fails while the
   rest are fine is what makes a fix possible, not a rewrite.

4. 4

   Watch the shape of a run, and alert on the quiet failures

   Run shape moves before quality does.

   Headless skills tray-gotchas tray-patterns

   Copy

   Use tray-gotchas, then monitor run shape and alert on it:
   
     Tool calls per run rising against the baseline for that agent
     Retry rate rising
     The same tool called repeatedly with near-identical arguments, which is
     a loop
     Duration rising while output length falls, which is usually an agent
     struggling
     A sharp change in the distribution of which tools get used
   
   These move before the outcome metrics do, because a degrading agent
   works harder first and returns worse answers second.
   
   Alert with the run identifier and a link to the whole run, not a metric.
   An alert saying retries are up leaves somebody to find the runs
   themselves, and that step is where most of these alerts get ignored.

5. 5

   Retain runs so a complaint can be investigated

   Three weeks later, guessing is the only alternative.

   Copy

   Retain complete runs, with inputs and tool calls, for a defined period,
   and treat retention as a policy question instead of a storage one.
   
   Where inputs contain personal data, note that these records fall inside
   the same subject access and deletion obligations as the source systems,
   and honour a deletion request here as well.
   
   Report: runs by agent, outcome scores, runs with no join to any outcome,
   mean tool calls per run against baseline, and the runs a human reversed.
   
   Runs with no join is the coverage measure. Each one is an agent acting
   somewhere the pipeline cannot see.

6. 6

   Validate it, then hand the scoring to the AI platform team

   Because new agents arrive and what counts as a bad outcome changes.

   Copy

   Run the per-step schema checks and the whole-workflow audit before this
   touches production. Replay a known bad run and confirm the pipeline
   scores it badly and the alert links to the run itself.
   
   Then open the same workflow in Tray Build so the AI platform team can
   add agents, adjust the outcome signals and tune the run-shape thresholds
   in the visual canvas.

What it connects to

Runs come from the agent runtime, outcomes live in the systems the agent touched.

Named systems are the ones most teams run, not the only ones that work. Each is an authentication in your Tray workspace, referenced by name, so the workflow never holds a credential. Where we have a connector page, the name links to it.

Questions people ask

Related guides