AI operations integrations and automations

Running agents and models in production, with the cost attributed, the tools scoped, and every run joined to the work it actually did.

8 guides  ·  Built with Tray Headless

What this work has in common

Running a model in production is mostly an operations problem. Teams need to know what an agent did, what it was allowed to do, what it cost, what data it could read, and whether the answer it gave turned out to be right. None of that comes from the model provider’s dashboard.

The design decisions repeat across these guides. Approval is based on the reach of the action. Tool scope is set by the job the agent is doing. Success is measured by the outcome in the system of record, such as a resolved ticket, a matched invoice or a correct field. Every run carries the identity of the person it acted for, so the audit trail names a person.

The usual systems are an identity provider such as Okta, the knowledge sources the agent reads (Notion, Confluence, Google Drive, the help desk), a warehouse such as Snowflake for run logs and cost, monitoring such as Datadog, and Slack, where most approvals and alerts land.

The guides cover exposing tools over MCP, approving agent actions, tracking LLM cost and usage, keeping a vector index in sync with its sources, finding unregistered AI tools, support deflection, agent observability, and document extraction with confidence scored per field.

Where AI operations breaks

Gating on model confidence

A confidence score says how sure the model is, and nothing about what happens if it is wrong. Decide what needs a human by what the action can affect: a refund, a deleted record or an email to a customer goes to approval, whatever the score says.

Tools shaped like the API

Exposing every endpoint of a system as an MCP tool gives an agent far more reach than its job needs. Scope each tool to a task, resolve the user's identity on every call, and make writes safe to repeat.

Spend found on the invoice

An agent stuck in a loop can spend a month's budget in an afternoon. Attribute usage to a team and a feature as it happens, and alert on the rate of spend, which moves hours before the total does.

Runs that fail with an answer

The worst agent failures return a fluent, wrong result and no error. Capture whole runs, join each one to the outcome it produced, such as the ticket that reopened or the record that was corrected, and alert on those.

The ai operations guides

Each one is the design, the sequence it runs in, the prompts that build it, and what changes when it runs in production.

How to expose an internal system as an MCP tool

Scope the tool to a job rather than an API, resolve identity per call, make writes idempotent, and log every invocation.

Integration

SalesforceOktaSnowflakeSlack

How to build human approval for agent actions

Decide what needs approving by blast radius, show the approver what will happen, expire cleanly, and keep the record.

Automation

SlackOktaSalesforceSnowflake

How to build LLM cost and usage tracking

Attribute spend to a team and a feature, alert on the rate rather than the total, and catch the loop before the invoice does.

Automation

SnowflakeDatadogSlackOkta

How to build a knowledge base to vector sync

Chunk on structure, carry permissions into the index, delete on delete, and re-embed only what changed.

Integration

NotionGoogle DriveConfluenceOkta

How to build shadow AI discovery

Find the AI tools nobody registered, from expenses, SSO and DNS, then route to a path rather than a ban.

Automation

OktaSlackSnowflake

How to build AI support deflection

Answer only what the knowledge base actually covers, escalate early with the context, and measure resolution rather than deflection.

Automation

ZendeskNotionSalesforceSlack

How to build an agent observability pipeline

Capture whole agent runs instead of single calls, join each one to the outcome it produced, and alert on the failures that return an answer anyway.

Integration

SnowflakeDatadogZendeskSalesforce

How to build a document intelligence pipeline

Score confidence per field rather than per document, validate against a system of record, and route only the uncertain fields to a person.

Automation

Google DriveNetSuiteSalesforceSlack

The systems involved

The applications these guides read from and write to, most used first. Each links to its connector page.

Connections these guides build

Solutions for ai operations

The solution pages for this work, with the customer stories behind them.

Guides for other teams