.DkNUKaKx_ZLAT4m.webp?dpl=dpl_FMjTLzBqpRNfBB3yRZS3sXvEUsZu){kind=small}
AI operations integrations and automations
Running agents and models in production, with the cost attributed, the tools scoped, and every run joined to the work it actually did.
15 guides · Built with Tray Headless
What this work has in common
Running a model in production is mostly an operations problem. Teams need to know what an agent did, what it was allowed to do, what it cost, what data it could read, and whether the answer it gave turned out to be right. None of that comes from the model provider’s dashboard.
The design decisions repeat across these guides. Approval is based on the reach of the action. Tool scope is set by the job the agent is doing. Success is measured by the outcome in the system of record, such as a resolved ticket, a matched invoice or a correct field. Every run carries the identity of the person it acted for, so the audit trail names a person.
The usual systems are an identity provider such as Okta, the knowledge sources the agent reads (Notion, Confluence, Google Drive, the help desk), a warehouse such as Snowflake for run logs and cost, monitoring such as Datadog, and Slack, where most approvals and alerts land.
The guides cover exposing tools over MCP, approving agent actions, tracking LLM cost and usage, keeping a vector index in sync with its sources, finding unregistered AI tools, support deflection, agent observability, and document extraction with confidence scored per field.
Where AI operations breaks
Gating on model confidence
A confidence score says how sure the model is, and nothing about what happens if it is wrong. Decide what needs a human by what the action can affect: a refund, a deleted record or an email to a customer goes to approval, whatever the score says.
Tools shaped like the API
Exposing every endpoint of a system as an MCP tool gives an agent far more reach than its job needs. Scope each tool to a task, resolve the user's identity on every call, and make writes safe to repeat.
Spend found on the invoice
An agent stuck in a loop can spend a month's budget in an afternoon. Attribute usage to a team and a feature as it happens, and alert on the rate of spend, which moves hours before the total does.
Runs that fail with an answer
The worst agent failures return a fluent, wrong result and no error. Capture whole runs, join each one to the outcome it produced, such as the ticket that reopened or the record that was corrected, and alert on those.
The ai operations guides
Each one is the design, the sequence it runs in, the prompts that build it, and what changes when it runs in production.
How to expose an internal system as an MCP tool
Scope the tool to a job rather than an API, resolve identity per call, make writes idempotent, and log every invocation.
Integration
How to build human approval for agent actions
Decide what needs approving by blast radius, show the approver what will happen, expire cleanly, and keep the record.
Automation
How to build LLM cost and usage tracking
Attribute spend to a team and a feature, alert on the rate rather than the total, and catch the loop before the invoice does.
Automation
How to build a knowledge base to vector sync
Chunk on structure, carry permissions into the index, delete on delete, and re-embed only what changed.
Integration
How to build shadow AI discovery
Find the AI tools nobody registered, from expenses, SSO and DNS, then route to a path rather than a ban.
Automation
How to build AI support deflection
Answer only what the knowledge base actually covers, escalate early with the context, and measure resolution rather than deflection.
Automation
How to build an agent observability pipeline
Capture whole agent runs instead of single calls, join each one to the outcome it produced, and alert on the failures that return an answer anyway.
Integration
How to build a document intelligence pipeline
Score confidence per field rather than per document, validate against a system of record, and route only the uncertain fields to a person.
Automation
How to scope an AI agent's tool access
Give each agent the few tools its job needs, act as the person asking or a narrow service account, split read from write, and review access on a schedule.
Automation
How to build MCP server access control
Open each MCP server to identity provider groups, make every caller sign in as themselves, cap calls, and close access the day someone leaves.
Automation
How to test an AI agent before release
Build a test set from real requests, score what the agent did as well as what it said, set a pass bar per category, and run it on every change.
Automation
How to build an MCP audit trail
Log every MCP tool call with the person, the assistant, the tool, what went in and what came back, mask what should not be stored, and send it to security's tools.
Integration
How to build an AI agent handoff to a person
Decide when the agent stops, pass the whole case to the right person, tell the user what happens next, and learn from every handoff.
Automation
How to manage the MCP server lifecycle
Publish MCP servers through an approval, keep a version history with a way back, tell the agents that depend on a tool before it changes, and retire what nobody uses.
Automation
How to build an MCP tool access review
Review on a schedule which MCP tools exist and which people and AI assistants can call them, send each owner the usage behind it, and revoke what nobody uses.
Automation
The systems involved
The applications these guides read from and write to, most used first. Each links to its connector page.
- Snowflake (15 guides)
- Slack (15 guides)
- Okta (9 guides)
- Salesforce (7 guides)
- ServiceNow (5 guides)
- Datadog (4 guides)
- Zendesk (3 guides)
- Jira (2 guides)
- Notion (2 guides)
- Google Drive (2 guides)
- GitHub (2 guides)
- Looker (1 guide)
Connections these guides build
Solutions for ai operations
The solution pages for this work, with the customer stories behind them.
- AI Agent Deployment with Tray.ai AI agent deployment, from pilot to production.
- AI and MCP governance with Tray.ai AI and MCP governance, from the first shadow AI tool to the audit log.
- MCP for Salesforce and HubSpot with Tray.ai Your CRM. Your agents. Under control.
Guides for other teams
- Revenue operations (26)
- Finance (12)
- Customer success (11)
- People operations (7)
- IT and security (15)
- Marketing (3)
- Data operations (7)
- Platform engineering (12)
- Legal and compliance (9)