The first agent is easy. The fifth one breaks everything.

This article was originally published by Rich Waldron, Tray.ai co-founder and CEO, on LinkedIn.

In my last article, I wrote about agent sprawl and the mess that happens when every team builds agents in isolation. Here, I’ll go a level deeper and tell you what actually breaks as agent adoption scales, and what the most mature teams are doing differently.

Most teams can build an AI agent. Few are ready for what happens next.

We’ve spent the last few months in working sessions with enterprise teams deploying agents across IT, HR, procurement, and sales. Funny enough, despite their various requirements and use cases, nearly every conversation starts the same way:

“We built our first agent in Slack and it worked… now things are getting messy.”

Deploying your first agent where the majority of work happens nowadays is a great starting place. It’s scaling it out, which everyone will inevitably need to do, where things get tricky.

Since many enterprises are in similar boats, I thought I’d share what challenges I see those teams running into—and what they’re learning the hard way.

1. One agent is manageable. Five without a plan is chaos.

When it’s just one agent, everything feels under control. You know what it does. You know who built it. You can keep an eye on it.

But then someone copies it to solve another problem. Then another team wants to use a different tool. Then someone wires in a model from a hosted playground.

Suddenly:

• There are multiple agents running overlapping workflows

• No one knows what data each one touches

• And when something breaks, no one knows who owns the fix

This is how agent sprawl starts. Not with malice, but with momentum.

2. Governance isn’t optional. It’s the reason most teams stall.

One company we spoke to had three agents running successfully but paused rollout on the fourth. Why? Because leadership asked one question:

“Can we trace what this agent is doing and who approved it?”

They couldn’t. And that’s where a lot of agent strategies hit the wall. Building logic is easy. It’s harder to:

• Apply human-in-the-loop approvals

• Maintain audit trails

• Prevent prompt injection

• Log and observe decisions across systems

When deploying agents, you need clear oversight. Otherwise, you’re just taking on invisible risk.

3. Every team wants an agent. Few have intake or ownership.

Another customer told us, “Demand has exploded. Everyone wants an agent, but we don’t have a strategy for what should be built or how.”

They’re not alone. This is becoming common. With agents, the possibilities really are endless so you can’t blame RevOps for asking for an enrichment agent, or HR for wanting one to handle onboarding. And despite the overwhelmed look on IT’s face, the market is more than happy to meet them halfway. A quick Google search turns up the perfect agent for their exact use case (on a snazzy website, of course).

But here’s what’s behind that worried look on IT’s face:

Without a system for intake, prioritization, and infrastructure reuse, every new agent becomes a brand-new implementation—with its own APIs, its own logic, and its own risk profile.

More agents = more surface area.

The lesson: Start with scale in mind even if you’re only deploying one

The teams that are making real progress aren’t the ones building the most agents. They’re the ones who:

• Begin with narrow, high-leverage use cases

• Centralize delivery and observability

• Build modular workflows and shared tooling

• Treat agents like systems, not demos

The truth is your first agent is easy because you haven’t hit the constraints yet. Governance. Ownership. Orchestration. Monitoring.

You won’t feel the pain with one agent. You’ll feel it when you try to build five. And by then, you’ll wish you had started differently.