Human in the loop: when an AI agent should stop

An agent is about to email four hundred customers, and the only thing between it and them is somebody noticing. At the other extreme, every action an agent takes waits for a person to click approve, and within a month the approval rate is 100% because nobody reads the requests anymore.

Both are failures of the same decision: when should an agent stop and involve a person? The glossary defines human in the loop. This post covers where it goes wrong in practice and what to fix first.

Two ways a person comes into the loop

A person joins an agent’s work at one of two points:

1. Before the agent acts. The agent has a plan and the access to carry it out, but the action is risky enough that its owner should see it first. This is approval.
2. When the agent can’t finish. The request is outside the agent’s job, it lacks access, or it has tried and failed. A person takes over. This is a handoff.

Approval sits in AI and MCP governance, because it controls what any agent may do. Handoff sits in AI agent deployment, because it decides what happens to the person the agent was serving. Most teams need both.

Where approvals break

1. The trigger is the model’s confidence. A model is often most confident when it has misunderstood, so a confidence threshold lets the worst mistakes through.
2. The approver sees a summary, not the action. A paragraph the model wrote about its own plan hides the recipient list, the amounts and the records it will change.
3. An unanswered request goes ahead. If an approval nobody answers defaults to proceeding, the step protects nothing on a busy day.
4. The request goes to whoever is nearest. The person who approves has to be able to judge the consequence, which means the owner of the system or the money.
5. Everything needs approval. Approval rates near 100% mean the threshold is wrong and people are approving without reading.

Where handoffs break

1. The user explains the problem twice. The agent tells them to raise a ticket, and the person who picks it up starts from nothing.
2. The case lands with the wrong team. The agent hands off to a general queue instead of using the routing rules the service desk already has.
3. The user isn’t told what happens next. The conversation ends, and they don’t know whether anyone has the request.

What to fix first

1. List the actions that need approval by their consequence. Anything that reaches a customer, moves money, changes many records at once or can’t be undone waits for its owner. Small, reversible actions run on their own.
2. Show the approver exactly what will happen. The recipients, the amounts and the records, taken from the action itself, not from the model’s description of it.
3. Expire to no action. An approval that isn’t answered in time is a no.
4. Hand off the whole case. Pass the request, what the agent found, what it tried and why it stopped, so nobody asks the user to repeat themselves.
5. Stop on rules you can read. Missing access, a request outside the agent’s job, or two failed attempts are better triggers for a handoff than how sure the model feels.
6. Watch the rates. Track how often agents ask for approval and hand off, and what approvers say. The common handoffs show the agent’s next job, and an approval rate near 100% shows a threshold to raise.

The guides to human approval for agent actions and agent handoff to a person have the prompts that build both.

Why it builds trust

Apollo’s IT agent resolves requests on its own and hands off the ones it can’t. It deflected 40% of tickets in its first two weeks.

“The agent only responds when it knows it can fully or partially solve the problem. If it can't, it escalates. That's why people trust it.”

For where approval sits among the other controls on agents and MCP tools, see the MCP governance framework.