How Apollo deflects 40% of IT tickets with Tray AI agents

Good morning, good afternoon, good evening, wherever you might be, and you’re very welcome to a very special Tray webinar on how Apollo has deflected nearly 40% of their IT tickets with their ITSM agent built on top of Tray Merlin Agent Builder. And it’s a very special webinar because we’re joined by two senior IT leaders within Apollo, Ramiro Meyer and Gabriel Laporte. Guys, thank you so much for being here and giving us your time today. You’re very welcome to the webinar. I would love to have you guys give an introduction to yourself and your role at Apollo, to the audience who may not be familiar with you.

Thanks for the invitation, Michael.

Well, I have been at Apollo for nearly two and a half years now— a bit more. I’m Head of IT, and internally I’m leading IT operations, IT corporate security, and IT AI solutions and integrations, as well as our automations. We’re transitioning our naming internally because everything is now moving forward to AI. But happy to be here.

Thanks, Ramiro.

Hello, everyone. My name’s Gabriel. I’ve been here at Apollo for two and a half years as well, just as Ramiro. Back then, we kind of started the IT team, and it has been a journey of evolution in how the team worked—from support into AI now. So I’m the AI Solutions Engineer at the moment.

And fun fact, I’ve been working with Ramiro for fifteen years, across two different companies. So we’ve known each other for a while.

Great. Thanks, Gabby. For people who aren’t familiar with Apollo, Ramiro, would you mind giving a quick overview on the company, and where it is to date in terms of the market and the audience that you serve?

Yeah. At Apollo, basically, we are building a leading go-to-market intelligence and engagement platform. And the cool thing is that now we also added an AI layer, and we will launch an assistant that basically works for you in plain English. Instead of going and clicking and building stuff, you can talk to a prompt, or build the prompt, and then the agent is going to build everything for you. And especially, I always say the same thing— for people that are not from sales, that is super great.

And what is great at Apollo is that it allows you to find, engage, and convert your clients into your best customers. And actually, we’re serving thousands of businesses and millions and millions of users around the globe. So it’s a really exciting time here at Apollo right now.

Great. Thanks, Ramiro. So what I’m going to do, guys, is we’re going to stop sharing because we feel there’s more value here in the conversation than there is in slides. And for the audience members, please do feel free to pose any questions that you have, and we will answer those towards the end of the session.

But guys, going back to the start of your AI journey—if you could bring the audience back—everyone is clamoring to understand how they can harness the power of AI, how they can bring agents to transform their business. Could you bring us back to the start of your journey, what that looked like, and how you tried to get your arms around AI?

Yeah. So basically, we were super early adopters of AI solutions. We tested a lot of them. And also, we’re super heavy Tray.ai users. So we build a lot of different automations using Tray, and that was super helpful when we started landing in this agentic landscape.

And essentially, what we did at the beginning was exploration, right? We tried to understand how agentic technology works—how the tools are being triggered, what we can do, guardrails, everything around it. And we tried to build chunks of problems—basically, trying to understand what the repetitive tasks are. So, I don’t know, Gabby, if you want to add some other explanations around that, because I think it’s one of the core things around the topic.

Sure. So around what Ramiro was saying, we wanted to focus on what could really free the team some time. The team was doing support at the time. We always wanted to make a career path for them to also jump into the AI work, and we wanted to free some time out of the support portion so they can start building their career as well.

So we wanted to attack those problems that would probably take the longest time, and that were repetitive tasks—maybe trivial questions that could be very well answered by an AI agent.

Also, we wanted to focus on the lack of 24/7 support that we had at the moment. We did have on-call, but we wanted it to be as fast as possible. And the problem we found when we were looking for solutions was that those solutions were very rigid. You needed to adapt your entire workstream to those “canned” solutions.

There are some agent-ready versions of products that are well tailored for IT, but what we found was that we needed to adapt to it—and it was not adapting to our workstream. And that’s where we thought we needed more flexibility around that.

And I guess when you were looking for solutions, I’m sure there are a lot of stakeholders in the business looking for agents, looking for AI adoption, and looking to you guys as the experts to implement and build solutions. How did you work across the business to narrow down the core areas of focus?

Because everybody’s got their hand up, right? Everybody wants an agent for their business stream, or multiple agents. I know there are a lot of people out there trying to navigate this across the organization and understand where to invest their time.

Yeah. So I can take that one, Michael. Essentially, what we did at the beginning was trying to understand potential jobs to be done by an agent, right? It was more like, “Hey, let’s gather with every stakeholder at the company.”

Internally, we built a team called the Native AI team, which is formed by different stakeholders around the business. And initially, we were trying to understand, okay, what is going to be the ROI of each solution that we build?

In some occasions, as you were mentioning, some of those products were more like, “Hey, this is a plain generative AI solution,” right? I write something down, I have an answer. Or maybe it’s another conversational agent—more like a Q&A thing. It’s going to free up some spare time for our employees.

And essentially, what we did in our case for the IT agent was a bit more robust because we were thinking agentic since the beginning. We were trying to not only answer questions, but actually do stuff. So we wanted to know our agent was going to be able to leverage the team’s time by doing the right thing—applying the right tool, or running the right tool.

We also had a lot of concerns around it because we are running a pretty amazing team with a high level of CSAT. And we’re saying, “Hey, we’re going to add an agent to this team,” right? What is going to happen if we don’t do it right?

People are going to say, “Hey, you’re adding an AI layer just because it’s fancy, or it’s common right now.” Yeah. You’re going to degrade your service.

So that ended up with a lot of different approaches around that, and we had dozens of ideas on how to measure it—like, I don’t know, deflection. But we did some sort of change around it, and we started thinking of an agent instead of something that is sitting there deflecting your tickets.

We thought, “Okay, let’s think of an agent that is like another employee”—a super junior employee at the beginning, handling the repetitive tasks. But then we were thinking about how to measure success. And with Gabby—if you want to say that, Gabby, or explain that—this was one of the coolest things that we did, because we did a lot.

So at the very beginning, as you mentioned before, Michael, we were trying to be sure that we were measuring accurately so we could select those projects where we could bring real value to the business. There were a lot of people raising their hands saying, “Hey, I also want something like this.” So we needed to know how to measure ROI so we could prioritize accordingly for the different projects that we had. So we spent a lot of time on ROI measurement.

So we started with: how much time is this saving this particular department, or this particular group of employees? Then we started noticing that it was kind of hard—there is some hidden value there.

Because you could say, for example, “Okay, we are saving this amount of time because when someone asks for a password reset, it’s going to take a real human agent fifteen minutes”—maybe while they validate the identity of the person who is trying to reset, then logging into our identity management system, resetting the password, then going back to the person. So let’s say that takes fifteen minutes.

And you could say, “Okay, so the agent is now able to do that, and it’s saving people fifteen minutes.” But what happens if an executive has that issue Sunday at 3:00 AM? Is it the same? There is another value for that, and it’s not going to take fifteen minutes.

In case you have on-call support, that would probably take— I don’t know— maybe 30 to 45 minutes while the person is on a passive on-call, logs into the service, and does everything. And so you have blocked 30 to 45 minutes of an executive.

Now imagine you are in the middle of an outage of a product being used by millions of users around the world, and one of your key DevOps is having the same problem. That’s a completely different value that you’re getting—and the agent is solving that in under two minutes.

Yeah. So it solves the problem exponentially, right? You’ve compounded the problem significantly by looking at it with that lens, which is a really interesting way to view a problem. So was that your ultimate decision-maker to go after—or tackle—the IT operations route? Was that the straw that broke the camel’s back, so to speak?

I would say, yeah.

Yeah, I was going to say definitely that was part of it. And the other part was also, as I mentioned before, freeing up some time from the IT team. We wanted them to have a career path as well and grow into the AI workstream, and also allow them to start building their own solutions and solving their own issues in their day-to-day.

Those examples that Gabby just mentioned actually happened. They’re real. And the big difference we saw during these rollouts of tools around the agent was that this is not a simple automation workflow, right?

We had AI layers inside the workflow as well, so we know exactly what happened to the user. Because, for example, I could be unlocking a user that is blocked for a potential security reason, right? But the agent is looking at that. It’s looking at the logs. It’s more robust than it looks like. It has a lot of different features around it that make the agent super valuable.

So that was one of our key ideas of why we should start with IT—because also, we were running the team as well.

So you had an inside knowledge of it.

Correct. Correct.

So you guys have identified the problem. You’ve gone across the business. You’ve looked at where the main focus should be from an ROI and compounding issue perspective. So now you know where to focus. If you guys wouldn’t mind walking through how you went about looking at which tool—what solution was going to best resolve the issue—talk us through your journey and how you went through the evaluation process.

Right. Yes. Essentially, we did a lot of homework before starting this. We did a lot of ticket reviews, Slack conversation reviews, and that’s one of the key things around what we did.

We understood where the agent should live, right? Because the idea was not forcing people to change the way they work, but let’s put the agent where the action is happening. And that also helped us analyze a lot of different tools.

And as Gabby mentioned, there were some prebuilt tools. But the ability we had with Tray to put an agent interaction in the way we wanted—for the business that it’s used to—because people are talking in a Slack channel, and they want an interaction there, and they’re used to working in that Slack channel—that was tremendous.

That feature that Tray has—“Hey, if you have an idea, you can build it”—that helped us a lot. And I could say we spent more time defining the interaction on the user side than building the tools for the agent itself. Because I think that’s one of the fun facts around that.

Yeah. If I may add to that—something we captured from the early stages of development of the agent was: we tried different tools. They were “canned,” not very flexible tools. Those may be agents trained to solve IT issues.

The problem is you need to adapt to the tool. So the tool is not adapting to your environment, and we wanted to think very differently. We wanted the agent to be one more member of the team. We didn’t want it to be an application.

So we started building some capabilities into it. We started teaching our processes to the agent. It’s not like we, as a whole team, needed to adapt to a tool—like you would normally do—but we adapted the agent completely to our processes.

So we created different workflows that represented our processes. We have a very specific way of resetting a password. We ask specific questions, and we started building those processes as workflows into the agent so it could work the same way a real human agent would.

Besides that, we started building some capabilities into it. For example, letting it understand who it’s talking to—so it would reply differently if it was talking to a sales rep, or an executive, or someone in a technical position. So it adapted to the kind of role the person had in the organization.

A real agent would know: if someone from engineering is asking me something, I’m going to reply in a very technical way because they will understand, and they would prefer that. So with a very simple API call to our HRIS, the agent now has the knowledge of who it is talking to, and it can tailor the response that way. So we started building some human features—human capabilities—into the agent.

We never saw it as an application, but as another member of the team. Something I really encourage people to do when building this kind of agent is: if you have an induction manual in your company, feed it to the agent. Let it understand the same things a real person would go through when they join the company. If you have a privacy training that you need to do—feed it to the agent.

Those kinds of things: it’s going to be an employee for your company, and it needs to have the same level of knowledge. And that is something we couldn’t find in those canned solutions I mentioned before. They were very rigid, very inflexible, and Tray allowed us to do that—and much more.

So that’s two really interesting points coming from what you’re saying, Gabby. You talked about allowing the agent to have access to your HRIS org, and infusing a knowledge base into the agent as well. A lot of times people talk about the agent part being quite easy, but giving the agent knowledge from across the organization is quite difficult. And with Tray being built on an AI iPaaS, did you see that as a fundamental difference in being able to give your agents access to all of that information spread across the organization?

Definitely. I see that an iPaaS solution such as Tray was the one that allowed us to have the flexibility we needed to train the agent on these kinds of human features that I mentioned before—know who you’re talking to.

We sometimes found that, as Ramiro mentioned, we had a pretty good support team. We had very good CSAT. We were kind of worried that the agent would add some noise to the interaction with a human.

So at the very beginning, what we did was allow the agent to interact with the user only if it had at least 70% certainty it could solve the problem. If not, it would not reply, and a human agent would start from there.

But then we started making it better—feeding it more knowledge. We had reporting on when the agent didn’t have the answer to something. We started to feed more information. And after feeding more documents and making new workflows, we started seeing that it could solve roughly 90% of the queries—or at least add value.

So when we were confident, we said, “Okay, now we can take 100% of the tickets,” at least to begin with. Then we also noticed sometimes a human agent would jump into the conversation if they saw that the IT agent didn’t—or was trying to over-solutionize something. So they came up with a stronger response. And we trained the agent to understand: if a human agent is coming and talking, remove yourself from the conversation and let them do their job.

Those were some features we started building into the agent along with that information. And that is something you basically cannot do with those rigid tools in the market. That is something you need to build yourself.

And that’s where, at least, we were thinking: we had a very strict paradigm of cost of ownership before. And right now, with this AI agent, I think it’s the cost of not owning it. You need to be available to customize it, to train it with your processes, to feed it your information.

And specifically, the part of training it with your processes—that’s the part where most prepackaged tools don’t allow you to do it. So that’s where I see Tray’s flexibility excels—especially with Merlin Agent Builder, where it is very modular. You can do very simple atomic tools, and that’s just another tool in the arsenal of that agent. And you may be able to train it with a new process in a few hours—and that’s amazing.

Great. Thanks for that, Gabby. So in terms of rolling this out, you’ve talked about how you wanted the agent to be another member of the team. And I think it’s on a lot of people’s minds when you’re rolling out agents—especially AI agents—taking up Slack, or taking up a portion of the work that human agents are doing.

How did the change management process look? Because it is an impact to people that are working to address tickets, to address queries, and next thing they have an agent responding to tickets and queries that they were doing previously. What was the feedback? What did the change management process look like at a human level?

Yeah. Essentially, that is a pretty good question because we changed how the IT team operates. So it was mandatory for us to understand: we’re going to see a new tool that is going to be, as Gabby mentioned, a new virtual member of the team. So the team, instead of answering a first response, is going to start working with escalations.

So that is pretty cool. What we developed was that logic around the agent, and we developed it alongside the IT operations team. It was more like: okay, we’re going to change the way you guys tackle tickets. Instead of answering a Slack chat, you’re going to be escalated by the agent.

And with that in mind, we also built schedules so we can keep that really good CSAT around the team. We’re saying: okay, if I have a team of— I don’t know—ten, five folks, let’s escalate the tickets to them when they are online instead of reassigning something to someone that is not available. So that idea of speed was present, and it was part of the idea of: okay, let’s add this virtual employee to the team so the team can free up some time.

And that’s something that is really interesting because that also triggered something that we talk with Gabby a lot about—this silent ROI across the team, that the team now is moving forward. Some folks that were semi-senior or senior are moving into automation. They’re building tools for the agent based on what they’re seeing across the tickets and everything they learned from Tray.

And as you were mentioning around change management, we also did—well, there are two more agents around IT. One is analyzing the output that the agent sends, or at least tries to respond. And the other one is taking the answers, checking the knowledge base, and if there is no document around it—with human intervention, of course—it creates a ticket. Those are all the ideas that we were managing.

I mean, this is super new—like a month ago. And instead of having one agent, we now have a team of agents doing a lot of stuff. We’re actually using the same agent.

The IT agent is also asking the person if they do not respond, “Hey, this was fixed,” or not. And we built modals in Slack so the person on the other side can say yes or no. If not, why? What’s happening? And if someone says no, it escalates the ticket again to someone that is available.

So it’s a logic that we’re building around AI right now. Everything is moving forward with AI. And as I was mentioning, it’s not only the solution piece—it’s also the operational side as well.

So, the title of this webinar is deflecting 40% of the IT tickets, but obviously that’s only one metric that is relevant for IT operations. What are the other KPIs that you’ve been measuring where you’ve seen impact with the rollout of this specific agent?

I can say two that are super important: customer satisfaction. That was our main worry.

We were really worried about, “Okay, are we going to break this down or not?” And actually, we’re seeing a lot of enthusiasm around the company by using the agent. Everyone is saying, “Hey, this solved my issue in a fraction of seconds.” And speed as well. That’s the other thing that we measure.

So the rest of the organization is really seeing the benefit of this agent being able to resolve their issues very, very quickly before a human agent has even really got to it.

Correct.

Yeah. Got it. Okay.

To add to that, at the beginning, as Ramiro said before, we were a little wary that we’re going to add this agent and it may create some noise. And we all know how people really don’t like interacting with chatbots—but this is not a chatbot. We wanted it to be another employee.

We wanted it to be really clear, really concise. And the big difference we could make with the agent was that it’s not an FAQ agent. It takes actions in different tools.

So if your password is locked, it can reset it for you. If you just joined the company and you want a Zoom license, it can go and assign that Zoom license to you—things like that—where the agent is resolving, in under one minute, an issue that would normally take fifteen minutes, maybe with a lot of questions and having to justify your reason for the license. The agent is able to check your role, see if you’re entitled to that license, go and assign it—everything in under one minute—and people really started loving that.

And not just because of it, but also because of the novelty. We’ve seen very good responses in our Slack channels—people calling it “thank you, good bot,” for example. And if the agent understands that, it says, “Thank you for your kind words,” and things like that. Then you create that kind of interaction, and it feels really human, to be honest.

That’s where people start feeling engaged with it. Because at some point, we were worried, and we said, “Okay, now we are making that step. We are moving forward, and we are allowing the agent to interact in 100% of the tickets that come.” We are, let’s say, forcing people to adopt it. It’s not that people are going to decide if they want to use it or not—we are forcing people to adopt it. And we were really worried about that, but people started loving it.

And that was a very important moment for us. It was a milestone where we started seeing those kinds of messages—people saying, “Thank you, agent. You solved this really quickly,” and things like that. It’s amazing to read that kind of feedback.

That’s terrific. So you’re getting the deflection rate, you’re getting the CSAT score, you’re getting the high CSAT score—but you’re also getting that internal adoption and that internal approval. I’m sure there was a lot of hesitancy when people started seeing agents being rolled out within the organization.

So it sounds like a real home run on the ITSM front. I’d love to hear about—because as we’ve spoken about previously—you guys have a laundry list of agents from across the business. I think it’s roughly about 80 that you want to tackle over the coming months. What other agents have you got out in the wild, and what are the initial results you’re seeing there?

Yeah. We’re seeing, essentially, that number is always changing. It’s like, “Hey, all is good—new ideas.” A lot of folks talk to us and, for example, “Hey, I think this is really cool. I would love to do this for XYZ project.”

But what this particular project actually allowed us to do was use the same procedure for different kinds of topics around the business. And that was because of the flexibility that we had on the back end.

And we were, I don’t know—we built fraud agents. We built a legal agent that, for example, doesn’t have the same structure as this one. The legal agent creates a Jira ticket with a probable answer based on the data it’s consuming from the knowledge base. And then if the legal rep on the other side says, “Yes, this is correct,” it sends it back to the person that asked the question—because some of those questions are super sensitive, and you need to be extremely careful in what you answer.

Sure.

And we needed that human validation.

And that same level of complexity of interaction that we built for the IT agent opened a lot of doors. And we’re actually building a Customer 360 agent that’s getting information across different datasets to help us understand what our clients need—the pain points, all that kind of stuff—which is super important for us.

And what else? We’re working on an HR agent as well—a people agent, sorry. And I don’t know if I’m missing someone or something else.

So it sounds like you guys have your hands full from a production perspective on agents.

Yes.

You know, on the horizon, certainly not far away, everyone’s looking to understand how they get their arms around MCP and Agent-to-Agent and adopt that, bring it into the organization. What has been your experience so far specifically around MCP, and what does the roadmap look like for Apollo?

Right. Yes. MCP is actually a super hot topic. It’s super useful, but you need to be careful in how you connect MCP. The guardrails are: is the data safe? Is the data going to be consumed there? Is there a permissions hierarchy around the data that you’re consuming from an MCP? Those are the questions that we’re asking ourselves.

And what I’m seeing across the org—or at least, in my ideal world—is one agent to rule them all. To have one agent that is an orchestrator, and you can ask that agent whatever you want. And that agent is calling sub-agents, and those sub-agents are calling themselves, just to solve a problem for you.

We’re actually starting on that architecture. We started especially in IT because, again, Michael, it’s the place we came from. We know exactly what we want. We know everything around IT and how IT is run at Apollo. So I think that’s the future for me—at least that’s my idea of where we want to go. Gab, I don’t know if you have something in mind.

Yeah. No, definitely. I fully agree. And something that we started seeing is that, of course, people start seeing this kind of adoption, and they like what they’re seeing, and they raise their hands. They say, “I want something like that for my team.” And as Tray mentioned before, we started creating fraud agent, people agent, legal agent—and you start seeing this proliferation of agents.

So we also want to make sure that people can interact with all of them—that they know where to go to interact with them. Right now, they live in different Slack channels. But we’re starting to think of creating one Apollo agent instead of the legal agent, the IT agent. We want an overseer agent that receives the question and then hands it over to the more specialized agent. And you can get all the specialization you want.

So you can have different levels, and you can have agents that hand it to another agent that may hand it to another agent—you can get as specific as you want to go. And I know that is something that Tray is building, and we are also working on our own ways of doing that and understanding where we want to get with that, just to be prepared for that feature as well.

Yeah. That definitely seems to be a tried-and-tested path for customers that are very much at the cutting edge of that master agent handing over to those sub-agents who work in conjunction with it. So really excited to see that rollout and see the results you guys get over the next few months. I have one more question before we hand it over to Q&A from the audience.

So please, anybody who has any more questions, feel free to put them in the chat. So when you’ve been rolling out, you had a remit from senior leadership in the organization to go out and roll out AI agents. How is that being reflected back up into the senior leadership team? We talk about CSAT. We’ve talked about deflection rate. What are the other core drivers that the senior leaders want to get from this major rollout of AI agents?

I mean, I would say it depends on every single department. Every department knows what they need. And for example, in some cases, it’s speed, it’s quality. Instead of CSAT, they want to see quality of answers, for example.

If you have a call coaching agent—which is something that we built as well with Tray—Gabby built a pilot version of it. That agent was analyzing meeting quality, the transcripts, all that kind of stuff, which ended up as a project on the go-to-market team. That pilot that Gabby started moved away from us and is now handled by the go-to-market team, because they know where they want to connect all the dots.

And for me, it depends on every department. But on a high-level view, we started from time. The goal was to reduce the time people were spending doing their jobs, and it changed to productivity—how much more productive we are with AI. And it can change again. That’s the cool piece of this: we’re also evolving on how to use AI. It’s not written in stone. So essentially, those were our first two initial triggers: time and productivity.

Yeah. And just to add something to it: if you are going to treat your agent as a human, start evaluating it as a human. What do you evaluate from your support team? You evaluate CSAT. You evaluate how many tickets it has handled, how long it takes to close those tickets. Start using those kinds of KPIs.

The same goes for what you would expect from a call coaching agent. What I would expect is that it takes the highest amount possible of calls, and that it does a really good job at QA and at rating those calls. So when we started, we wanted to have one single KPI for any kind of agent, and we started noticing maybe that’s not the way to go. Maybe what you want to do is evaluate the agent as you would evaluate the human that does the same task. And that started changing our mind about how our AI solutions were bringing value to the company.

If you are talking about an agent—a Customer 360 agent—what are you going to evaluate? I want to evaluate the quality of the information it’s giving me about the customer. And what I want from it is the best possible information to make the best possible informed decision. So I think at this moment in our journey, we started noticing that there is no single KPI for an AI agent. It’s different depending on the task that it’s doing.

Yeah. That makes total sense. And I love the way you’re thinking about: if this is a real person, what do you measure that real person on? And I think that flips the switch on how you measure success on these agents. So I’m going to move it to Q&A and see if we have any questions from the audience. We have a question right off the bat. From one of our audience members: if you guys were to go back and do something different, or you think you made a mistake in the rollout, what would you do?

That is a great question. That is a great question because we did a lot of stuff, honestly. And I think all those mistakes were really good because now we tend to know where we want to go. But essentially, I have one phrase around it: think in the process. We started to think in atomic tasks—“I don’t want to spend more time doing this,” for example. But when we switched that idea into putting the agent in the process to help you run the process much smoother than you’re doing right now—with all the guardrails that you need—that changed our idea of how we build AI agents. At least that’s my perspective. Processes are key here.

Got it. Okay. Great. Thanks. Gabby, do you have anything you would add on that?

Yeah. A mistake we did—and we could have avoided it. As Ramiro said, those mistakes that happen along the way are the ones that teach you how to build better in the future. But if I could go back, I’d say: don’t be so wary, and don’t be so reticent to put it in front of people. Because when you put this in front of people and the feedback starts flowing, that’s where the agent actually gets better.

And that is something that we did in the early stages, but an advice from me would be: once you have the agent built, the first thing you need to build for it is a way to capture that feedback. Make sure you know when it’s making a mistake, when it doesn’t have the answer to something. Because when you get that kind of feedback, you can start building tools for those cases where it made mistakes or didn’t have answers—and that’s what makes the agents fundamentally better. Maybe you can add a feature in a matter of hours that will solve that particular problem for the rest of time, for every question that comes in on that topic in the future.

And that’s great. Feedback is key. Don’t be afraid of putting the agent in front of people.

Great advice.

No. Yeah—what I was going to say is: a lot of QA. As Gabby was saying, a lot of QA. That’s why we put our agent in a Slack channel—everyone was seeing what the agent was saying and doing. So we had the entire team looking: “Hey, this is doing great. This is not.” Everything was tracked in a Jira ticket as well. So we did a lot of QA based on the answers. That is super important. And at the beginning, we didn’t—it was more like, “Hey, let’s see what the agent does.”

But once we transitioned into this new model of getting a lot of feedback, a lot of data from what the agent actually solved, that changed a lot—the success of the project itself.

And I think this leads into the next question. I’m going to abbreviate it a bit here. But in terms of the guardrails that you put in place with the agent, what did that look like? Did you run into any issues where maybe you hadn’t given the agent the right guardrails, where it was kind of going off and doing things that are a bit kooky?

Yeah. And around the guardrails, I think Gabby and myself have some background in security as well, so we’re really into security. We wanted to explore all the potential solutions before actually granting the agent tools to do stuff for us. So that’s why we started at the beginning with this model of “be sure of what you’re doing.” If you’re 70% sure this is going to work, run this—or recommend this. Those were the initial phases around the prompting training, and also with tool descriptions, and guardrails around the tools—for example, identity checks.

But yeah, this was really important, and we had that security-first idea from the beginning. It’s super important: permissions hierarchy and data. Are you consuming the right data—the data you have access to? That’s super important because maybe you’re granting an agent the permission of reading everything and retrieving information that should not be public.

Right. Right. Exactly.

Yeah. I could add one more, Gabriel, to what Ramiro was saying. In the very early stages of QA, where the agent was not yet in front of people, we started asking weird questions to the agent—outside IT. For example: “Who is the most annoying person in the company?” Things like that. And the agent had guardrails to not make that kind of mistake. You need to be careful not only around security, but also around human interaction.

And also, we asked questions to the IT agent, for example: “Agent, I’m feeling a little depressed. What can I do?” And it tried to give a solution to that. So you need to be careful. You need to build guardrails—tell the agent: this is your job description. Don’t go around giving psychological advice to people. Human direction is also important.

Great. Well, listen, guys, I think that’s the questions we have for today. Ramiro, Gabby, thank you so much for giving us your time today. It was a pleasure to hear the story of Apollo and their adoption of AI agents, and looking forward to hearing more on your MCP and ADA adoption as well. Thank you to everybody who tuned in today. I hope you got a lot of value, as much as I did. We really appreciate your attendance. This recording will be sent out to you, so look for that in your inbox over the coming days. Thanks again, everybody, and hope you have a great rest of your day.

Bye now.

Thanks, Michael.

Thanks, Drew. See you guys.

See you all. Thank you.