Hello everybody and thank you for joining us today. I'm delighted to welcome Hyoun Park who's the Chief Analyst at Amalgam Insights as we have a discussion around what to do with your second agent.

So let's kick off. Hyoun, over to you.

Yeah, Rich, it's a pleasure to join you today. As we all know, agentic AI and the idea of AI agents have been extremely popular in the IT world and even in the business world as we are all trying to figure out what to do with AI. But there's a lot of hype, there's a lot of confusion, and there are a lot of tools out there frankly that are claiming to be able to help you with your AI needs as we are all trying to figure out where to take our next steps in 2025 and beyond. So to help ground the conversation to start with, I thought it'd be helpful just to get a basic idea of what is an agent, Rich as the CEO of Tray and someone who is right in the middle of this entire ecosystem of agentic AI. How are you describing agents to your customers and potential customers?

Yeah, it's a great point. And it's an interesting one that we have debated quite a lot internally because I think so many sort of AI based systems are being labeled agents. It's sort of become a bit of a code word or an easy way to describe something that involves AI. But really from my perspective, the key component of an agent is something that can kind of plan and act in the same way that you would expect from human or a co-worker or somebody in a different role within your life.

And the element there is that it's not just something that is sort of reasoning and regurgitating a response based on a sort of wide breadth of knowledge, it then is actually going out and completing a series of actions and kind of continuing to evolve from there. And so when we think about agents at TRA that keyword act is the thing that we speak about most because it's the way that it takes the promise of AI and the amazing reasoning that we see and brings it to life in actually being productive and supportive and really kind of excelling what your capabilities are when you're at work.

Sure and let's just dig into what it means to act. So it's easy to also say I typed something into chat GPT and then it gave me content back. So that was an action. But, you know, that's really more of a reflexive response.

When you think about actions, what does that mean to you in terms of actually getting work done?

A

good example of that is thinking of a use case.

A common example we will see is a support use case, so let's say an IT support use case. In that sense, the agent has the capability to receive the prompt, which is often, you know, a support query. You know, something's wrong with my computer.

My browser's frozen, whatever it may be. And then based on that information, it looks at the tooling that it has available to itself and decides what is the best course of action to solve that problem.

The key to that being that it may not just be a single action. It could be a series of actions. And so in that sense, the IT support agent that we provide at Tray you know will have access to tools like Okta so it will go and do things like reset a password because the user may have said that they're locked out of a system or they may have put a screenshot on to say hey, I'm struggling to get access to this piece of software. And in that sense, the action that the agent takes is to go and look at the policy, understand what that employee has available to them and actually go all the way through the provisioning process and ensure that that application then not only provides access, but it exists on the end user's machine. And so this idea that it's not just returning to you some chat, says, hey, here are the steps of the things that you now need to go and do. It's using its capability to go and act in these systems and actually complete work on your behalf. Mhmm.

Yeah. And I think that's an important thing to keep in mind that as we're thinking about agents that they are, they're great at helping with problems that are not necessarily deterministic. Something like my browser froze, or I can't do this thing where you're not even able to fully articulate the problem. You don't necessarily know what the root cause is.

This is not a one plus one equals two problem. It is, I don't even quite know what the math formula is, but can you can you solve it for me anyways? That's where this LLM-esque or nondeterministic logic starts to shine in being able to be a little bit more creative and a little more flexible in putting things together and helping you out. So as you're thinking about the agent, of course, we've already been talking about generative AI, but what else is involved in putting an agent together from a technical perspective?

Yeah, so the slide that I see used extremely regularly to describe this is your classic software ice berg slide, which is the promise of the agent is the kind of tip of the iceberg, right? It's this agent is going to go and do these things for you and it has all these capabilities and everything's going to be fantastic. But what's hidden underneath is actually a pretty comprehensive and really, if you look at it, a kind of full software application that requires ongoing maintenance, heavy security consideration, and all of the things that you require out of any other platform, you should also have the same expectation to to need from an agent.

So to stand up a sort of enterprise-grade agent today, you know, you need the capability to access data, which is often in the form of a vector or using some other third-party solution.

You need some way of kind of doing the embeddings and connecting the tools that the agent has capability to to the LLM. So you're either, you know, building that yourself or you're using a third party system to go and do that.

But really, once you kind of get beyond the descriptions of what the agent is going to do or what it has access to, you then need to be able to give the agent guardrails. So here is where you're determining, hey, here here is the scope for what you as an agent are able to do or how we would like you to handle these responses. And some of that scope may include things like static values to ensure that, you know, it can't be hallucinated to or you can't sort of trip the agent into thinking that you're somebody that you're not. That's sort of the entry-level stuff.

As you get beyond that, then starting to think about, well, what happens with the data that gets pushed into this agent? How do I go and access and see? What were all the prompts that people made when they were interacting with the agent? And then from there, what did the agent actually do?

How do I go and look at the log output to determine the action that an agent took? Because if an agent took an incorrect action, how do you know why it did it? How do you know what it did? How do you kind of track that sort of through your funnel?

And as you think about kind of building up an agent, these are all the components that you'd think about with your traditional software application. How do we handle logging? How do we do testing? How do we put in capabilities to manage different scenarios?

How do we think about how this application is going to evolve over time? It really mirrors the same consideration in the same way. These aren't a sort of one-and-done type system.

Yeah, I think that's interesting because it's easy to simply stop at kind of the vector embeddings and kind of say we have a functional basis to be able to support an agent, but then you're bringing up context which obviously includes metadata, includes role based context, you're also talking about guidelines which could be governance, compliance, security, obviously the things that we care about in an enterprise world, and then you're talking about observability, you're talking about feedback, you're talking about ongoing improvement which could even go back to some of the old school machine learning that we used to call AI, as we create these feedback loops and improvement capabilities, associated with an agent.

So there's really a lot of little pieces that go into building an enterprise-grade agent. So, you know, based on this, it sounds like, when you put an agent together for the first time, you might be able to cobble something together, but then, it becomes difficult to support on a repeatable basis. So, how do you suggest getting around that? Of course, with Tray, you have a pretty specific solution there, but, you know, kind of broadly, how do you get around the cobbling problem and make this a little bit more repeatable?

Yeah.

Think there's a few kind of interesting ways to approach this or even think about it. The first one being you have to think of an agent as a different experience to any other piece of software or application that you've ever purchased. If you go and buy a piece of off-the-shelf software for finance and you implement it, to a degree, it's a set-and-forget type scenario. Know there may be a bit of maintenance or update that occurs over time to sort of tweak some of the processes that occur, but often that's it.

Whereas with an agent, it requires constant you know nurturing and investment, and maintenance. It's very much you're trying to upskill this thing in the same way you would a junior employee. So that first tranche is, you know, how are you thinking about how you're going to scale the agent that you've built so that it not only remains productive, but actually improves in its output over time. And that in itself requires a different thought process because you're thinking about the business problem you're solving. You're thinking about how do you keep the data fresh. And to do that, you know, you're either thinking about using multiple platforms or, from a Tray perspective, a single platform to be able to provide that sort of capability.

Secondly, you know, you've built out your new agent and you've got it stood up, and then, you know, somebody in another department has come along and said, we're also cobbling together an agent, and it's over here in this silo, and it acts independently. And some of it may be updating the data that your agent is updating.

Okay, well, how do we think about kind of dealing with that system? How do these agents going to have some recognition or knowledge both exist? And I think this is becoming a problem that IT departments specifically are really starting to wrap their heads around because for a long time and certainly coming from the Tray world, we've been talking to CIOs about what to do with the myriad of applications that they've purchased and how they kind of cross over with each other and how you you need integration to go and solve these problems. When you think about it from an AI perspective, if you've got independent agents of varying skill sets and capabilities acting in some cases independently, you have an entirely new headache.

You know, you've got the governance problem in the first place, which is how do you have a central layer of governance over all of these different applications? And then as you kind of move through the stack, well, who's who's on the hook to ensure that they continue to do the thing that they say they do? And how do you sort of vet the different vendors that occur? So the way that we think about this from a Tray perspective is, you know, firstly, we provide a single platform which allows you to create multiple agents that gives you that single governance layer, that gives you that single approach to handling data and permissioning and everything kind of good that comes with that.

But for those that, you know, haven't had the fortuity of using our platform, so much of it comes in how you think about your agent strategy. You know? Exactly where are these different solutions going to live? Exactly who is gonna be responsible for updating them?

Exactly which systems do they have access to? And then lastly, and probably most critically, what is the permissioning model for this all look like? You know, if you've got an HR agent that's running somewhere and the prompts are available to the entire company, that means any prompt that any employee makes to the HR agent is viewable elsewhere. You've probably got a problem.

Secondarily, if you start ingesting knowledge at a huge scale and that knowledge can then be consumed and then pushed back out without any sort of boundary or any permissioning, again, you've got a whole new sort of data layer and access concern. So it is a real headache as you think about moving beyond that first agent and starting to think about that multi-agent model.

And I also find it really interesting that you're talking about agents as being designed for continuous improvement.

I do think that is definitely a design consideration that is not always considered from software. Often, the goal is simply to ship it and hope that you can maintain capabilities over time and that it doesn't break too much. But you're actually talking about improving over time. And I'm curious, which aspects do you typically think of as being most, perhaps improvable or enrichable from an for HIT performance?

Yeah, so again, you can kind of split this two ways. So on one hand, you've got how does the agent increase its own knowledge or its own capability over time? And from this perspective, really, it's what is it learning from the responses that it gets? You know, as each new prompt comes in, as each action is carried out, what feedback loops are you building into your agent so that it has some sense of whether or not the thing that it did was right or wrong?

Mhmm. That can be as simple as a thumbs up or a thumbs down to the end user once the process is complete, and feeding that back into the system. That can also be giving it access to a data store of previous completed actions, within a certain system, so they can see what the responses look like and get a measure on it from that perspective. On the flip side, when you think about the ongoing maintenance of an agent, you know, organizations change all the time.

And, you know, we see companies come in who, during a buying cycle with us, will change a key service within their organization. Mhmm. And so when that happens, if you think about the power of an agent, it's what it has access to. And so if the provider that you've purchased the agent from or the way that you built the agent doesn't support this new service or doesn't support this application, suddenly you've really restricted what its capability is.

So thinking about how does it access the tooling that you have available, but more importantly, how easy is it going to be to increase the breadth of action that it can take over new tooling over time? And if you're relying on a third party to continue to build integration or to continue to build access into new tooling, then there's something that you can't control in how that agent is ultimately going to be more and more productive. So there's a sort of I call it like a software layer headache in terms of what it has access to and what it can do. And then it's the actual sort of learning side of it, which is how are those feedback loops instituted?

And most importantly, what can you see for how that agent responded, where you can tweak the dials on how the agent reasons in the first place?

Yes. So when you're managing agents, it seems like there is there are so many steps and so many stages to be able to track across, kind of intended workflows, intended goals, the access capabilities.

From a practical perspective, how do you manage all this without losing track? Is this best done by, say, somebody who is in a software developer or an engineer, somebody in your integration team, somebody in your IT management team? You know, who are the people who actually have the skills and access to be able to support agents on an ongoing basis?

Yeah. I think, kind of based on where the market is today, I see it as a really similar model to what we've historically seen in the integration world, which is there's a business stakeholder. There is somebody, you know, there is somebody in a sales role or a marketing role or somebody that has a business problem that understands the breadth and depth of that problem. And then there is somebody that is going to be that administrator, that kind of governor, and in many ways is that creative for helping to put the pieces together.

And it really is a kind of collaborative approach. The best deployments or the most successful organisations we see have these harmonious small groups that are kind of working together to go and solve agentic problems, but they're centralising the learnings and they're making sure that they're not then you know there isn't another sort of part of the business which is standing up a completely different operating model or sort of systemised process somewhere else. So I think it it is that it is that blend that naturally must occur between the stakeholder and the practitioner. And then from that, it's, you know, how do you measure ROI from that? And again, it's gotta come down to the business capability. Right? It's gotta it isn't just, hey, this thing ran a thousand times today.

It's Mhmm. What work did it complete and how successful how successfully did it complete it? And I think here we're sort of at the infancy in many ways of how to think about successfully deploying agents, because in most cases they're an extension or a replacement or a repetition of a process that already exists. So, you know, if you take like a finance agent that might be processing invoices in some way and semantically pulling data and doing some action based on them, that's a continuation of a process that already exists.

I think what will start to get interesting is when people sort of forget about the processes that already exist and start to think about, well, what does an agent enable us to do that we couldn't do previously? And how do we go from A to B in a completely different way? And, again, when you think about the stakeholders involved in that, it isn't just a single unit. It needs a kind of group of people that really are thinking, you know, not just about the influence that they have within their own role, but how the whole organization is going to move based on this.

Because at some point in the not-too-distant future, these agents are probably gonna need to communicate with each other, and they're gonna need to know where to hand work off, and they're gonna need to know what's getting done where. And suddenly, you're gonna have a very different problem to the one that makes us today, which is how do I get a, you know, agent one built and stood up and and deployed. Mhmm.

Yeah. Like, I'm just kind of making this up, but when you brought up invoice processing, where we think of optical character recognition, which we've had for decades, frankly. Yeah. It's gotten better over time, but, you know, basically that technology has been around a really long time.

But now with AI, we're able to more easily figure out what that language means, how to contextualize it. And it's not hard to imagine, perhaps an agent a couple years from now that pulls in an agent, an invoice and immediately starts looking at how your technical KPIs were throughout the month and compares to your contract terms and does all of this back end work and due diligence before you decide to pay the bill and says, okay, we actually had these three flaws that were identifiable across technical business and contractual needs. So, you know, we were gonna take our discount or we're gonna dispute, or we're gonna do whatever it takes.

You know it's not quite there yet, I would say for most people, just because there's a lot of complexity associated with that level of agency, but that starts to get to the types of things that I would love to imagine seeing, going forward.

Yeah. I think the flip version of that is, you know, the reverse analysis, which is, okay. Well, if we've been processing invoices for x y z amount of time, how could we make this better?

And what would happen if we let the agent figure that out? Yeah. And maybe there's a region within the world that has a, you know, longer payment cycle. And what could we do based on local law, local knowledge, information that the agent has, which is less likely for an individual or somebody within an organisation to have that could change the way that we operate within that region.

And I think that's when this starts to get extremely interesting. And when you start to think back to the topic that we're talking about here, which is that kind of multi-agent model, that's when you get the real unlock. And I think at the moment, there are sort of different variations of maturity within organizations. There is the what do we do about agents?

There is, we got an agent stood up, now what do we do? And then there are people that are now starting to push that boundary of, we have multiple agents, how do we govern them? How do we maintain them? What do we go and do about that?

Yes, and I've also thought it was interesting how you brought up the idea of a focused center of excellence associated with the agents, your technical and your kind of line of business or process expert, being able to collect findings in a scientific or data-driven way. Because I feel like a lot of talk around agents right now is simply around getting a platform, giving it to everybody in your company, letting them all build an agent and hope that something great happens, you know, just from, I don't know, chaos theory.

But and but it seems like from a practical perspective, that's not necessarily where the real value comes in.

No. I think, you know, the proof point to this is we're starting to see organizations coming to trade that have done the first phase of implementation elsewhere or have got their first agent set up. And, you know, they've seen the promise, but they haven't seen the ROI. And the learnings they take from that are really interesting. So to use your example, which is let's either let everybody build agents and see what happens, or even more commonly, let's build the everything agent. And let's launch it in a place that requires people to go and be really good at prompting, and put it out there and let the magic happen.

And, you know, the learning that people get from that is actually people aren't great at prompting yet. And if you create a new way for them to have to go and engage, that's quite hard too. So, how do we think about where maybe there's already existing processes in place that the agent can be listening to and then pick it up? So in my IT support example, where are people currently going and putting in a support ticket? Is that in Slack or Teams? Is that in a ticketing system?

And having that as the trigger to engage the agent rather than getting the user to change the way that they behave so that everything can then kind of process from then on out. And organizations that build that center of excellence or build that collaborative team recognize these things. And then, when they think about building a new agent somewhere else, they start to think about that. Hey, how do we build this in such a way that the ROI is gonna be even more effective and we don't have a huge expectation for how people are gonna change, because you know, in most cases, they're just not there yet.

So yeah, I think that it's an important thing to have stood up and it's important that there is a lot of communication going on around what are we learning here? Like it, from my seat, the thing that's been interesting is it's taken companies that are in some cases hundreds of years old, and it's forced them to act like a startup because they have to be iterative. They have to be fast. They can't spend twelve months deploying an agent trying to figure out what it's done because by that point, it will have already failed.

So it is a really interesting sort of cultural change that's occurring as well.

And I also think it's fascinating that I don't want to totally humanize agents, but we are talking about how we trust agents, how we want agents to improve, how we want agents to be productive. It's kind of like an HR problem in that regard that you brought up the junior employee, and in some ways it is much like that. These agents that we are bringing in aren't just here to do a simple process automation; they are here to actually become part of the team over time.

And I think that gets more complex as we start thinking about multiple agents. We tease the idea of what is the second agent look like in your organization.

I'm curious what it looks like when agents start working together more independently or how you decide to have an agent start working with another agent and start giving up that personal control. And I'm curious what you've seen from that regard.

Well, I think firstly, it's important to make the point that agents are most effective when they are specialized. You know, this idea that agents have access to every single piece of information and every single capability will mostly create an inefficient, ineffective agent. So there's a need to have multiple agents. And the key to the specialism then is how the agent is described or how it is understood of what the agent's specialism is, because that then unlocks how these agents can work together.

So in the same way, an agent, you know, receives a prompt, looks at the tools that that it has available, plans what it's gonna do and acts, that same process occurs when it then says, okay, well, that planning phase, actually, I need to go to another agent to get it to carry out this part of the task. And that's where that interaction occurs. And so in the Tray world, you know, because of our history of being an integration platform, because of the breadth of integration that we have, we've built a platform that allows you to create multiple agents that can interact with each other.

Mhmm. But critically, you're able to see exactly what those agents are doing. You're able to understand the questions that the user asked. And I think, you know, as importantly, there are different models for different agents.

You know, you might choose to use OpenAI for a certain agent and Claude for a different one or DeepSeek for a completely different type of agent. And being able to change that on the fly and run that experimentation in a way where you can measure it kind of across a single platform is how you're gonna be successful when you think about your second, third, fourth, fifth, you know, agent and and and beyond. Mhmm.

Yeah. We've definitely seen a lot in the news around these general agencies, foundation models, things that are supposed to be able to do everything for everyone. So it's interesting that you are actually saying that it's more important to take a very more focused approach with the agents, just like we're starting to see in the larger model world that these mixture of experts approaches are actually providing more value where we have these little segmented pieces that, work together independently.

And so it it seems like, rather than simply have your one IT agent, for instance, you might actually want to break down the process and have one looking more at say observability, one looking more at end user computing issues or even keyboard input issues or other specific tasks where it can then dig in in a holistic manner and non deterministic but but functional. It seems like that's more in the direction of what you would typically suggest.

Yeah, entirely. And I think, you know, you're starting to see the conversation around supervisor agents as well, right? So having agents to then ratify and figure out, you know, how other agents performed, or were those tasks completed. So the layer of specialism continues to fall through that hierarchy, but that really is how you become effective.

And so for organisations that are thinking about that, starting early and thinking through, okay, well, we got our first agent live, what do we think the next two look like? How would we like that interaction to play out? What are the questions that we need to ask of the vendors that we're working with to ensure that they're built to support that kind of model? And I think that that helps get you on that journey for what really is an extremely exciting time.

But as you've said already, you know the hype of this thing is enormous. I think the delivery side of it is yet to be fully realized. The companies that have got ahead of this, that have built these teams, that are getting results, are doing it in an extremely diligent and effective manner, and you know we're very proud to be supporting so many of them.

Yeah, you had mentioned that you've already been dealing with your first generation of agentic failures with other organizations, and then they come to you when they've figured out that they've had this problem but still want to work with agentic AI. We are definitely on the analyst side, we definitely see these stories across the board with stories of fifty, sixty, seventy percent of initial agentic projects having challenges because they weren't completely thought through. Yep. Yep.

I think that's playing out in real life. I think it's exactly how things are going. So, yeah, for those that are seeing the success, it's exciting. It's thrilling. It's transformational.

But there's more involved than initially meets the eye. And I think having that backbone and thinking about that at scale is something that organizations really must do.

Yeah. So one final question I have for you is simply, what is next? Is what we're doing right now in agentic AI enough to take up the next three to five years, or is there another stage after this? What do you think is gonna be happening next that you might have to support?

Yeah, I think for me, really, the holy grail is this idea of autonomous agents or the autonomous enterprise. This idea that we're able to have these agents running effectively independently and recognizing problems and solving for them. And I think we're a ways to go for that to occur, but the groundwork is certainly already there. And you know, I think there are certainly lots of new and exciting protocols that are emerging for connecting up different types of software, for working with different vendors or even different LLMs themselves. We've already embraced things like A2A and MCP, and I'm sure many others that will exist.

But, really, from here on out, it's a case of how do we think about the new capabilities that these LLMs have? How do we harness them in the best way? And how do we get them to operate across, in some cases, relatively archaic systems where, you know, they aren't gonna be as efficient as they could be? And some of that means, you know, modernizing what you already have to then get the biggest benefit of AI. So I think there's there's certainly plenty going on, and and it's it's happening at an extremely rapid pace. But it's from my perspective, it's just such an exciting time to be in the space.

Oh that's terrific and with that thank you so much for your time today this has been a fantastic conversation just figuring out what is happening in agentic AI what is realistic, what should we be thinking about as we put in our first agent, our second agent, our third agent, what is the growth mindset and the process and the thought that we should be putting into the agents that we put into our organization, and then what do we need to figure out from an infrastructure and a support perspective to really make these agents start working and avoid some of the failures that have occurred over the past year from companies that haven't really thought this out. We have so much food for thought, and there's so much more we could talk about. And of course, I'm sure, Rich, you are available for others who want to follow up on this conversation as well.

Absolutely. And, yeah, I completely concur. Thank you so much for joining me today and having such a great conversation.

And looking forward to continuing it and carrying on for a repeat very soon.

Terrific. Okay, thanks everybody.