Technology
Building Agents at Scale: Lessons from the Front Lines With Gary Stafford
In this episode of AI Explained, host Will Pong engages with Gary Stafford, Principal Solutions Architect at AWS Trans, to explore the intricacies of building AI agents at scale. They discuss the stra...
Building Agents at Scale: Lessons from the Front Lines With Gary Stafford
Technology •
0:00 / 0:00
Interactive Transcript
spk_0
Welcome and thank you everyone for joining us on today's AI Explained on Building
spk_0
Agents at Scale, Lessons from the Frontlines featuring AWS Trans.
spk_0
My name is Will Pong, I'm VP of Product Management here at Fiddle or AI and I'll be your host
spk_0
today.
spk_0
We have a very special guest on today's AI Explained and that is Gary Stafford, Principal
spk_0
Solutions Architect at AWS Trans.
spk_0
He's a seasoned technology leader, speaker and author where he's applied as deep expertise
spk_0
in AIML, data analytics, enterprise architecture and software development in order to support
spk_0
a wide range of enterprises and partners across industries.
spk_0
Welcome Gary, I'm really looking forward to this conversation.
spk_0
Thank you, that you said a high bar.
spk_0
Oh, I'm very excited to dig in with you here.
spk_0
But first and foremost, maybe you could take a minute and share a little bit of your
spk_0
background in history with us.
spk_0
What should we know about you as we enter into this conversation about agents in AI?
spk_0
Sure.
spk_0
So I've been at AWS for about six years now, six years in September.
spk_0
I started in the Bay Area with our venture capital team with startups but have worked
spk_0
across enterprise, green field, what we refer to as digital aid of businesses.
spk_0
So I've spent most of my time here with customers that are all in on the cloud and then most
spk_0
recently in the ad tech space and the meeting entertainment space.
spk_0
That's awesome.
spk_0
So a lot of range of different companies there, but I think one thing that we share in
spk_0
common is thinking about that enterprise layer in particular and how we bring AI transformation
spk_0
and adoption to that layer.
spk_0
So I'm curious, just asking you this question, as lots of enterprises are exploring the idea
spk_0
of AI, how should enterprises decide whether or not a particular problem or challenge in
spk_0
front of them calls for an agentic or an AI type approach.
spk_0
What kind of rubric or sort of process do you put these teams through to figure out what's
spk_0
right for them?
spk_0
Sure.
spk_0
I think it's interesting and I somewhat unique with generative AI and agents in a genetic
spk_0
workflows.
spk_0
Oftentimes that is the reason right there stockholders there, their senior executives have told
spk_0
them that they need to develop a product or service that has gen AI, even if it's just
spk_0
for marketing purposes.
spk_0
And I think that aside, which is a legitimate request from a customer, I think what we try
spk_0
to do is go in and understand the problem first and I think of all the Amazon concepts
spk_0
that we have.
spk_0
I think working backwards is probably my favorite one.
spk_0
So if we're allowed, sometimes the customers like I have to use AI, let's just start there.
spk_0
But if we have the opportunities really work backwards and understand the problem, to
spk_0
because to that point, I think it was the same with generative AI.
spk_0
Oftentimes, these are solved problems, whether it's statistical analysis, data, analytics,
spk_0
traditional machine learning, there's already solutions out there which are probably quicker
spk_0
and maybe cheaper for customers.
spk_0
So I think we really seek to understand that problem before we suggest a solution.
spk_0
So what I'm hearing from you is the impetus can come from one of two places.
spk_0
One is sort of a top down initiative, almost like a skunk or a lab.
spk_0
Hey, we need to get an AI, get in there fast.
spk_0
We don't know what to do.
spk_0
But we have a great partner in AWS.
spk_0
Let's go lead on Gary.
spk_0
That's sort of pattern one.
spk_0
Yeah, pattern number two is sort of, hey, we have these processes.
spk_0
They could be customer facing, they could be back office facing.
spk_0
We know we want to bring AI to these processes in order to transform our business.
spk_0
So it's sort of like these two things over there.
spk_0
I'm curious on the sort of top down side or the mandate side to your point.
spk_0
What do you see teams end up doing in that case?
spk_0
Do they stand up centralized teams?
spk_0
Do they sort of have one team be a guinea pig and try something?
spk_0
What are some patterns you've seen on that type?
spk_0
Yeah, I think initially, and probably this might be just because at AWS,
spk_0
we tend to be like TORN and we're talking to technical folks.
spk_0
Often times those conversations going back to even generative AI like 2023
spk_0
started with the technical folks.
spk_0
But I think as AI is expanded, the importance of AI of a of
spk_0
agentic systems have expanded.
spk_0
More often now we're talking to certainly somebody with a level of technical
spk_0
understanding, but oftentimes there's a product development person in there,
spk_0
a product manager and a business person.
spk_0
And oftentimes that's because we're trying to solve a larger problem.
spk_0
We're trying to add a feature to a product versus just trying to implement
spk_0
a new technology like a new database engine.
spk_0
You're really talking about a new feature, a new product,
spk_0
or a new technology the company hasn't explored before.
spk_0
Yeah, that's a great point.
spk_0
This isn't just a capability.
spk_0
It's actually now some kind of outcome you're trying to drive, right?
spk_0
So it sounds like you have to be more holistic than just saying, okay,
spk_0
I've got this latest way that'll drive down your latency by this much.
spk_0
Just go implement it, right?
spk_0
Yeah, absolutely.
spk_0
So switching to that side then some of the sort of use cases that we talked about.
spk_0
What have you seen as maybe some trends or sort of repeatable things
spk_0
across the folks that you've worked with in terms of use cases that the one
spk_0
to explore using agentic technology?
spk_0
Sure.
spk_0
So maybe just start internal facing outward in AWS has always had a strong
spk_0
area of expertise within multiple industries, but in the last few years,
spk_0
we've really segmented and really focused on each one of those industrial segments.
spk_0
And I think we'll have to lead with that and look for solutions that are
spk_0
specific to industries, but the reality is, and I want to say the majority of
spk_0
the time, but let's say half the time, the customer is looking to solve a problem
spk_0
that probably transcends that industry right to your point.
spk_0
You said backend, it's maybe a name backend system or workflow that they have
spk_0
that if they can optimize that they reduce labor, they can reallocate that
spk_0
labor to maybe focus on their core business more.
spk_0
So I would say it's a combination of both customer facing products and features,
spk_0
but also just more backend task automation.
spk_0
Okay.
spk_0
So let's take a half a step back then as we start thinking about solutions there.
spk_0
We've both used the term genai and agentic a little bit, but I'd love to know from your point
spk_0
of view, what's the difference, especially for folks that are a little bit newer to these
spk_0
technologies? What does it mean to use genai generally? And then what are agents? And how
spk_0
should we think about those capabilities being different? Sure. And I, I'll refer, I love
spk_0
Anthropic. I think Anthropics is a great job in really defining agents, obviously defining
spk_0
MCP and I love to reference their website and they have some great definitions. But when I think
spk_0
of an agent, I think of three things right, an LLM, the ability to communicate with an LLM,
spk_0
the ability to use tools and the ability to reason in an agentic loop, right? So the ability to
spk_0
leverage genai to leverage a large language model, which is capable of a generative process,
spk_0
whether that's text or image and the ability to reason about tools and use those tools to
spk_0
derive an answer to a question or a query. I see. So just reflecting that back to you,
spk_0
I think genai approaches, even as we think about the last couple of years here, are a little bit
spk_0
more about that single turn of processing information, right? I'm a user asking a question of a
spk_0
chatbot or maybe even if I'm a backend process, I take in some input and I give an output. But you're
spk_0
saying one big difference from agents, especially as Anthropic is not just to find, is the idea that
spk_0
that reasoning isn't just applied to the input from the user, but that reasoning is also applied
spk_0
to how to do the job. So reasoning about what plans to go make or what tools to go use and etc,
spk_0
that is what separates an agentic use case from sort of a more straightforward LLM based
spk_0
use case. Yeah, I would say does that mean? It does. Yeah, I think they can be non-deterministic,
spk_0
right? The way in which it decides, use those tools is purely non-deterministic or it determines
spk_0
the use based on the input and what usually varies in that case. Or it can be deterministic in
spk_0
those frameworks that have more deterministic, dag-based workflows. But I think either way where the
spk_0
agent is making the decision on which tools to use, I see that more as an agentex system where there's
spk_0
less determinism. That makes sense. So let's tie that back to some of the business use cases that
spk_0
you've seen in the verticals that you've looked at, whether it's M&E or elsewhere. What are some
spk_0
particular use cases where this agentex approach where the sort of the software itself is making
spk_0
plans and figuring out what to do? Where is that particularly useful compared to the recent world
spk_0
where we might have to code all of that logic ourselves? Yeah, I think the obvious one in this
spk_0
transcends AWS and I think it goes for all vendors in this space is really code development, code
spk_0
agents, co-pilots, coding assistance in being able to develop what started off as a simple method
spk_0
or a class and now developing entire application, developing the testing, developing the documentation
spk_0
with our product cure, or more of a spectrum in development where you're writing the requirements and
spk_0
it's creating the process and the flow using agents to develop that. I think that's the obvious
spk_0
use case. That's probably one of the larger use cases that I think the entire industry has seen.
spk_0
I think beyond that there's the back office task, there's the back office automation. I have
spk_0
one, two or three humans in the loop with specialized knowledge and this might not be there
spk_0
oftentimes it's not their primary job but they're required to perform some task and that's usually
spk_0
an obvious area of automation to free them up to do their higher value tasks and then there's
spk_0
then I think that third one is more product and product feature customer facing, being able to
spk_0
add AI or agentex capabilities to existing products or develop new products that have those
spk_0
capabilities and working with customers to do that. That makes sense and let's dig into the
spk_0
back office conversation a little bit more. Traditionally, you might argue that that world you just
spk_0
described is still broadly true whether or not there's agents. I always think about back office
spk_0
automation taking more stuff and putting it in code rather than having humans do wrote work.
spk_0
How is this different than just using Zapier or using some integrations that automate work
spk_0
generally? Why do agents make the difference here compared to the tools we've had before?
spk_0
Maybe I'll answer that with the approach. Again, everyone has a little bit of different
spk_0
approach when they approach or talk to a customer and I come from a consulting background.
spk_0
I think taking more of a consulted approach is understanding who are the humans in the loop?
spk_0
What are their tasks are they specialized? But I think specific financial question,
spk_0
is there institutional knowledge there that needs to be captured?
spk_0
They're performing this task because they have particular knowledge that maybe no one else in
spk_0
the company does. Those are the type of tasks that I think agents can help with. It's not
spk_0
deterministic. The inputs can vary. The problem, although the problem may have boundaries around it,
spk_0
the specific tasks can vary somewhat depending on the day to week, the month, the particular
spk_0
scope of the problem. Then those humans and the loop are applying a particular knowledge that
spk_0
they have specific knowledge to the problem. Those are usually the types of problems where I think
spk_0
an agenteic system can help versus... I see. That's really interesting. Do you have an example
spk_0
that maybe you can walk us through? You can feel free to sort of scrub it or sanitize
spk_0
a little bit for our conversation. Help us understand what kind of use case where you've seen
spk_0
that institutional knowledge be able to be transferred to an agent and therefore automating work
spk_0
that needed a human before. I'll use one in its come up multiple times. It's not specific to any
spk_0
customer. I'll use the sales process. Whether that's reaching out to a new prospect or responding
spk_0
to an imbound inquiry or more often, let's say a contract is up for renewal or customers
spk_0
term for their contract is coming up. Oftentimes that's very much dependent on the sales team,
spk_0
how they address that right. They get a sales force or a sales force report or whatever system
spk_0
they're using. Then it's really up to them. Who do they engage? How do they engage? What channels
spk_0
did they use to engage? Oftentimes, if there's a high turnover on a sales team or folks are moving
spk_0
around and it counts, which happens everywhere, there's not a lot of knowledge of that customer.
spk_0
So we see that as an area. We see a lot of use cases where customers saying, I want to
spk_0
leverage the knowledge bases that I have. I have different systems that I use to drive sales. I
spk_0
have customer 360 data. How can I use that more intelligently? And most often, I'll stop talking
spk_0
as instead of reaching out to 20, 30, 40% of those customers, how can I touch 100% of my customers
spk_0
using automation and agetic workflows? But do that intelligently, leveraging
spk_0
leveraging knowledge that we have on those customers or institutional knowledge across the sales team?
spk_0
Yeah, that really resonates with me because you can think of it in extremes, the 100% human piece.
spk_0
You might have a rocking sales team that has a really great playbook. They have a great regional
spk_0
sales manager or something like that. But they're going to apply that playbook all the way across,
spk_0
right? MedPick or hey, at our company, we do it like this and this and we reach out after these
spk_0
many days. And you may actually, you know, lose some of the context, especially if I'm a new rep
spk_0
ramping up. I might do some of the same tasks that the last person before me just did. And now the
spk_0
customer isn't happy either instead, you're feeling irritated, right? So there's a side where a human
spk_0
is not enough. You need that nuance and history of how that outcome has gone. And then to your point
spk_0
on the other side, if we tried to do this with all code before agents came, the amount of branching
spk_0
logic that we'd have to write and how engineers have to be experts at the sales process, it becomes
spk_0
really brittle. And that's why people don't end up doing it. There's a long tail of automation of
spk_0
if this, then that at three or four or five different levels that are just really painful to do
spk_0
with traditional software. Now you picture flipping this, give an agent instructions like you would
spk_0
and you rep and give it access to your CRM and all this information. Now you can say, look, this is
spk_0
the last time we did a checkpoint with them. We have it done in EBR in a while. And my playbook says
spk_0
to do that every quarter. And by the way, if that's already happened and somebody's reached out in
spk_0
the meantime, so on and so forth, you give it those instructions. And now the agents doing that job
spk_0
of creating its own branching logic and an engineer doesn't have to write that out ahead of time
spk_0
and maintain it if anything changes. So yeah, that really resonates with me in terms of back office
spk_0
and that really transcends every industry. Every industry has that that need no matter what
spk_0
their what industry they're in. That's right. You could be any enterprise process, right? This could
spk_0
now be HR, how we hire people or go through a performance process. This can be marketing.
spk_0
Our developer. Yeah. Is that right? Yeah. That's awesome. And you know the complexity there too,
spk_0
right? Upskilling people looking across the board, hiring and moving on and all that kind of thing.
spk_0
So that really works for me as that sort of second example. So let's flip the script a little bit.
spk_0
Third example you gave us is now we want to leverage AI to better serve our customers,
spk_0
not just to make our company more efficient, but actually to improve our offerings. So
spk_0
what have you seen on that side? You know, as you're on the front lines here, have there been any
spk_0
kind of particular patterns of where customers decide to use agents that makes sense?
spk_0
I'll answer a question maybe within the Rommel Agency. I'll start with MCP servers. So I work with
spk_0
a lot of customers, a lot of teams that have very mature APIs. So there, B2B or B2B to see,
spk_0
maybe not B2C, but they're working with businesses or indirectly those businesses are then
spk_0
having consumers consume their services. And oftentimes something is simple standing up an MCP
spk_0
server. So being able to communicate with that API using natural language and then having their
spk_0
business partners stand up an agent which communicate with their MCP server. And that often can
spk_0
be a fairly light lift for a company that has a mature engineering team that has mature API.
spk_0
That's not a huge stretch for them to be able to add that capability. So we see that as time.
spk_0
Ineasy path, those are obviously for customers that have that ability.
spk_0
So basically modernizing the way that software can talk to software.
spk_0
That's right. If you've got agents on either side, just throwing out some vanilla APIs may be
spk_0
actually very difficult, setting up something like this, foster that collaboration.
spk_0
Yeah, let me just just another channel to consume those their resources, their intellectual
spk_0
property, their products in addition to an API. And you saw that we, and I know of several
spk_0
the cloud vendors have released agent marketplaces or MCP server market. It's just to make that
spk_0
easier to consume those services in the cloud. That makes sense. That makes sense. While we're on
spk_0
the topic of standards of protocols, you mentioned MCP for those that are less initiated to this world,
spk_0
what is an MCP? How is that different than sort of API approach that you talked about?
spk_0
And why do teams need to learn a new skill like MCP in order to sort of make this work in the
spk_0
world of agents? Yeah, and I would, and I'm happy to answer your question, but I would also,
spk_0
I love referring folks back to Anthrop because I think they've done a great job.
spk_0
It's kind of the original or that protocol and a lot of the thinking around that.
spk_0
Their documentation is excellent. Refer folks that. But I think the model to the context protocol
spk_0
along with some of the other standards that are evolving are important to customers, especially
spk_0
enterprise customers. You mentioned enterprise customers. I think the challenges AI,
spk_0
AI is a new gen AI is relatively agents are very new and in the absence of a lot of testing
spk_0
and evaluation and long-term production type architectures, I think in minimum, I'm seeing
spk_0
customers that want to adopt standards. So they want to make sure that they're adopting a framework,
spk_0
let's say an agentic framework, that they're adopting one which follows standards, which gives them
spk_0
some guarantee of robustness of performance of security that they're not using a proprietary
spk_0
system that will quickly evolve. And there's no guarantee, obviously. But I think that's the first
spk_0
question that we hear from a lot of customers is what are the protocols, what are the standards
spk_0
that this particular framework uses that I can rely on that's an industry standard?
spk_0
Yeah, that makes sense. One thing that we hear a lot from our customers and folks that we talk to
spk_0
is also this idea that as much as we talked about so far serving developers and building new ways
spk_0
to go do things, we use that sales use case where we're actually seeing a third party as well,
spk_0
really getting involved. The sales forces of the world are also building agentic platforms on
spk_0
their own systems and so on. So what you start seeing is a proliferation of first and third
spk_0
party agents hanging out in an enterprise that really need to start working together.
spk_0
Gary, tell us a little bit about this multi-agent world and how agents have to talk to agents.
spk_0
I think MCP is a part of that to your point, but what have you seen when it comes to orchestrating
spk_0
a bunch of these different players? If we think about the analogy of humans, this is when
spk_0
sales need to hand off to marketing or product needs to hand off to customer success,
spk_0
this is what humans might do. How do we help these multi-agent systems perform really well also?
spk_0
Yeah, that's right. I think we think about that so when we engage with a customer,
spk_0
it's really working backwards from that existing human structure right that typical
spk_0
management hierarchy structure, specialization structure. So I may be very capable performing
spk_0
multiple tasks, but there's other specialized tasks where I need to rely on my peers.
spk_0
And oftentimes that's a good entry point from moving from a single agent.
spk_0
An agent that has multiple tools available, I'm a carpenter. I have a dozen or two dozen tools
spk_0
that I can use and I'm confident in versus I'm not a plumber or an electrician. I need to reach
spk_0
out to my peers in order to perform those specialized tasks. So when we engage a customer,
spk_0
we try to understand what are the processes, who are conducting those processes, and are they
spk_0
specialized? So is there a logical separation of concern from a human and the loop hierarchy
spk_0
that we can translate into agents? Does it make sense? Because oftentimes they're very compatible.
spk_0
Right? If there's a natural segregation of duties there because of specialization,
spk_0
then that's probably a good sign that you want to move from a single agent to a multi-agent
spk_0
architecture in my opinion. Yeah, that's a great analogy, right? Let's say I'm a marketer and I want
spk_0
to run a campaign against my most loyal customers. And what does that mean? It's an orchestration of
spk_0
how much they use the product and maybe how long they've been with my company, what their spend
spk_0
might be and so on. So for me as a marketing human, I want to reach out to all these other humans
spk_0
or departments in my organization to figure it out. So let's say now I am an organization that wants
spk_0
to get into agents and I want to say, look, I want to bring that into technology. I heard
spk_0
Will and Gary talking. I want to automate all that. So what should I consider as an organization
spk_0
to sort of take that process and transform it? Right? Do I have to learn anything or do my teams
spk_0
need to know anything about that sort of agent to agent communication? Yeah, I mean, I think there's
spk_0
obviously we have customers that do it all themselves. They're very competent and technical. There's
spk_0
certainly other customers that are extremely successful and maybe to some degree focus more on their
spk_0
core business and I'll just that right to with a lot of great partners out there and folks that can
spk_0
build that for them to build those systems. But I think it's really, it's really comes down to
spk_0
understanding what the problem is. What are all the tools that you're using as a human today?
spk_0
What are all those different capabilities? What are all the systems? You know, we often look at
spk_0
what are all the data sources that you touch? What are all the systems that you touch those APIs
spk_0
trying to document all those? And you know, through that documentation, you're really kind of
spk_0
drawing out the architecture for that agentex system single or multi agent agentex system
spk_0
in really diagramming out the existing process and then overlaying that with where can we implement
spk_0
automation that makes sense and that there's an ROI on the customer.
spk_0
So I mean, that's a great strategy. I'm curious as we kind of get into implementing that then in
spk_0
sort of like building those systems a matter. Are there tools that we should be considering or
spk_0
thinking about that facilitate that sort of agent to agent conversation? Let's say I did build a
spk_0
marketing agent that knew everything about my campaigns and I wanted it to be able to talk to a
spk_0
sales agent that had all that context about the relationship. How would I go about and do something
spk_0
like that? Yeah, I think, you know, I was going to say six months, but that's probably too far
spk_0
back with agents. But even three months ago, the answer probably was you would need to build something,
spk_0
right? You would need to maybe you already have existing code or methods or functions to connect
spk_0
to those those secondary systems. And it could be as simple as annotating those tools. But I would
spk_0
say today, and I think you touched upon this before kind of with the explosion of let's say MCP
spk_0
servers a lot of the systems that are very common in enterprises have MCP servers or they're being
spk_0
developed very quickly. You know, you mentioned Salesforce and similar systems. So it's it's probably
spk_0
easier today to plug into an existing MCP server and do that in a secure way with that agent and
spk_0
give that agent those capabilities without having to write code or having to go back and even
spk_0
something like samples annotating existing code for for the agent. So I think that's quickly evolving,
spk_0
but it's becoming easier and easier to connect to those third party systems as we see vendors developing
spk_0
interfaces into their systems that are gentick interfaces into their system.
spk_0
That's really great. That is really the the culmination of what we said earlier, right?
spk_0
MCPs are kind of a new way to expose this information functionality of the product. It's much more
spk_0
than saying these are API 2.0. You know, this is actually almost like creating a GUI for your agent,
spk_0
right? This is how agent understands how another agent's built, how it can talk to it, how it can
spk_0
work with it, and so on. So that's really great guidance for anyone who's looking into this and
spk_0
saying, look, I've got a multi turn. I've got this whole ecosystem of different actors here. I want
spk_0
to make sure they talk to each other to do a particular task. You know, the other sort of standard
spk_0
protocol that we tend to see a lot is this idea of being able to see all of that activity. So now
spk_0
that we have those agents, let's say we got them talking to each other, how do we know that the
spk_0
handoff is successful? What were they saying to each other? And did they call each other unnecessarily
spk_0
or did they do it well? So we hear a lot about open telemetry or hotel for short. The ability to
spk_0
see in sort of monitor what these folks are doing. Tell us a little bit about that standard.
spk_0
Have you seen this sort of help enterprises as they're trying to stand up these solutions?
spk_0
Are there other ways to think about observability? How do you see it?
spk_0
Yeah, I was kind of laughing because I have a customer. I have actually a lot of customers that are
spk_0
into open standards and use both first party AWS, but many use third party observability platforms
spk_0
and hotel are important. Having open standards is very important to them. And it's nice to see
spk_0
strands agents, which is AWS's open source framework, but many of the other frameworks have
spk_0
adopted the open standards, which I think is great. And it shows the importance of those open standards
spk_0
for that. I think what's interesting and I find myself and maybe other folks on the webinar
spk_0
today too is real time logging is more important to your point. I built a system single agent,
spk_0
multi agent. I have multiple tools and I get to an answer, but how did I get to that answer?
spk_0
Did it? Did it? You know, I would expect it to maybe make two or three calls, but it made 15,
spk_0
or I had a funny use case I was developing a system and I got an answer back. And it was only
spk_0
later that I realized actually that was the LLM answering directly. The agent had not actually
spk_0
used any of the tools to go out to the web and find the answer to the knowledge base. It tried to
spk_0
answer the question without the tools and had I not use the observability tools were available and
spk_0
looked through the logs. I probably wouldn't have realized that. So I think that real time logging,
spk_0
especially during the testing and the development phase is critical. Not only to make sure it's
spk_0
working, but how is it working? Is it even though it's non-deterministic to some degree, it's still
spk_0
deterministic in that you would expect it to call certain tools to call certain tools maybe in
spk_0
a certain order or if it's calling too many tools, right? You know, if it's going a minute or two
spk_0
minutes and you get that time down, how can you do that? So I think observability is critical,
spk_0
critically important in agentic systems. I think you got it out of the box. A lot of these agents
spk_0
are people pleasers by nature, so to speak. And they kind of their job oftentimes is to tell you
spk_0
what you want to hear. So that's where these tools that you talked about being able to see what happened.
spk_0
It was a synonymally or not. And then going further and actually tuning what you want to have
spk_0
happened, maybe that's more explicit instructions in your prompt. Maybe that's actually behaving other
spk_0
parameters that will help you sort of stay more on rails or get more creative. There are a lot of
spk_0
things that we can play with to sort of manage agents, almost like we manage our employees, right?
spk_0
Give them coaching and say, look, this worked out well, this was a bad example man. Like, let's do
spk_0
it a different way next time. And the nice thing is, of course, with agents is you can encode that
spk_0
and make sure that they do it that way going forward. Yes, I think that's a lot of field not to go
spk_0
up to answer, but how are you encode that right? That system prompt, that agent prompt,
spk_0
and the set of rules that you give it. And I think similar to any other system,
spk_0
right? Rules when there's a failure rules and you know, there's there's a there's a there's a
spk_0
degree of complexity. And I think that's still evolving in terms of what direction you give the agent
spk_0
and what context you give the agent in advance of using the tools. So that's a perfect segue,
spk_0
because you're kind of mentioning it in this example, you're not just a strategist and consultant,
spk_0
you're also a builder, right? And you've gotten hands on with this stuff. I think a couple of
spk_0
examples that I'm aware of is a perplexity like search, you know, whether it kind of function,
spk_0
tell us a little bit, you can choose either of those examples or maybe walk us through one of them.
spk_0
Yeah. What have you learned from a builder point of view about, you know, this sort of agent piece.
spk_0
And more importantly, you know, how you kind of do this orchestration to your point,
spk_0
how do you know what good looks like compared to in the past when you wrote code and expected it to be
spk_0
performed the same way every time? Yeah. Can I pick a third example? Not to go up please.
spk_0
So we just did a talk at LA Summit last week and we and it's a recommendation, a personalization
spk_0
engine using an agentic approach. And that's probably just top of mind for me because I spent quite
spk_0
a bit of time on that. So we get a lot of a lot of requests for not how can I replace my existing
spk_0
personalization or recommendation system, more traditional machine learning approaches, which
spk_0
worked very well. But can I augment that with generative AI? Which, you know, today I would say it
spk_0
really, really more of an agentic approach, which incorporates generative AI into that. So spend a
spk_0
lot of time building out a system. And I think to your question, what did I learn? The first one,
spk_0
and it was part of the talk is a majority of that system is not AI. A more majority of the system
spk_0
is kind of those mundane. How does the customer log in? How do I allocate the customer? How do I
spk_0
load balance the requests? How do I store the customer data? So the first 80% of that system is not
spk_0
agentic. It's really more of a feature of the system that I can plug into the system to give it
spk_0
agentic capabilities. And I think that gets lost right as we, there's such hype around agentic
spk_0
AI and generative AI. You would think that that is the application. But the reality is that's more of a
spk_0
feature of the application of a smaller part of the application, the entire application that you need
spk_0
to think about when you're developing, especially a new product. But yeah, no, it was I think going
spk_0
through that process. I think I talked about the one example right, getting answers back,
spk_0
and not the observability that I was collecting in the metrics that I was collecting the
spk_0
telemetry. And then realizing, you know, it's not actually calling the tools. I think it's calling
spk_0
here in some cases. It's using the LLM to try and answer it with its limited knowledge,
spk_0
with the context of the knowledge that the model had through its journey versus actually using
spk_0
the tools and how do I instruct it to use the tools. And I think the one last thing, the one
spk_0
pattern which was interesting and I haven't seen a lot around that yet is I have two tools. And
spk_0
in you're seeing this a lot often, there's always limits, right? There's token limits, there's
spk_0
number of call limits on a lot of these third party tools. So I enabled it with two tools. Now, how do
spk_0
I load balance those two tools? I have two external sources of information, the relative of the same,
spk_0
or maybe I'm using them within certain contexts is how do I directly agent to use those tools in a
spk_0
particular way without dictating the order. So I think there's a lot of work around that, especially
spk_0
when you have a multi agent agent system, multiple tools, multiple MCPs. And there's definitely
spk_0
some coordination needs to happen through prompting and rules beyond just defining the tools. It's
spk_0
how do you use the tools effectively? One thing I find so interesting about that is it kind of changes
spk_0
the role of testing where if you think about traditional software development, testing is about
spk_0
making sure that happy paths work the way they're supposed to. Like if we already kind of know what
spk_0
should happen. And to an extent software software, we think that's of agents to there's some tasks that
spk_0
they have to complete. But the path way to that is also deterministic to your point in traditional
spk_0
software. So when we write things like unit tests, we say, look, I wrote the code to do x, y, and z,
spk_0
make sure that the outcome happens the same way and then look for anomalies in my system, right?
spk_0
Resources that weren't there, et cetera. Agents are a little different because you're not trying
spk_0
to pre-describe a path to your earlier point. How do I tell it the tools that it needs, but not be
spk_0
so explicit about when to use them? It's about where they're dependent on me in my instructions
spk_0
in order to win. So I'm curious, when you think about tuning up that system as you're getting ready
spk_0
for prime time or production, what are some tools that you've used to be able to
spk_0
a, understand how your app is working at any given moment? We talked about observability,
spk_0
but I'm curious if there's anything that you've used to actually improve that process.
spk_0
Basically tune up your agent before you release it out to the wild with customers or employees.
spk_0
Yep. So I'll tell you what I guess were my thoughts or what I'm trying to develop. So I think
spk_0
in addition to traditional testing, right? Performance testing, unit testing, integration testing,
spk_0
and again, that application AI is just a feature. So all of that conventional testing is still
spk_0
important, right? You have methods which are annotated as tools, but there's still methods which are
spk_0
testable. I think another area of research, which probably needs more thought, is around to your
spk_0
point testing. And even though maybe the route that the agent takes or agents take is non-deterministic
spk_0
in terms of the tools it uses, there's still probably a logical, if not ordering a logical
spk_0
process that it's taking right, I know it has to call current time and date. I know it has to
spk_0
create a sandbox and execute a script. I know it has to reach out through this web tool
spk_0
or to this third party API. And I think within the context of that, you can write, I'll call them
spk_0
non-deterministic test, right? You may not define the exact order, but you know some of the steps.
spk_0
And I think that type of testing, and I'm sure there's research into that, I find myself trying
spk_0
to write those type of tests. Kind of rejects test, right? And conventional. Yeah.
spk_0
conventional sensor, just kind of rejects and to make sure it's calling the right tools.
spk_0
And may have called it three times. I don't care as long as it called it once. Because if it didn't
spk_0
call it that could point to an error over a call to a tool, I didn't expect why is it calling that
spk_0
tool? Maybe that's a sign that I need to go back and retool my prompt to read, prompt engineer,
spk_0
my system prompts. You know, you've got me excited on this topic because I think a lot of what
spk_0
you're describing, it almost blurs the lines of how we coach people versus how we write software.
spk_0
Right? A lot of times we tell high performers, look, I don't care how you get there, I just need
spk_0
you to go do these things, right? You know, this is really important in this project to make sure we
spk_0
cover one, two, three, four things. Look into it. Let me know what you need to do. Let me know if
spk_0
there's some blockers, but otherwise have a good day. And then, you know, when we're done with this,
spk_0
give me a report, right? We often give instructions to agents in a similar way. Look, you've got to
spk_0
accomplish these things and you have these tools at your disposal. How you use them, feel free.
spk_0
You know, we trust you. Go ahead and try it. And that I think is the thing you've got to see, in other
spk_0
words, not just giving those instructions, but starting to understand how your agent interprets those
spk_0
instructions. And whether or not you need to put more guidance around them as a second, third,
spk_0
fourth round, and kind of go. That's right. That's right. So here's what I'm thinking about as you're
spk_0
sharing that then. Let's say we're getting to that point. Okay, we've augmented our instructions.
spk_0
Now we know, okay, in this case, you know, Billy the agent, you have to go do this instead of that.
spk_0
And we feel like we've got a pretty good instruction set here. And pretty good tool calls and things
spk_0
like this. Now we have to think about putting this in production. So tell us a little bit about that
spk_0
as you're thinking about putting these solutions, whether it's a personalization engine you talked about
spk_0
or other things. What are some pitfalls that we might encounter when we're trying to scale this up
spk_0
to many, many users? Yeah. So a couple, couple thoughts. One thought is, and we're all guilty of this,
spk_0
you know, with the excitement around a generative AI and agents in all these evolving technologies
spk_0
is oftentimes you're only focused on the POC or the MVP. And in some cases, that is the
spk_0
that is the deliverable right, whether we want to admit that as vendors, sometimes the objective
spk_0
was to create a POC to prove it. We could implement AI sometime in the future. But for for for
spk_0
folks that want to develop a solution and go into production, I think oftentimes with AI, we're
spk_0
not putting enough thought into Willis scale right? Am I picking the right model? I'm not saying
spk_0
build out a production architecture for a POC, but also I see sometimes a lack of thought in
spk_0
terms of how will this scale? And now you have a successful POC, but now you have just as much
spk_0
hard work to try and productionize it because there wasn't a lot of thought put into that during the POC.
spk_0
If that makes sense. It makes sense, you know, one of the core examples that are coming to mind for me
spk_0
is identity and access. You know, we talked about back office just now. And as a developer,
spk_0
oftentimes what do we do? We say, okay, in order for my agent to work, I'm going to give it
spk_0
God mode powers. Right? Here's this admin token that can access anything and everything because I'm
spk_0
just trying to build this, right? I don't want to worry about all that. But then we release a
spk_0
to a production and we realize all of the calls, like all of the Slack messages that are being sent
spk_0
or all of the emails that are being, you know, afforded or whatever it is or documents created and so
spk_0
on. They're all created by ability agent. It's not actually by the users that are doing it. So now I
spk_0
have no insight on who does what now, given people powers that the production have in the organization,
spk_0
I'm really stuck. So being able to build agents that can inherit the access of the user requesting
spk_0
things or can intelligently hand off to different folks in the organization, that's really difficult.
spk_0
You know, that's an example of production issues that may occur. I'm curious if you encounter data
spk_0
or any other kind of similar sort of production things that you don't think about when you're building,
spk_0
but you think about when you deploy and scale. Yeah. Yeah. So maybe two thoughts. One is,
spk_0
and certainly not to disagree with that, agree with this complex, but I also think and I see a lot of that.
spk_0
Let me put it in the right. A lot of those, a lot of that thought has already gone into distributed
spk_0
systems into micro services based architectures, right? I have multiple micro services. Each one of
spk_0
those have a specific role, just like an agent. They have different permissions, right? So a lot of,
spk_0
I think a lot of oftentimes we're sometimes guilty of ignoring existing architectures because we
spk_0
think AI is totally new, but there's a lot of people that argue a lot of those patterns are just
spk_0
repeating themselves. So I think looking at how do we, how do we secure distributed systems? How do
spk_0
we secure micro service architectures or clusters that talk to other clusters, multiple clusters? We
spk_0
have a lot of customers that do that right there, talking across the counts across clusters. A lot of,
spk_0
so a lot of I don't think that thinking is brand new. I think we can leverage a lot of that
spk_0
existing learning that teams already have and how to secure those systems, how to make sure that
spk_0
they're redundant that they can take failure well those type of things. The other thing, and I
spk_0
think you and I talked about ahead of time, they don't want to make this all about AWS. And I'm
spk_0
sure there's competitors out there and there's a while there's other products that are evolving. I
spk_0
think what's really unique about that is that it's looking beyond just AI, generative AI in the
spk_0
agents, but how do I deploy those and manage those at scale? And I think that's obviously an area
spk_0
of interest for a lot of vendors is moving beyond just having an agentic framework, but to your
spk_0
point, how do I how do I deploy that? How do I scale that? How do I think of things like memory,
spk_0
authentication and authorization that you mentioned? Securing how do I ensure that if I'm going to
spk_0
reach out to a third party that I do that securely and use standard authentication and authorization
spk_0
methods? So I think agent core and again, I'm sure there's there's there's similar products that
spk_0
are coming out. They seem to be fairly first a market pad. I think that's a really interesting
spk_0
area, right? So frameworks that allow you to production wise agents and agentic systems is is a I
spk_0
think it will be a growing area. Yeah, I think that's really smart because to your point, there is a
spk_0
common infrastructure to agents that we don't see for other types of software. Like you said,
spk_0
you know, real time memories really, really important. How do we sort of spin that up and make it work
spk_0
well? Similarly, how do we think about GPU support? I don't normally use GPUs when it comes to
spk_0
writing software, right? So even spinning up a new cluster or load balancing has multiple
spk_0
dimensions now. It's not just sort of how quickly can I spin up another ECU to instance, for example?
spk_0
And so I'm totally with you having some of those tools that allow us to productionize the software
spk_0
itself is really important. I think that's a net revolution right is agents have come into being
spk_0
and are evolving very quickly. I think you've seen this in other technologies and I think
spk_0
that'll follow that same path to be able to production wise is easier.
spk_0
Yeah, I think that's really smart. So here's where the you know, I'm processing what you're saying
spk_0
and it makes me start thinking about this tension or dichotomy that I hear, which is to your point,
spk_0
agents have a bit of a superhero Kate B. Says and we say anyone can build agents, right?
spk_0
Hey, you're the HR person. You should build the agent, not an engineer like you know the business
spk_0
process and we've kind of seen this rinse and repeated with tools that you know, I won't name specific
spk_0
companies, but folks that say, hey, you can build no code B2B software or you can build no code
spk_0
integrations and things like this and try to like move that technology and abstract it so that
spk_0
the business logic can sort of take center stage. I'm starting to see this pattern with agents too.
spk_0
Yet there's a tension that's created because of what you just shared. They're very much computer
spk_0
science like software engineering techniques in order to make great agents. So how do you
spk_0
how do you sort of address that tension as you're working with some of these great organizations
spk_0
and they're trying to get the stuff out to the enterprise. They want non-technical users to
spk_0
define the agents, but you need this sort of software skill in order to build ones that will serve
spk_0
the enterprise well, right? So how do you address that for them? Yeah, maybe comment first. I think
spk_0
we're all guilty of that, right? AI, machine learning, even data analytics. It's a very complex
spk_0
subject and I think you don't want to overcomplicate it if it's not necessary to solve a problem,
spk_0
but I think we're also all guilty of under simplifying it, right? To your point, we simplify to a
spk_0
point where we hide the complexity and then the complexity becomes apparent later. And that's what's
spk_0
down POC's moving into production often, right? The customer didn't realize the POC was easy, right?
spk_0
They to your point, they they they clicked a few buttons. They're able to launch something in
spk_0
testing, but now they actually want to scale it up. Now you need to understand a little bit more about
spk_0
the underlying technology. So I think that's I think we're all guilty of that. I think the
spk_0
entire industry needs to kind of have a better balance of explaining complexity and not under
spk_0
explaining it to to to to to a point. Having a connection earlier point, which is, you know, agents
spk_0
and Gen AI are an incredible capability. There are new things that we've never seen before and
spk_0
transforming the way software is built, but it doesn't mean that it replaces the whole process.
spk_0
Building great solutions still involve everything we've known before. This is additive,
spk_0
rather than, you know, strictly a ripping replace at least as it is today. So you've really got to
spk_0
use both pieces together in order to build something awesome. I mean, so we'll just go back. I don't
spk_0
think I fully answered your question though. Like in terms of the low code, no code or getting people
spk_0
involved. I think we're those people and those people are probably more critical than folks like
spk_0
myself or you that are technical in nature is they're specialists, right? We're trying to represent
spk_0
not replace them and maybe replacing but but not necessarily replacing but augmenting them in
spk_0
order to do that. It's more important understand what their specialized knowledge is, right? And
spk_0
that may not be technical. They understand the company. They understand the products. They understand
spk_0
how they would solve the problem and we want to view the genetic system with that knowledge. So I
spk_0
think they're critical to the process. It doesn't matter how many technical people have. If you don't
spk_0
have the domain expertise, the specialized expertise, you can't you can't build that system
spk_0
and replicate that domain knowledge within the system. Yeah, strongly agree. That was a point
spk_0
you made earlier around. That's the point of all, right? There's a transfer a lot of that
spk_0
special sauce to the agent and also to keep the agent accountable and honest, we're still going to
spk_0
need that human in the loop that that allows us to sort of make sure that that works well. So
spk_0
I totally agree with you. It's a holistic thing rather than saying either technology can replace
spk_0
the people or people you're just going to make it and technology is just agnostic and we don't care.
spk_0
This sort of coming together is really critical and I would really agree with that.
spk_0
So if we zoom out a little bit so far, our conversation has been about how do we identify
spk_0
problems that agents can help with? You know, what are the capabilities themselves when we talk
spk_0
about agents? What can they do that previous kinds of software tools couldn't do? And now we've
spk_0
talked about the idea of nothing that we've built a solution trying to get it out there and
spk_0
getting it to scale. So let's imagine, you know, we're all through these three pieces. Now,
spk_0
the final part of our conversation is about, now that we're at scale, you know, we've done the right
spk_0
things. We've scaled well. We have right permissions in place. All kind of thing. And great.
spk_0
Now we have thousands or even millions of users coming to our customer service solutions or
spk_0
to our marketing solutions or whatever it is. How do you protect your tools as they go forward?
spk_0
You know, one thing that we soften the poll from our participants here is that, you know, safety,
spk_0
compliance, risk, this is a big part of their story. So how do you think about making sure these
spk_0
systems are safe, especially because they have that degree of non-determinism? We don't always
spk_0
quite know what they're doing. So how have you advised different teams or companies in this regard?
spk_0
Yeah. So I think two parts that answer. The first part is not a cop out. But I think, again, to
spk_0
some degree, a lot of what you're building is traditional software. And I say that to the point of
spk_0
we have systems in place, right? Mature development organizations have mature practices in place,
spk_0
DevOps practices, deployment practices, SDLC processes. We're able to efficiently effectively
spk_0
test these things. I think to your point, the non-deterministic part of this is relatively new to a lot
spk_0
of us. But I think it also follows those same practices, having pipelines, being able, a good
spk_0
example that comes up in I we see it a lot is custard, I have to say, customers, I feel like I'm
spk_0
saying negative. We all tend to like want to use the latest model, right? But if I change models,
spk_0
even if it's a minor version, my responses can be very different, right? The end result from the
spk_0
model, which is underlying if it's a reasoning model, which is underlying the agent, the behavior
spk_0
can be very different and totally change my system in some ways break the system. So I think
spk_0
being being careful about testing and also testing in advance, right? What's the next model we
spk_0
might want to evolve to? And testing that in advance and already building out those prompts and
spk_0
having a way to not only replace the model, but you may have to replace maybe those system prompts,
spk_0
which then speaks to maybe having better configuration management. Think of those prompts as
spk_0
configuration, think of the model configuration and how easily those things can be compartmentalized
spk_0
and modulated and replaced in a system as you roll that out to production. So I think just
spk_0
thinking through that, I think that's something we don't think about it here that complained like
spk_0
we switch models in the stopped working, right? But to some degree, you wouldn't do that with
spk_0
traditional software. You wouldn't change out a database engine or maybe more specifically like a
spk_0
package version of a particular framework you're using without thorough testing. But oftentimes,
spk_0
I think with AI with the enthusiasm around it, maybe we're jumping models to the latest model a
spk_0
little too quickly without thoroughly testing it. It's a great point because there's different levels
spk_0
that you talked about. You mentioned packages. I think the developers on the call will understand
spk_0
this. You're using some OSS package. You don't just go to the 3.0 release. You probably build a new
spk_0
build with that thing, try to go test it. Maybe go to behind a flag, try it out, make sure it's good
spk_0
before you go live. But to your point, we're a little bit quick to just switch over our value when
spk_0
we build agents and go. So one thing I'm hearing about with this configuration piece, it's true
spk_0
for apps when we think about overall commits or overall versions. But we also have to think about
spk_0
that infrastructurally too because these are some underlying components, right? Kind of sits in
spk_0
the middle. So when we think about infrastructure as code, maybe that's one thing that we need to
spk_0
think about, which is what framework are we using? Was there a new version of that? Is that something
spk_0
that we have to serve safeguard? So to your point, even though the processes are familiar to
spk_0
that build software, the way we go about them or adapt them to the agendic world can be different.
spk_0
So you need new tools to be able to bring that to pair. And if I could, I think especially with
spk_0
agents, I know just in doing a talk last week, I think in the three weeks of let up to doing the
spk_0
talk and doing the deck, I bet the three frameworks that we reference change at least three to four
spk_0
times, which is abnormal, but not for this, not for this quickly evolving technologies, right?
spk_0
None of the frameworks are 1.0. They're evolving quickly. They're adding features. Oftentimes,
spk_0
those are breaking changes. So I think more so, whether it's as simple as a framework or something
spk_0
is major changing a model or a family of models is using the same testing rigor. I think that was my
spk_0
first point that you would apply to any other software product you were building. Do you have any
spk_0
advice or best practices for how to rethink those processes in an agendic first world?
spk_0
Is there anything that needs to change or evolve as a result of how the software works?
spk_0
Maybe goes back to the point about oversimplifying a complex process,
spk_0
understanding the implications of changing a model. You know, a great example is,
spk_0
hey, I have this prompt, I tested five models, this one worked best. But the reality is,
spk_0
for each one of those five models, that prompt should have been slightly different and
spk_0
optimized for that particular model provider or that family of models. So I think sometimes there's
spk_0
maybe a little bit of an activity around just what can change, right? I have a system proper
spk_0
works rate. I'm not going to change it. The answer is that that prompt needs to continue to evolve
spk_0
as the system evolves as you add more tools, as the framework evolves, as the model changes.
spk_0
Those things are not static, right? There's just as dynamic as other parts of this agendic system.
spk_0
I don't know if I answered your question, but I think it does. I think if I was to reflect it back
spk_0
to you, what I'm hearing is the process for rapid iteration, we've learned a lot about how to rapidly
spk_0
iterate software and do it in a good way. Adding these additional tools that can measure
spk_0
non-determinate six systems and feed that into the process of regression testing or testing CICD
spk_0
different builds like that. That's how we have to think about it. Let's give us new signals
spk_0
into existing process. So we can continue building good software. I don't know if
spk_0
those are there yet. I think those tools are still evolving, right? That natural arc is agents come
spk_0
into being POCs, MVPs into production. I mean, there are some out there. There's some companies
spk_0
doing some great things, but to your point, non-deterministic testing, I think is still a growing area of
spk_0
research and development for companies to develop those tools, just like you saw it with
spk_0
Genre of AI and around testing models and evaluating models versus
spk_0
conventional machine learning models. Testing and evaluating Genre of AI models is very different
spk_0
in some degree. I'll echo you earlier. I'm not about to make this
spk_0
a informer for my company either, but for anyone who knows what Fiddle AI is, we deeply believe
spk_0
in what you're talking about, both that it's early in terms of building the tools that will help
spk_0
people build great, ingenetic software, but also that this is a core need that people need to
spk_0
happen. To that point, I think the other trend that I would say, and maybe pick your brain on a
spk_0
little bit, is the idea of preventative medicine. Because these tools and systems are non-deterministic,
spk_0
on any given output, we don't quite know what it will be. It's very good to be able to observe
spk_0
and rapidly iterate, but that's usually downstream from the actual action that's happening, right?
spk_0
So how do you actually stop the bad outcome from happening in the first place?
spk_0
For example, getting prompt injection or jail breaks or on our side, make sure that your agent
spk_0
doesn't leak PII or something like that. Are there any considerations that you've thought of
spk_0
there or any suggestions you would have for the audience when it comes to protecting them against
spk_0
those outcomes in real time? Yeah, I would maybe, and I see Karen's hitting us up for questions here.
spk_0
I would say the great starting point in existing, and they've been around us really not just
spk_0
specific to AWS, is guardrails. And most of the popular frameworks have access to guardrails or
spk_0
some great third party guardrails and obviously more mature products out there from cloud vendors
spk_0
and other vendors, but using guardrails effectively, I think is a great first step. I see a lot of
spk_0
customers that that's kind of an afterthought, but I think building that in right from the start,
spk_0
even if they're just for toxicity, maybe PII, and then build those guardrails up as easy as you need them
spk_0
as a good starting point. In other words, testing is super important as part of the flow how we build
spk_0
software, but also having some kind of runtime protection is equally important. You've got to
spk_0
have to balance those two totally agree. All right, well, with a couple minutes left here, Gary,
spk_0
let's see if we can answer a question or two for the attendees that are here. I think there's
spk_0
been a lot of good stuff that's coming through. So why don't we start from the top here? How do you
spk_0
decide when it makes sense to add an orchestrator instead of just extending a single agent?
spk_0
I'll tell you how I start. I tend to always start with a single agent unless it's very clear. I have
spk_0
two very distinct. There's a clear separation of concern. I tend to start in this is probably how we
spk_0
tend to develop software to with a single agent and adding tools to a point. Then I might go back
spk_0
and refactor and say, you know what, it makes more sense to split these tools off. This really should
spk_0
be a separate agent. There's a clear separation of concerns here. There should be a hyperplane here.
spk_0
Let's separate this in these tools and move those into another agent. But I think I tend to
spk_0
like, I tend to develop right. I just start writing code and at some point I split those into classes
spk_0
or modules. I do the same with agents. I don't know if that's the best approach. It's just personal
spk_0
kind of pattern that I follow. With customers, I think with more organized, right? We try to
spk_0
understand again who are the humans in the loop? Are they specialized tasks? And if they are,
spk_0
that's we tend to start there with developing probably multiple agents or some type of multi-agent
spk_0
architecture depending on how they're used to operating. Yeah. That flows perfectly in next
spk_0
couple of set of questions because I think people are really wondering about this multi-agent
spk_0
kind of approach. So the next question I've got here is when you've tried connecting multiple
spk_0
agents together, what has been the hardest part technically? Is it coordination, latency, or something
spk_0
else? I think it's coordination. So what worked well. I had a single agent with probably too many
spk_0
tools and logically I should have separated. It was working. And now I've separated that and now I'm
spk_0
having trouble orchid, the orchestration between the two specialized agents or the orchestrator agent
spk_0
and the two specialized ones. I've also tried swarms. So and I know that there's a couple frameworks
spk_0
that have that concept of no overall orchestrator that the agents are kind of self organizing. I've
spk_0
tried to implement that sometimes and it's probably not the best use case. So it's trying to understand
spk_0
maybe where a swarm architecture makes sense over a more hierarchical orchestrator, specialization
spk_0
architecture. Yeah, there's a trade-off there. For every benefit, let's say in answer fidelity or
spk_0
speed or expertise you get from a multi-agent framework, you now have introduced new edges to your
spk_0
graph. Now you have to worry about every handoff was the handoff correct and now you're testing those
spk_0
connection as well as the agentic behavior itself. So definitely a consideration of saying,
spk_0
do I need that separation of concerns to your point? I think that's a single most important thing
spk_0
is the nail of strategy and then the technology of connecting them is fairly straightforward.
spk_0
Maybe I'll just add without going to too much depth because probably longer conversation.
spk_0
Also maybe a different model would make better sense for this task versus that task. So maybe a more
spk_0
specialized model that can address this particular task. That's maybe not the best reason to
spk_0
have two agents, but if I can split those functions off to a separate agent and the underlying
spk_0
model under that agent is something different that has a very specific performance characteristics
spk_0
that can solve that problem or it's cheaper, faster, that part isn't as common as other parts.
spk_0
Maybe I can get away with using a smaller agent or a faster agent. So thinking about a model,
spk_0
thinking about the underlying models and maybe using multiple models and using the agent as a
spk_0
point of separation between them. I think that's great advice. Semantically a lot of folks I talk to,
spk_0
I like to say chat GPT is an English major, cloud is a STEM major, think about it that way.
spk_0
You know, that's important. And then also do you really need clouds PhD or do you need a really
spk_0
smart bachelor to be able to solve this problem? A very small model or two model sometimes.
spk_0
Exactly. What are we doing? Four-row man-of-versus, you know, the full Fat-01, that's what's critical.
spk_0
So, doesn't Gary, it's been a pleasure getting to know you and talk to you about this, you know,
spk_0
front line of agents here. It's been so exciting to be able to discuss the front lines,
spk_0
but again, we just want to thank you for your participation and we'll be in touch really soon.
spk_0
Yeah, we'll thank you, Karen. Thank you. Appreciate the time today. All right, have a great day.
spk_0
Thank you.