Ah, well, the traditional problems that you have. I think
with every piece of software that you're trying to
productionize, it's fairly easy to build something
that works, that solves a problem. But as soon as you
put it out into the world, you have a few things. First of all, you
need to make sure it's secure. Second of all, you need to make sure that
it scales. Because if by all means I I hope for you
as a startup it actually works and people enjoy using it,
the next step really is how does it scale? How does it not fall
apart under the load of people wanting to
try? And the third thing
really is observability, being able
to really get telemetry, look
into what's actually happening. Foreign.
YouTube blog covering the German startup scene with
news interviews and live events,
AWS is proud to sponsor this week's episode of startup raid
IO. The AWS startup team compromises
former founders and CTOs, venture capitalists,
angel investors and mentors ready to help you prove
what's possible. Since 2013, AWS
has supported over 280,000 startups across the
globe and provided US$7 billion
in credits through the AWS Activate
Program. Big Ideas Feel at home at
aws with access to cutting edge technologies
like generative AI, you can quickly turn those ideas into
marketable products. Want your own AI
powered assistant? Try Amazon Q. Want to
build your own AI products privately?
Customize leading foundation models on Amazon Bedrock.
Want to reduce the cost of AI workloads?
AWS Trainium is the silicon you're looking for.
Whatever your ambitions, you've already had the idea.
Now prove it's possible on AWS. Visit
aws.Amazon.com
startups to get started. So you build a chatbot.
Cool. But now your team is stuck wondering how to connect it
to real APIs, make it reliable or roll it out
to thousand users. That's where this episode comes in.
AWS Dennis Troup walks us through how
to productionize AI securely at
scale and without breaking your app. We
dive into agentic workflows, MCP
and how real startups go from MVP to market.
Let's see in our episode today in cooperation with aws.
Dennis Straub is a developer advocate at aws. Focus
on on helping startups and enterprises to take their
gen experiments into real world deployment. With
decades of experience in secure infrastructure and
developer productivity, Dennis has helped teams across
industries automate, integrate and scale their
projects using modern cloud native tools.
Today we talk about model context, protocol, agent based
architecture and what it really takes
to make AI Work in production Dennis, welcome back
to celebrate. O oh thanks,
thanks for having me again. Totally my pleasure. For everybody who's
not aware of this, this is number two of a series of two
interviews. But actually one of your colleagues would join us as well.
So in total we have four interviews with you guys
here. Dennis
Productionizing AI Tell us about it.
What's the hard truth? What's the biggest gap you see between
Genai prototypes and action production systems?
Well, the traditional problems that you have, I think with
every piece of software that you're trying to productionize,
it's fairly easy to build something that
works, that solves a problem. But as soon as you put
it out into the world, you have a few things. First of all you need
to make sure it's secure. Second of all, you need to make sure that it
scales. Because if by all means I hope for you as
a startup it actually works and people enjoy using it, the
next step really is how does it scale? How does it not fall
apart under the load of people wanting
to try? And the third
thing really is observability, being
able to really get telemetry,
look into what's actually happening. Does it do what it's supposed to
be doing? Am I running risks in
terms of, for instance, running up a large bill
because it does something that I didn't expect it to do?
You may run into edge cases that you didn't have in your
prototyping environment. So the traditional problems
that you usually have when you're building something that was
small and you put it into the market, the second,
second thing that very often happens is as well is
you. Most, most pieces of software are not an island.
They have to connect to something. They have to connect to third party APIs or
to your own APIs, or to customers, internal APIs
and data sources. And that can be hard as well because
how do you make sure that the agent then first of
all securely connects to these APIs and
second, doesn't mess with them, doesn't
do anything that it's not supposed to be doing. That is something that you really
need to look at. As soon as you run into production production situation, you
will most likely have security, scalability
and connectivity issues with
internal and third party tools. Talking a little bit here about model
context protocol is getting a lot of traction in
AI engineering world. What is it and what does
it matter now? One of the big questions that everybody had in
regards to LLMs, to language models when they came out a few years ago,
really was there
a certain amount of unreliability especially when it
comes to them producing facts.
These models have been trained on an incredible
amount of data, but they don't really understand the data.
So by the nature of how these models work,
they don't really know whether what they're saying is
actually true or not. They just look
whether it matches a certain pattern that they have
seen quite often. And that's a really big limitations because in many
applications, in many use cases it's really important to work with
factual data and not with assumptions or with, with
made up information that
doesn't actually, doesn't actually represent the facts. That can be
pretty dangerous. So most, most people in
companies who've been working with LLMs, especially in their early days,
ran into this problem. Well, it doesn't really work because
it isn't really reliable in what it says, it makes up data,
it confuses things and so forth. So
there's basically two ways to solve this
problem. One way is
adding more information to the model, fine
tuning the model, giving it access to a
vector store so that it can do semantic search to retrieve
actual data to, to basically either
double check against actual data or just
use this data as part of its process, which is called
RAG retrieval, augmented generation. And I
don't want to go too deep into rag and I have a very
specific opinion of rag, not a bad one, but what it actually
is, I don't look at it from the perspective of a data scientist.
Actually I think RAG is the
second thing or belongs to the second thing that we can do which is
basically connecting the AI to the real
world add runtime. So
at the moment in time when I actually use
the model, pre training,
fine tuning and these things, they all happen before
I deploy the model into production and everything
after that happens when the model is in production, when my
application runs and RAG tool use
and other mechanisms basically solve this problem,
connecting the AI to the real world. And that could be a database,
that could be a vector store with a semantic search, but that could also be
something where the AI can actually act like call an
API to trigger a process or to send an email
or, or do things like that. And the challenge
was that every API works differently. Every
API has a different authentication mechanism, every database has a slightly different
SQL dialect and so forth. So all the tools
that I wanted to use, starting from a simple
calculator, all the way to a weather API or to
a third party SaaS, providers API or anything, they
all have their proprietary
API. So it was my task as a
developer to basically manually
code the connectivity the connection to all of these APIs
to be able to get the data or send a request or do whatever
to get it into the model or to have the model act on my process.
MCP Model Context Protocol is an open source
protocol nad has been
published and recommended by Anthropic. It's being widely adopted
by Google, Amazon, Microsoft and many others.
It looks like it's actually turning into an industry standard,
which is a good thing because it simplifies
that connection layer between the
actual tool or API and
your agent. It's a fairly
simplified protocol that contains a few primitives like tools.
So what can I actually do with that
API as a model? That tool could be
a browser engine, could be a runtime for me to run
code in a sandbox environment, could be an API to a third
party provider, could be
a piece of code that I wrote myself, could be basically everything. And with mcp,
you're able to just wrap this as a so called server,
it's an MCP server. And then you can use the client
library of the SDK inside of your own application
to just connect to the server and
include it into your agent so that you don't have to rebuild
it. For every single API that you to want to use, that's, that's mcp.
It has a few more primitives like it's able to send
notifications to the client, it's able to provide
resources, static resources to the provider, and a few more.
But the most important and most interesting use case for most
AI applications really is the fact that it can expose
its capabilities or the capabilities of the underlying
API as tools that the LLM can
understand and use. You said
before that AI systems are only as powerful as their
connections. What does it take to connect models to real world
APIs or workflows? I'm not sure how to answer
this question. You need, you need. It
takes a few things on a few different levels. Well, first of all, you should
be aware of what you want to connect to
and also the implications of
connecting to these things. For instance, if you connect to your,
to your email account, you give the model effectively access
to all your data, possibly including pii, most
likely including sensitive and personally identifiable
information. That is something that you just need to be aware of.
So once you connect your AI to this system and
connect it to another system, you need to
be aware of the fact that technically these two connected,
these two systems are connected through an intermediary,
but they are effectively connected. That is something. So it
takes, it takes thinking about
what do I Connect and do I really want
these things to be connected with each other?
We're talking about boundaries, we're talking about service isolation,
something that we've been talking about software architecture for a very long time time,
service isolation, just to make sure that a
service which could be an AI agent only does what it's
supposed to do and doesn't accidentally
expose information
to something else, even though it shouldn't. And this is
even more important with LLMs because LLMs are by
nature not deterministic. You cannot just
code the, the model so that it's
doesn't cross this boundary. You could try, but it's really
hard and I wouldn't, I wouldn't suggest you do that. So
looking at this, let's say I have two, I
have two systems and I need a connection between these two systems, one
being my email, the other being maybe a chatbot
that I provide to a customer as part of my website.
I wouldn't want this to be one thing. I wouldn't want this to be one
agent. I would want to separate these two, isolate these two. One could be
an agent that talks to the customer through a
chatbot interface that then realizes, well, we do what
that classifies the use case. Well, is this a complaint or is this an
order or is it a general inquiry? And then
if it's a complaint, this
agent may then send a message
to a different agent that is responsible for
processing customer complaints and does that. And
they don't share data between each other.
It's more of a handoff situation where the classifying agents
says, well, I call the compliant agent,
tell them there's a complaint from this customer, and this agent
comes back with maybe, okay,
thanks. Somebody's going to call you
and tells this the actual, the agent that's talking to the customer. And the
agent tells the customer, well, okay, somebody's going to call
you or could you give me your email address or phone number? Would
you like to be called or would you like to get an email? How do,
how would you like us to contact you? So the
negotiation with the customer and the negotiation with
the CRM with access
to sensitive information should be isolated. That is
something that I would have in mind
from the start if I wanted to do something like this.
I'm not sure. Did I answer your question? Well,
partially. I
totally do understand that we are very, very early in this whole world
of AI agents and how the systems work. And
currently I, without like global standards, I do believe
you can really completely
answer that questions. But let's talk about
Agentic workflows, how they are different from simple
prompt chains or aig
RAG pipelines. Sorry, RAG pipelines? Yeah.
So rack pipelines are completely different. Beast rack
pipelines themselves don't actually. Well, they actually involve
AI in a certain way, but not in the way that we talk about it
right now. RAC pipelines are more a
data preparation. How do you prepare your
data, your siloed information, your customer,
your CRM, your product database, whatever you have? How do you prepare
that so it can be used in a useful way
by an AI? The other thing, prompt
chains. Prompt chains can be part of an agent. So first of
all, but very often you actually don't even need an
agent. But a prompt chain could be enough. A prompt chain just being
you have a fairly deterministic workflow where you send something to an
LLM, you get a response, maybe you send a second prompt based on the
response, and then at some point the thing is done.
The way I look at it is I distinguish between three types of
agentic applications, or not agentic, three types of
LLM or
generative AI enhanced applications. The first being
non agentic. That are all the use cases
where you send something to an LLM and the LLM
responds and that's it. Or where you send something to the.
Or a chatbot, you chat with the LLM, the
LLM response and you have a back and forth between the person
and the model. That's non
agentic workflows, they can become quite
complex and complicated, but they are mostly
predetermined. Either they're a loop like in a chat,
or there's a sequence of steps that needs to
be done and this sequence is almost always
the same or maybe has some decisions in between that you can
do with traditional conditional
steps. The second being. So the first of the three
is non agentic. The second is agentic
AI. This is where
the AI, the model, actually makes
decisions, plans. Does
reasoning understands that it doesn't
have all the information it needs, asks you about
this information, or reaches out to one of the tools
via mcp, the tools that it has available to
get the information it needs for this specific use case, and is
able to adapt the workflow based on the
interaction, based on the available information, based on its
own reasoning. So very often an agentix system
starts by analyzing the
task, coming up with a plan,
maybe even storing that plan somewhere in the file
system as a checklist for itself, using a tool,
again using MCP, for instance, that gives it access to
a contained file system, a
temporary directory, basically where it can store intermediate
information. So it puts its checklist, its plan in there and
then it does something and then it goes back to the checklist, checks this thing
off and then it realizes, well, for this I need some more information from
the customer, goes back, chats with the customer
to receive this information and so forth. So an
agent basically
perceives, acts.
So it gets information from
either from the user through the prompt or through a tool, through a
database or something, gets information,
decides based on this information and then acts
within certain bounds. And these boundaries are basically
defined by the use case and the capabilities of this agent.
The third so non agentic, gentic and the third basically
being multi agent systems. This is where multiple
agents interact with each other to,
to provide for an even more
complex use case. This is something that
I talked about before where you may have an agent
that interacts with the customer through the website and is able to
kick off different processes depending on the customer and what they want.
And this agent then communicates with an agent that's responsible
for complaints and another agent that's responsible for ingesting orders
and so forth.
So it can become infinitely complicated.
I'm basically thinking about agents as like I think about
microservices. Actually I would say agent is
when the AI deviates from simple. If this, then that rules,
would that be a good definition? Even
complex? If this and that rules. There are
fairly complex workflows that can be
deterministically defined
where the entire work process, no matter how
complex the process is, is in itself
is deterministic and
algorithmic. So you can do it with step functions or you can do
it with a, with an orchestration engine or something like that. In
that case I wouldn't necessarily use AI, an AI agent,
I might use AI, for instance, as a front end to that process
that understands natural language. So if I want to have,
if I want to have, want to have the
possibility to use Slack, do
anything through Slack, I have an agent in Slack and I'm able to
tell the agent, please do this for me. In the past
I would have to use a very specific command format for Slack
Ops. And now with LLMs I can use natural language
and I just use natural language. Then I have the LLM as
basically the client and the
LLM is not an agent. It just takes the
request and has a list of processes, classifies
the request, extracts the relevant information and
kicks off the processes. It becomes
agentic as soon as, yeah, it becomes agentic as
soon as this thing
may have to make more involved decisions like
maybe we need more information or maybe we have to call
somebody. It becomes fairly complicated fairly quickly. So it's really hard
to talk about it. But what I'm trying to say really is don't build an
agent for everything. First of all, don't use AI
if the problem can be solved without
in a fairly easy way. Second, don't build an agent if it can
be done by a simple deterministic workflow,
even if it involves an LLM. And don't
build complicated. Well, let me say it this
way. If the solution to your problem is more complicated
than the problem you're trying to solve,
you are probably doing it wrong,
if that makes sense. I see,
I see. What does a typical production AI stack looks
like in 2025, especially for startups that are scaling fast?
Well, you need a few things. One is obviously model
serving. So you need the model
somewhere. Could be locally, could be something that you do yourself. As a
startup. I wouldn't recommend doing that as a startup. I would really
just, really just suggest you use an existing model
provider that provides the
model through an API. Could be Amazon Bedrock, for instance. We have lots of
different models and the list is growing. We have open weights model like Llama,
Mistral, Deep Seq, Others
we have commercial models like Claude, our
own Nova model family we have.
Now I'm blanking out. We have Cohere. We have many different models
that you can use for different use cases. We have many, also many
general purpose language models. So you could use Amazon
Bedrock to just talk to a model through an API, through
a secure API, while it's secure, so you
don't have to worry about what does the model provider do with your data. We
don't do anything with it. We don't even use it for model training.
We just provide the model to you so that you can use it in a
secure way, so that you can even build GDPR compliance
systems. And so that's model
serving. You may need databases, either your own
databases, maybe a vector store if you want to do some semantic search.
But that's advanced, I wouldn't start with that. And you need
something that orchestrates the process. So
something that basically takes the input
calls the actual language model. Because the language model itself cannot do
anything. It only takes text
and, or depending on the modality, we're talking about language models right now. So it
takes text and it produces something based on that text. It doesn't do anything.
So the orchestration engine connects to
MCP tools, to the front end, to whatever you
want to do, and The LLM and the orchestration framework
could be one of the many open source frameworks that we have, like
Langgraph is one or
Llama Index is another one. True AI is one strands agents.
This is one that we have open sourced about two months ago,
which is model agnostics, even provider agnostics. You can use
trans agents with models from OpenAI
or directly with Llama API or
even with Olama on your own machine. So you need an orchestration
tool or engine, you need a
model somewhere. You may need a database or some data
for the model to work with. You might want to think about
primary and secondary models that's a bit more advanced than as well. So primary
models is the general purpose model
that does the majority of the work. And then you may want to use
secondary models for instance for very simple use cases, so you don't
need to use the expensive ones. You can use very cost effective models
for simple summarization tasks, while you may want to use a
more expensive reasoning model for the overall orchestration for
instance. Another thing that's part of the stack is evaluation and
monitoring. And that is something where I really would say as
a startup you should put that in place as early as possible.
Monitoring. I think monitoring observability is self explanatory. You
should be able to see what's happening. And you should also very
early implement cost monitoring. Because if something
goes wrong, especially in a non deterministic system
like agentic AI system,
if it runs into a loop, it may run up
a big context that it sends to the recursively
sends to the LLM and all of a sudden it become very expensive. You wouldn't
want that. So please set up cost monitoring
very early. And also I would recommend
implementing an evaluation mechanism. Evaluation is
basically testing but full LLMs. So
you take a specific model and you have a number of prompts for your
system and some data that you get from your database or
through RAG or through mcp and you plug these things together and
you test them in different scenarios, maybe with different user
inputs to a point where in most cases you are
satisfied with the response. So you have you reach
a certain threshold of reliability of your
system to do what it's supposed to do.
All of a sudden a model provider
deploys an update of their model, a new version
that has been trained on different data or has been fine tuned in a different
way, which could break your
system apart because a variable, a very important
variable has changed. Or
maybe you change your prompts that
you use as part of the pipeline or Your data changes, the
structure of your data changes. This could all lead to
the fact that your overall system degrades in terms of
reliability when it comes to
how good the results are. And you can solve that by
implementing an evaluation pipeline so that whenever you change
anything, you run a
number of prompts or a number of use
cases against the system to see
if the reliability drops beneath your threshold. And
if it does, the test fails and you have to go look at it.
That's very important. If you implement something like this as early as possible,
just like with testing in general, you will be able to
to iterate with much quicker than
if every time there's a new model update, or every time the data structure
changes, or every time you update your own prompts,
the system falls apart because you realize it doesn't
reliably create the solutions or
the responses that I was looking for. Apart from
that, well, we were at the question, what's part of the stack? So model
serving, data access and orchestration,
then mostly primary model to start with. And I would suggest
just starting with a primary model, then evaluation and monitoring,
then maybe a data pipeline if you actually want
to use live data that changes over time. But again that's a fairly advanced
topic and obviously security and compliance. Whenever you
use sensitive data, whenever you use proprietary information,
make sure that you comply to your internal
compliance frameworks, to your customers, compliance frameworks to
legal compliance frameworks, make sure that you use proper
authentication, make sure that your agent can only do what it's supposed to
do, that your agent doesn't have access to your customer
database, while also chatting on the Internet with random
people and perhaps by
accident giving them access to your customer database.
That's important. Security and compliance, that's part of
any production stack and should be
because these are the basic things that you need to make
sure that you don't run into problems, most likely
earlier than later. How do you
guys from AWS support this kind of production
grade AI stack from bedrock to step functions to vector
DBs? Well, first of all we have a number of services
around the bedrock family of services,
and that is Amazon Bedrock itself, which
is first of all model serving,
where we provide secure and private
access to models from different providers, including
the current frontier models of most providers
where you can just use models and be sure that
your data is not being used for training or used for
anything else. So we basically run these models
inside of our own escrow accounts. They are air
gapped. Everything you send to the model
is not stored or reused for anything.
It's just sent to the model. The model's basically brought to
life, loaded into the GPU cluster. It
runs and then it returns the response and then the
models basically goes back down and all the data is gone
apart from the actual model weights themselves because they
need to be used for subsequent calls. So that is one thing.
Amazon Bedrock, which provides access to models,
including our own family of models, including
the capability to actually fine tune certain models
or distill models into
smaller models. If you want to use smaller models, basically you want to use,
let's say you want to use Llama Llama 4, but you don't want to use
the version that Meta provides. You want to distill this into a
smaller model. You can do that with Bedrock. Again, very advanced features.
I would not suggest starting with that. It's
time intensive, it's costly, it's a use
case for enterprises and it may be a use case for you
once you are further ahead on the road in
adoption. The second thing that we provide is Bedrock
Guardrails along with a few other capabilities.
Bedrock Knowledge Base guardrails is
basically to mask sensitive data or
to block requests that contain sensitive data or responses that contain
sensitive data or to block requests or responses, responses
that violate ethical codes that you've defined or something like
that. And Knowledge base is basically
direct access to vector stores. And we also provide
obviously services for vector stores with OpenSearch
or with Postgres on RDS. We've just
released S3 vectors, Amazon S3 vectors.
So you can even store your vectors on S3 as object
stores, but and use that which is extremely cost effective
if you compare it to traditional vector stores. Because traditional vector
stores have to be. They're basically database servers, servers
and they have to run and they cost money. And S3
vectors stores your vectors on S3. So you only
pay for storage, not for a machine that's running all the time. You pay for
storage and then of course for access. So it's, you
can, you can even. You can save up to 90% of the cost
compared to. To database
based vector stores. And then
we have Bedrock Agents, which is out of a box system that provides
agents in a fairly opinionated
way. So you can just build an agent using Bedrock
agents. We don't have to do that much,
but these agents are self contained inside of
aws. And the third thing, and that is something that's
just in purview since. Since a few weeks at the time of
recording. Maybe ga. When you listen to this episode. GA means general
generally available. It's In a public preview right now, that's
Bedrock, Amazon Bedrock's Agent
Core and that's family of services. That's very
interesting because it, it gives you all the
individual capability capabilities as building
blocks that you can use. That is, it
has access to Bedrock models, obviously, but it also, you can
also use this, use it to use Bedrocks, any,
sorry to use models anywhere. So you can
also use this, use it with OpenAI, with llama API,
with your own ollama or whatnot. And it's
provider, no, it's framework agnostic. So you can deploy
your CREWAI agents or your langdref agents.
You don't have to do it the AWS way.
The next capability is memory, because very often it's important
to maintain information across sessions. So when
I talk to the agent right now I wanted to retain information
about previous conversations. And that's a capability. It's called
Agent Core Memory. Again, Agent Core memory can
just be plugged into an agent that you run on Agent Core, but it can
also just be plugged into an agent that you run somewhere else. Again,
it's provider and framework agnostic. The third
capability is Tools. So right now,
as of now, we provide direct access to
code environment sandbox, which is completely isolated.
So if you, if, if your agent creates code or if you're the
user of your agent's agent sends code, the
agent can then just use one of these
sandboxes to run that code in a completely secure
and isolated environment. And right now it provides a
python runtime and typescript, most likely more in the future.
It also provides browser access to a browser
environment so that your agent can use the Internet again
in an isolated environment. Another capability, of
course, is Security Identity. You can do everything,
of course, using IAM identity and access management with
AWS, but you can also use OAuth with
any kind of OAuth provider or your corporate
identity provider, your commercial identity provider that
you're using anyway to make sure only the people who
should access your agents are actually able to access
your agents. And then we have, and I
realize I've been talking a lot and it's a lot of stuff. I'm going to
summarize that in a second. Then one of two more things. Observability
out of the box using OpenTelemetry, so you can use your
existing observability stack if you want.
And Gateway
Agent Core Gateway allows you to just wrap any
API that you may already have and expose it as
an MCP server, including discovery,
including even the ability to
sell your Own agent or your own MCP server on the
AWS marketplace to other AWS customers if you want.
So in summary, what Agent Core provides is all the building blocks that
you might need to build an agent memory,
a runtime for the agents and the MCP servers, a gateway.
If you already have your MCP or your server and just want to wrap it
as MCP uses, has identity, it has
observability, tools and memory.
That's what we. It's a lot, I realize.
Yeah, it is. I
was wondering for our audience, have you built an AI prototype that
almost made it into production? What blocked you
tag us with your war story?
We'll be back after a very, very short ad break
then is some startups are using multiple
foundation models at once. What's AWS's approach
to multiple model orchestration and
how do you manage that securely?
Multimodal usage is one of the core premises
because we say it doesn't make sense to use one for everything,
which is why we started bedrock the way we did in the first place. It's
not one model that you can use. You can use models from
many different providers with many different capabilities. Because
in many use cases you may want to use a general purpose large language
model, but you also may want to use a model to create your embeddings or
to create images. Could be from a completely different provider. Or you want to
use a reasoning model for involved tasks and a
much less expensive small model for basic
tasks like summarization or classification.
And these can be from different providers. Then there's models that are
specialized in translation, language, translation of languages.
There's models that may be specialized in code, creating
code. So Bedrock has already always
Bedrock and AWS has always looked at it through the lens of different
customers need different things, different models and individual
customers may need different models models for different use cases
or even inside the same use case. So I wouldn't
start that way. If I would just build a prototype,
I wouldn't start with multiple models, I would start with one to start
understanding the moving parts and how it works and what and the limitations.
But you can certainly use multiple models, and
in any kind of production application I probably would
because that helps me reduce cost,
reduce latency, because the larger the model, the longer it
takes for the model to respond, to take
these things into account. So yes,
multimodal, using different models, even from different providers,
is certainly something that I would suggest looking into
once you have your basic use case down
and once you go into well, how can I optimize cost how can I
optimize latency? Is there a specialized model that
helps me with certain tasks inside the workflow?
How do we make sure it's secure? Well, just like everything
on aws, everything goes through the
AWS API. So every model invocation goes through the
AWS API, which is protected through iam, through
identity and access management. So you can have
really fine grained mechanism to say
who or which service or which third
party or which agent is allowed, which process is
allowed to interact with individual models with intro
with individual data stores or tools that you provide.
For our audience. I was wondering what's one tool or
pattern that help you finally scale your project?
Shared on threads or LinkedIn and taxpayer o
Dennis, what's your take on LLM ops or
GenIML ops? Is it the same as
traditional ML Ops or something new?
I'm getting into hot waters when I start
talking about that because it's
not the same. So. And I, I'm not sure if. What did
you say? Gen AI Ops, LLM Ops,
ML Ops OP definitions
are in flow. Well, I'm pretty sure there is a definition for MLOps
and there's probably also a definition for LLM ops.
But the thing is that
with the democratization of generative
AI since the Chat GPT moment, effectively when everybody wants to
build on top of generative AI,
there's a new kind of discipline
emerging which sits at the intersection of
software developers and
data scientists and
machine learning engineers. And that's what's
emerging as AI engineers. That's the term
that's being used increasingly for this, where you
don't go build the models yourself,
you don't even necessarily fine tune the models and then
deploy the models somewhere and run them. You use
models, you build applications that use
these models and traditional approaches to build an
application that has both the intelligence
of a language model or any kind of generative AI model
and the, the, the
capabilities of the piece of software you build. So the AI
engineer understands how
LLMs works, understands the limitations, understands the difference
between models, but the AI engineer usually
doesn't deploy these models, doesn't build and train these models.
That's what happens, that's what the ML engineer does.
And when it comes to operations, I think it's very
similar. The LLM ops, or
more more specifically defined probably
ML Ops is really the
operational aspect of building and deploying and running
models, training models and everything around
that. And the gen AI Ops, or
AIOps if you will,
is DevOps. But
now it includes AI
as another very important component
which requires its own
capabilities like evaluation. So you test AI
differently than you test
a front end or then you do load tests
on an environment. You have to approach it a
slightly different way. It's the same thing in terms
of what you have to do. You have to make sure it works. Every time
you change something in your application, you have to make sure that it still works,
that you don't have any regression, that you didn't introduce any bugs.
Now there's a new class of regression, there's a new
class of bugs, there's a new class
of thing that may introduce latency or that may
introduce additional cost. And that class is based
on the integration of AI. And in
my opinion, the operational aspect of
this becomes native part of DevOps over
time. There are a lot of tools right now, there will
be more tools in the future, but it's very different
from what the data science and the ML engineer. Have you
seen any counterintuitive success story
where there was less tech and that actually led to better
performance of the AI in production? The most
counterintuitive thing is something that I see fairly often really
is when you approach it saying, well,
let's do this with AI and you realize actually we don't need AI for
this, or let's build an AI agent because
everybody's talking about agents right now, which is a good thing because
it's an evolving space. But you realize actually I don't really need an
agent because I can simply use an LLM
for this. So the most counterintuitive
is something that I have seen throughout my entire career. And software
engineering is less complex, very often
is more effective. So whenever you build something,
I encourage you to try and to experiment with AI and AI
agent, but I also
encourage you to not try to just solve
everything with AI. And that may be counterintuitive advice,
but it has always been sound advice in my experience. Experience.
Last question for us here in the second interview
and thank you for sticking around with me because we're together here in a session
for more than two hours now. Zoom out for us, Dennis.
What's the future of AI architecture? In something like two to
three years, do you see MCP and ancient frameworks
becoming the new standard? I have no
idea.
Literally, I have no idea. If you look back through
the last two to three years since the ChatGPT
moment, basically
everything has changed so dramatically.
The technology, the infrastructure, the capabilities of the
models themselves, the availability,
the open source framework, the work that the community is doing,
the many, many startups that are around
Certainly many still try to solve old problems with new tools. But there are
also so many niches where something incredible is actually
happening and there's so much innovation happening.
I don't even know I'm going to go on vacation a week from now for
three weeks. I don't even know what the world will look like when I'm back.
It's really hard. I think agents. Well, first
of all, AI is not like a flu. It won't go away.
It's going to stick around. Agentic
AI, it's being hyped right now. But I also think
it is a very important topic that either sticks around
or evolves into something even more
capable. The thing is,
the thing is the best, the best time to get in, to get
involved is now because it's never, it's never going to be as
simple as it is today. And I realize it is really hard.
I'm able, I'm. I'm lucky to be able to work with this stuff
every day, all day long. And I'm still overwhelmed. I'm still
overwhelmed. I've subscribed to so many newsletters and
there's so much news and so many tools to look at and so many frameworks
and so many. I don't know. I don't even know where to start until
I realized most of these newsletters and most of the
experts that are around all of a sudden, they just copy from each
other. Many of them, not all of them, but many really just copy from each
other. And I,
I'm fairly convinced that many of them
really are just AI tools creating content on the
socials, in newsletters and so forth. So it's really hard
to, it's really hard to
distill the actual signal from the noise right now.
But at the same time, it has never been as easy as today
because it's only getting more complicated. It's only going to be more
so for you. Really important is to get started now
and at the same time try to understand the fundamentals, not necessarily
the math behind these models. You don't need a PhD in science
or in math or anything. I, I certainly don't. I'm a developer. I don't
understand AI, to be, to be honest. But
what I do understand very well by now is
how can I use AI in software application?
What impact does it have on the capabilities of what I build, but also
what impact does it have on the way I work? That's two different
levels. And I'm, I'm able to do
that because I did the work to at
least understand the fundamentals of
what these models actually are and how
they work in terms of their capabilities. So
why do they get things wrong?
Why do they have what we call hallucinations? Why do they
have a hard time doing basic
math while being able to talk for
hours? So these are the things. And I
invite you to listen to
Joe's podcast. I invite you to have a look at the stuff that we put
out at aws, to the things that I put out on the socials.
Follow me on LinkedIn. It's just Dennis Troup. I think
Joe's going to put my context into the details. Ask
questions, talk to people. Figure out,
figure out how this stuff works.
Experiment, play around with it. Don't be stupid.
Don't connect a random piece of AI to your email.
Don't put something out on the Internet and then run up a bill because
somebody. DDoS is you. Experiment
in an isolated environment, maybe inside of an AWS account or
on your local machine where everything's isolated and protected.
You don't have to worry. Worry about external
influences and maybe threats. Experiment, play around with it
and at the same time think about, think
about the things that you might want to solve for
yourself. Think about, think about the things that you
need to do manually, manually every day
because it was too hard to automate or it was too impossible. It was
impossible to automate or it was too costly, or you just didn't get around
to automating it. Maybe AI can help you
with a small problem that you have every day,
that you're trying to solve every day, and it bothers you
and it's so annoying. That is what I did.
That's how I got started. That's how I learned. I looked at what I'm
doing every day and there's so much stuff that I never got around to doing.
And it was. I was complaining about them all the time and it bothered me
all the time. And all of a sudden I realized I can build a small
agent that just does it for me. And it doesn't even need
to connect to sensitive data or it doesn't even connect to
the Internet or anything. It is just a small
CLI tool that I run automatically
every day and it takes care of some stuff for me.
It pulls some statistics or
looks. If there was some new conversations on Slack that I need to know about,
these are the small things that I built. And by building these small things,
I. I learned how they work, I learned how they fail.
I learned about all the things that can go wrong. And then I
started being able to build larger things, to build more
complex applications, to build actual agents, actual
agentic systems that I now run for
more and more things. I have to admit though, I'm not running anything in
production because I'm not building production software anymore. I haven't
for a few years, unfortunately. But the great thing is, in
my role, I get to experiment with that stuff a lot.
Dennis, awesome last words. Thank you very much
for being such a good guest and telling us so much about
AI and aws, how they're working together.
Thank you so much for having me. It was a great time. Thank you.
That's all folks. Find more news, streams,
events and
interviews@www.startuprad.IO.
remember, sharing is caring.
Sam.