Hi, everyone.
Welcome to another lecture for CS230 Deep Learning.
Today, we're going to talk about enhancing large language model
applications.
And I call this lecture Beyond LLM.
It has a lot of newer content.
And the idea behind this lecture is
we started to learn about neurons,
and then we learned about layers,
and then we learned about deep neural networks,
and then we learned a little bit about how to structure projects
in C3.
And now we're going one level beyond into, what would it
look if you were building agentic AI systems at work,
in a startup, in a company?
And it's probably one of the more practical lectures.
Again, the goal is not to build a product
end to end in the next hour or so,
but rather to tell you all the techniques
that AI engineers have cracked, figured out, are exploring,
so that after the class, you have the breadth of view
of different prompting techniques,
different agentic workflows, multi-agent systems, evals.
And then when you want to dive deeper,
you have the baggage to dive deeper and learn faster
about it.
Let's try to make it as interactive as possible, as
usual.
When we look at the agenda, the agenda
is going to start with the core idea behind challenges
and opportunities for augmenting LLMs.
So we start from a base model.
How do we maximize the performance of that base model?
Then we'll dive deep into the first line of optimization,
which is prompting methods, and we'll see a variety of them.
Then we'll go slightly deeper.
If we were to get our hands under the hood
and do some fine tuning, what would it look like?
I'm not a fan of fine tuning, and I talk a lot about that,
but I'll explain why I try to avoid fine tuning as much as
possible.
And then we'll do a section 4 on Retrieval-Augmented Generation,
or RAG, which you've probably heard of in the news.
Maybe some of you have played with RAGs.
We're going to unpack what a RAG is
and how it works and then the different methods within RAGs.
And then we'll talk about agentic AI workflows.
I'll define it.
Andrew Ng is one of the first ones
to have called this trend agenetic AI workflows.
And so we look at the definition that Andrew
gives to agentic workflows, and then we'll
start seeing examples.
The section 6 is very practical.
It's a case study where we will think about an agentic workflow,
and I'll ask you to measure if the agent actually works,
and we brainstorm how we can measure
if an agentic workflow is working
the way you want it to work.
There's plenty of methods called evals that solve that problem.
And then we'll look briefly at multi-agent workflow.
And then we can have a open-ended discussion
where I share some thoughts on what's next in AI.
And I'm looking forward to hearing from you all,
as well, on that one.
So let's get started with the problem of augmenting LLMs.
So open-ended question for you--
you are all familiar with pre-trained models
like GPT 3.5 Turbo or GPT 4.0.
What's the limitation of using just a base model?
What are the typical issues that might
arise as you're using a vanilla pre-trained model?
Yes.
It lacks some domain knowledge.
Lacks some domain knowledge.
You're perfectly right.
We had a group of students a few years ago.
It was not LLM related, but they were building an autonomous
farming device or vehicle that had a camera underneath, taking
pictures of crops to determine if the crop is
sick or not, if it should be thrown away,
if it should be used or not.
And that data set is not a data set you find out there.
And the base model or pre-trained computer vision
model would lack that knowledge, of course.
What else?
Yes.
[INAUDIBLE] pictures are very dark [INAUDIBLE]
OK, maybe the-- you're saying--
so just to repeat for people online,
you're saying the model might have been trained
on high-quality data, but the data in the wild
is actually not that high quality.
And in fact, yes, the distribution of the real world
might differ, as we've seen with GANs, from the training set,
and that might create an issue with pre-trained models.
Although pre-trained LLMs are getting better
at handling all sorts of data inputs.
Yes.
Lacks current information.
Lack what?
Current information.
Lacks current information.
The LLM is not up to date.
And in fact, you're right.
Imagine you have to retrain from scratch your LLM
every couple of months.
One story that I found funny--
it's from probably three years ago or maybe more five years
ago, where during his first presidency,
President Trump one day tweeted, "Covfefe."
You remember that tweet or no?
Just "Covfefe."
And it was probably a typo or it was in his pocket.
I don't know.
But that word did not exist.
The LLMs, in fact, that Twitter was running at the time
could not recognize that word.
And so the recommender system sort of went wild,
because suddenly everybody was making fun of that tweet using
the word "Covfefe," and the LLM was so confused on, what does
that mean?
Where should we show it?
To whom should we show it?
And this is an example of a-- nowadays,
especially on social media, there's so many new trends,
and it's very hard to retrain an LLM to match the new trend
and understand the new words out there.
I mean, you oftentimes hear Gen Z words like "rizz" or "mid"
or whatever.
I don't know all of them.
But you probably want to find a way that
can allow the LLM to understand those trends without retraining
the LLM from scratch.
What else?
It's trained to have a breadth of knowledge.
And if you wanted to do something specialized,
that might limit [INAUDIBLE].
Yeah, it might be trained on a breadth of knowledge,
but it might fail or not perform adequately
on a narrow task that is very well defined.
Think about enterprise applications that--
yeah, enterprise application.
You need high precision, high fidelity, low latency.
And maybe the model is not great at that specific thing.
It might do fine, but just not good enough.
And you might want to augment it in a certain way.
Yeah.
Maybe it has [INAUDIBLE] so it makes the model
a lot heavier, a lot slower.
[INAUDIBLE]
So maybe it has a lot of broad domain knowledge that might not
be needed for your application.
And so you're using a massive, heavy model
when you actually are only using 2% of the model capability.
You're perfectly right.
You might not need all of it.
So you might find ways to prune, quantize the model, modify it.
All of these are good points.
I'm going to add a few more, as well.
LLMs are very difficult to control.
Your last point is actually an example of that.
You want to control the LLM to use a part of its knowledge,
but it's not--
it's, in fact, getting confused.
We've seen that in history.
In 2016, Microsoft created a notorious Twitter
bot that learned from users, and it quickly became a racist jerk.
Microsoft ended up removing the bot 16 hours after launching it.
The community was really fast at determining
that this was a racist bot.
And you can empathize with Microsoft in the sense
that it is actually hard to control an LLM.
They might have done a better job to qualify before launching,
but it is really hard to control an LLM.
Even more recently, this is a tweet
from Sam Altman last November, where
there was this debate between Elon Musk and Sam
Altman on whose LLM is the left wing propaganda
machine or the right wing propaganda machine,
and they were hating on each other's LLMs.
But that tells you, at the end of the day,
that even those two teams, Grok and OpenAI, which are probably
the best funded team with a lot of talent,
are not doing a great job at controlling their LLMs.
And from time to time, if you hang out on X,
you might see screenshots of users interacting with LLMs
and the LLM saying something really controversial
or racist or something that would not be considered great
by social standards, I guess.
And that tells you that the model is really hard to control.
The second aspect of it is something
that you mentioned earlier.
LLMs may underperform in your task,
and that might include specific knowledge gaps,
such as medical diagnosis.
If you're doing medical diagnosis,
you would rather have an LLM that is specialized for that
and is great at it and, in fact, something
that we haven't mentioned as a group, has sources.
So the answer is sourced specifically.
You have a hard time believing something
unless you have the actual source of the research that
backs it up.
Inconsistencies in style and format--
so imagine you're building a legal AI agentic workflow.
Legal has a very specific way to write and read,
where every word counts.
If you're negotiating a large contract,
every word on that contract might mean something else
when it comes to the court.
And so it's very important that you use
an LLM that is very good at it.
The precision matters.
And then task-specific understanding,
such as doing a classification on a niche field,
here I pulled an example where-- let's say a biotech product is
trying to use an LLM to categorize
user reviews into positive, neutral, or negative.
Maybe for that company, something
that would be considered a negative review typically
is actually considered a neutral review
because the NPS of that industry tends
to be way lower than other industries, let's say.
That's a task-specific understanding,
and the LLM needs to be aligned to what
the company believes is the categorization that it wants.
We will see an example of how to solve that problem in a second.
And then limited context handling--
a lot of AI applications, especially in the enterprise,
have required data that has a lot of context.
Just to give you a simple example,
knowledge management is an important space
that enterprises buy a lot of knowledge management tool.
When you go on your drive and you have all your documents,
ideally, you could have an LLM running on top of that drive.
You can ask any question, and it will read immediately
thousands of documents and answer, what was
our Q4 performance in sales?
It was x dollars.
It finds it super quickly.
In practice, because LLMs do not have a large enough context,
you cannot use a standalone vanilla pre-trained LLM to solve
that problem.
You will have to augment it.
Does that make sense?
The other aspect around context windows is they are, in fact,
limited.
If you look at the context windows of the models
from the last five years, even the best models
today will range in context, window, or number of tokens
it can take as input, somewhere in the hundreds of thousands
of tokens max.
Just to give you a sense, 200,000 tokens is roughly two
books.
So that's how much you can upload
and it can read, pretty much.
And you can imagine that when you're
dealing with video understanding or heavier data
files, that is, of course, an issue.
So you might have to chunk it.
You might have to embed it.
You might have to find other ways
to get the LLM to handle larger contexts.
The attention mechanism is also powerful, but problematic,
because it does not do a great job at attending
in very large contexts.
There is actually an interesting problem
called needle in a haystack.
It's an AI problem where--
or call it a benchmark--
where, in order to test if your LLM is good at putting attention
on a very specific fact within a large corpus,
researchers might randomly insert
in about one sentence that outlines
a certain fact, such as Arun and Max
are having coffee at Blue Bottle,
in the middle of the Bible, let's say,
or some very long text.
And then you ask the LLM, what were Arun and Max having
at Blue Bottle?
And you see if it remembers that it was coffee.
It's actually a complex problem, not because the question
is complex, but because you're asking the model
to find a fact within a very large corpus,
and that's complicated.
So, again, this is a limiting factor for LLMs.
We'll talk about RAG in a second.
But I want to preview--
there is debates around whether RAG
is the right long-term approach for AI systems.
So as a high-level idea, a RAG is a mechanism, if you will,
that embeds documents that an LLM can retrieve and then
add as context to its initial prompt and answer a question.
It has lots of application.
Knowledge management is an example.
So imagine you have your drive again.
But every document is compressed in representation,
and the LLM has access to that lower
dimensional representation.
The debates that this tweet from [INAUDIBLE] outlines
is, in theory, if we have infinite compute,
then RAG is useless.
Because you can just read a massive corpus immediately
and answer your question.
But even in that case, latency might be an issue.
Imagine the time it takes for an AI
to read all your drive every single time you ask a question.
It doesn't make sense.
So RAG has other advantages beyond even the accuracy.
On top of that, the sourcing matters, as well.
So it might-- RAG allows you to source.
We'll talk about all that later.
But there's always this debate in the community
whether a certain method is actually future proof.
Because in practice, as compute power doubles every year,
let's say, some of the methods we're learning right now
might not be relevant three years from now.
We don't know, essentially.
And the analogy that he makes on context windows
and why RAG approaches might be relevant even a long time
from now is search.
When you search on a search engine,
you still find sources of information.
And in fact, in the background, there
is very detailed traversal algorithms
that rank and find the specific links that might be the best
to present you versus if you had to read-- imagine you had
to read the entire web every single time you're doing
a search query, without being able to narrow
to a certain portion of the space.
That might, again, not be reasonable.
OK, when we're thinking of improving LLMs,
the easiest way we think of it is two dimensions.
One dimension is we are going to improve the foundation
model itself.
So, for example, we move from GPT 3.5 Turbo, to GPT 4,
to GPT 4.0, to GPT 5.
Each of that is supposed to improve the base model.
GPT 5 is another debate because it's packaging other models
within itself.
But if you're thinking about 3.5, 4, and 4.0,
that's really what it is.
The pre-trained model improves.
And so you should see your performance
improve on your tasks.
But the other dimension is we can actually engineer--
leverage the LLM in a way that makes it better.
So you can prompt simply GPT 4.0.
You can change some prompts and improve the prompt,
and it will improve the performance.
It's shown.
You can even put a RAG around it.
You can put an agentic workflow around it.
You can even put a multi-agent system around it.
And that is another dimension for you to improve performance.
So that's how I want you to think about it-- which
LLM I'm using, and then how can I maximize
the performance of that LLM?
This lecture is about the vertical axis.
Those are the methods that we will see together.
Sounds good for the introduction.
So let's move to prompt engineering.
I'm going to start with an interesting study just
to motivate why prompt engineering matters.
There is a study from HBS, UPenn,
as well as Harvard Business School, and--
well, there is also involved Wharton--
that took a subset of BCG consultants,
individual contributors, split them into three groups.
One group had no access to AI.
One group had access to--
I think it was GPT 4.
And then one group had access to the LLM,
but also a training on how to prompt better.
And then they observed the performance of these consultants
across a wide variety of tasks.
There's a few things that they noticed
that I thought was interesting.
One is something they called the jagged frontier,
meaning that certain tasks that consultants are doing fall
beyond the jagged frontier, meaning AI is not good enough.
It's not improving human performance.
In fact, it's actually making it worse.
And some tasks are within the frontier,
meaning that AI is actually significantly improving
the performance, the speed, the quality of the consultant.
Many tasks fell within and many tasks fell without,
and they shared their insights.
But the TLDR is--
there is a frontier within which AI is absolutely helping
and one where they call out this behavior, or falling asleep
at the wheel, where people relied on AI on a task that
was beyond the frontier.
And in fact, it ended up going worse
because the human was not reviewing the outputs carefully
enough.
They did note that the group that was trained
was the best, better than the group that was not trained
on prompt engineering, which also motivates why
this lecture matters, so that you're within that group
afterwards.
One other insights were the centaurs and the cyborgs.
They noticed that consultants had the tendency
to work with AI in one of two ways,
and you might, yourself, be part of one of these groups.
The centaurs are mythical creatures
that are half human, half--
I think, half, what, horses?
Yeah?
Horses?
Half horses, half something.
And those were individuals that would divide and delegate.
They might give a pretty big task to the AI.
So imagine you're working on a PowerPoint, which consultants
are known to do.
You might actually write a very long prompt on how
you want it to do your PowerPoint and then let it
work for some time and then come back
and it's done, when others would act as cyborgs.
Cyborgs are fully blended, bionic human robots,
human and robot, augmented with robotic parts.
And those individuals will not delegate fully a task.
They would actually work super quickly with the model
back and forth.
I find that a lot of students are actually more working
like cyborgs than centaurs, but while maybe in the enterprise,
when you're trying to automate the workflow,
you're thinking more like a centaur.
That's just something good to keep in mind.
Also, a lot of companies will tell you, oh, we're
hiring prompt engineers, et cetera.
It's [? a cure. ?] I don't buy that.
I think it's just a skill that everybody should have.
You're not going to make a [? cure ?] out
of prompt engineering, but you're probably
going to use it as a very powerful skill in your career.
So let's talk about basic prompt design principles.
I'm giving you a very simple prompt here.
Summarize this document, and then the document
is uploaded alongside it.
And the model has not much context around
what should be the summary?
How long should be the summary?
What should it talk about, et cetera?
You can actually improve these prompts by doing something like
summarize this 10-page scientific paper on renewable
energy in five bullet points, focusing on key findings
and implications for policymakers.
That's already better.
You're sharing the audience, and it's
going to tailor it to the audience.
You're saying that you want five bullet points,
and you want to focus only on key findings.
That's a better prompt, you would argue.
How could you even make this prompt better?
What are other techniques that you've
heard of or tried yourself that could make this one shot prompt
better?
Yeah.
[INAUDIBLE]
OK.
Right example.
So say, you mean, here is an example of a great summary.
Yeah.
You're right.
That's a good idea.
[INAUDIBLE]
Very popular technique.
Act like a renewable energy expert giving a conference
at Davos, let's say, yeah.
That's great.
Someone-- yeah.
Say you're really good at it.
Yeah.
You are the best in the world at this.
Explain.
Yeah.
Actually, I mean, these things work.
It's funny, but it does work to say act like x, y, z.
It's a very popular prompt template.
We'll see a few examples.
What else could you do?
Yes.
Of course, you'd like to critique your own model.
Critique your own project.
So you're using reflection.
So you might actually do one output
and then ask it to critique it and then give it back.
Yeah.
We see that.
That's a great one.
That's the one that probably works best
within those typically, but we see some examples.
What else?
Yeah.
Break the task down into steps.
OK.
Break the task down into steps.
You know how that is called?
No.
OK.
Chain of thoughts.
So this is actually a popular method
that's been shown in research that it improves.
You could actually give a clear instruction
and also encourage the model to think step
by step approach, the task step by step,
and do not skip any step.
And then you give it some steps, such as step one,
identify the three most important findings.
Step two, explain how key each finding
impact renewable energy policy.
Step three, write the five-bullet summary
with each point addressing a finding, et cetera.
So chain of thoughts, I linked the paper from 2023 that
popularized chain of thoughts.
Chain of thoughts is very popular
right now, especially in AI startups
that are trying to control their LLMs.
OK.
To go back to your examples about act like XYZ, what
I like to do, Andrew Ng also talks about that,
is to look at other people's prompts.
And in fact, in online, you have a lot of prompt repositories
for free on GitHub.
In fact, I linked the awesome prompt template repo on GitHub,
where you have so many examples of great prompts
that engineers have built. They said it works great for us,
and they published it online.
And a lot of them start with act as.
Act as a Linux terminal.
Act as an English translator.
Act like a position interviewer, et cetera.
The advantage of a prompt template
is that you can actually put it in your code
and scale it for many user requests.
So let me give you an example from Workera.
Workera evaluates skill.
Some of you have taken the assessments already.
And tries to personalize it to the user.
And in fact, if you actually read in an HR system
in an enterprise, in the HR system,
you might have a Jane is a product manager level 3,
and she is in the US, and her preferred language is English.
And actually, that metadata can be
inserted in a prompt templates that will personalize
personalized for Jane.
And similarly for Joe, whose is preferred language is Spanish,
it will tailor it to Joe.
And that's called a prompt template.
[INAUDIBLE]
So the question is do the foundation models
use a prompt templates, or do you
have to integrate it yourself?
So the foundation models probably
use a system prompt that you don't see.
Like when actually, you type on ChatGPT,
it is possible, it's not public, that OpenAI behind the scenes
has like act like a very helpful assistant for this user.
And by the way, here is your memories about the user
that we kept in a database.
You can actually check your memories.
And then your prompt goes under, and then the generation starts.
So probably, they're using something like that.
But it doesn't mean you can't add one yourself.
So in fact, if you think about a prompt template for the Workera
example I was showing, maybe it starts
when you call OpenAI by act like a helpful assistant.
And then underneath, it's like act like a great AI mentor that
helps people in their career.
And OpenAI is, from template, also
has follow the instruction from the creator
or something like that.
It's possible.
Questions about prompt templates.
Again, I would encourage you to go and read examples of prompts.
Some of them are quite thoughtful.
Let's talk about zero shot versus few shot prompting.
It came up earlier.
Here's an example.
Again, going back to the categorization of product
reviews, let's say that we're working on a task
where the prompt is classify the tone of the sentence
as positive, negative, or neutral.
And then you paste the review, which is the product is fine,
but I was expecting more.
If I were to survey the room, I would bet that some of you
would say it's negative.
Some of you would say it's neutral.
Because you actually have a first part
that is relatively positive.
It's fine.
And then the second part, I was expecting more,
which is relatively negative.
So where do you land?
This can be a subjective question.
And maybe in one industry, this would be considered amazing.
And another one, it would be considered really bad
because people are used to really flourishing reviews.
And so the way you can actually align the model to your task
is by converting that zero shot prompt.
Zero shot refers to the fact that it's not
being given any example.
Into a few short prompts, where the model
is given in the prompt, a set of examples to align it to what
you want it to do.
So the example here is again, you
paste the same prompt as before with the user review.
And then you add, here are examples
of tone classifications.
These exceeded my expectation completely.
Positive.
It's OK, but I wish it had more features.
Negative.
The service was adequate.
Neither good nor bad.
Neutral.
Now classify the tone of this sentence
after you've heard about these things,
and the model then says negative.
And the reason it says negative, of course,
is likely because of the second example, which was it's OK,
but I wish it had more features, which we told the model that
was negative.
Because the model saw that it's aligned now
with your expectations.
A few short prompts are very popular.
And in fact, for AI startups that
are slightly more sophisticated, you
might see them keep a prompt up to date.
Whenever a user says something and they
might have a human label it and then
add it as a few shots in their relevant
prompts in their code base.
You can think of that as almost building a data set.
But instead of actually building a separate data set
like we've seen with supervised fine tuning
and then fine tuning the model on it,
you're just putting it directly in the prompt.
It turns out it's probably faster
to do that if you want to experiment quickly
because you don't touch the model parameters.
You just update your prompts.
And if it's text examples, you can actually
concatenate so many examples in a single prompt.
At some point, it will be too long,
and you will not have the necessary context window.
But it's a pretty strong approach
that is quick to align an LLM.
OK?
Yes.
[INAUDIBLE]
So the question was is there any research on how long
the prompt can be before the model essentially loses
itself or doesn't follow instructions anymore?
There is.
The problem is that research is outdated every few months
because models get better.
And so I don't know where the state of the art is.
You can probably find it online on benchmarks
on like we see that--
I give you an example.
On the Workera product, you have a voice conversation
for some of you that have tried it,
where you're asked to explain what is the prompt.
And then you explain, and then there's
a scoring algorithm in behind.
We know that after eight turns, the model loses itself.
After eight turns, because you always
paste the previous user response,
it just starts going wild.
And so the techniques we use in the background
is we actually create chapters of the conversation.
Maybe one chapter is the first eight prompt.
And then you actually start over from another prompt.
You can summarize the first part of the conversation,
insert the summary, and then keep going.
Those are engineering hacks that engineers might have figured out
in the background.
Because eight turns makes a prompt quite long actually.
Let's move on to chaining.
Chaining is the most popular technique out of everything
we've seen so far in prompt engineering.
It's not chain of thought.
So chain of thought we've seen is think step by step,
step 1, step 2, step 3.
Do not skip any step.
This is different.
This is chaining complex prompt to improve performance,
and this is what it looks like.
You take a single step prompt, such as read this customer
review and write a professional response that
acknowledges their concern, explains the issue,
offers a resolution, and then you
paste the customer review, which is I ordered a laptop.
It arrived three days late.
The packaging was damaged.
Very disappointing.
I needed that urgently for work.
And then the output is an email that
is immediately given to you by the LLM
after it reads the prompt.
So this might work, but it might be hard to control.
Because think about it.
There's multiple steps that you have listed,
and everything is embedded in the same prompt.
And if you wanted to debug step by step and know which step is
weaker, you couldn't.
You would have everything mixed together.
So one advantage of chaining is you would separate the prompts,
so that you can debug them separately.
And it will also lead to an easier manner
to improve your workflow.
Let's say a first prompt is extract the key issues.
Identify the key concerns mentioned
in this customer review.
Pace the customer review.
Second prompt.
Using these issues, so you paste back the issues,
draft an outline for a professional response that
acknowledges concerns, explains possible reasons,
and offer a resolution.
So this is not--
Prompt number 3, write the full response.
So using the outline, write the professional response.
And then you get your final output.
So in theory, you can tell me, oh, the second approach
is better than the first one at first.
But what you can notice is that we can actually
test those three prompts separately from each other
and determine if we will get the most gains out of engineering
the first prompt, optimizing it, or the second one,
or the third one.
We now have three prompts that are independent from each other.
And maybe if the outline was better,
the performance of the email, how much the open rate will be
or the user satisfaction on the response
will actually get higher.
And so chaining improves performance but performance,
but most importantly, helps you control your workflow
and debug it more seamlessly.
Yes.
So if we that the three prompts independently work really well,
if we combine them into one prompt,
and we highlight a step by step thinking process,
does on average, we get a [INAUDIBLE] by itself,
or do we still have to do that breakdown?
So let me try to rephrase.
You say, let's say we look at the first prompt which
has all three tasks built in that prompt.
What exactly do you mean?
You mean like if we evaluate the output
and we measure some user insight, satisfaction,
et cetera?
Why don't we just modify that prompt and essentially see how
it improves user satisfaction?
Yeah.
[INAUDIBLE]
I see.
So why do we need the three steps?
I mean, think about it.
The intermediate output is what you want to see.
Like if I'm debugging the first approach,
the way I would do it is I would capture user insights.
Like here's the email.
How good was the response?
Thumbs up, thumbs down.
Was your issue resolved?
Thumbs up, thumbs down.
Those would tell me how good is my prompt.
And I can engineer that prompt, optimize it,
and I would probably drive some gains.
But I will not be able easily to trace back
to what the problem was.
While in the second approach, not only I
can use the end to end metrics to improve my process.
I can also use the intermediate steps.
For example, if I look at prompt 2 and I look at the outline
and I see the outline is actually, meh, it's not great,
then I think I can get a lot of gains out of the outline.
Or the outline is actually really good,
but the last prompt doesn't do a good job at translating it
into an email.
So the outline is exactly what I want the LLM to do,
but the translation in a customer facing email
is not good.
In fact, it doesn't follow our vocabulary internally.
Then I knew the third prompt is where
I would get the most gains.
So that's what it allows me to do,
have intermediate steps to review.
Are there any latency [INAUDIBLE]
We'll talk about it.
Are there any latency concerns?
Yes.
In certain applications, you don't want to use a chain
or you don't want to use a long chain because it adds latency.
We'll talk about that later.
Good point.
So practically, this is what chaining complex
prompts look like.
You have your first prompt with your first task.
It outputs.
The output is pasted in the second prompt
with the second task being defined.
The output is then pasted into the third prompt
with the third task being defined and so on.
That's what it looks like in practice.
Super.
We'll talk more later about testing your prompts,
but there are methods now to do it,
and we'll see later in this lecture with our case study
how we can test our prompts.
But here is an example of how you might do it.
You might have a summarization workflow prompts
that is the baseline.
It's a single prompt.
You might have a refined summarization
which is a modified prompt of this,
or a workflow with a chain.
And then you have your test case, which is the input
that you want to summarize, let's say.
And then you have the generated output.
And you can have humans go and rate these outputs.
And you would notice that the baseline is better or worse
than the refined prompt.
Of course, this manual approach takes time,
but it's a good way to start.
And usually, the advice is get hands on at the beginning
because you would quickly notice some issues,
and it will give you better intuition on what tweaks
can lead to better performance.
However, if you wanted to scale that system
across many products, many parts of your code base,
you might want to find a way to do that automatically
without asking humans to review and grade summaries.
One approach is to use platforms,
like at Workera, our team uses a platform called Prompt Food that
allows you to actually automate part of this testing.
In a nutshell, what it does is it
can allow you to run the same prompt with five different LLMs
immediately, put everything in a table.
That makes it super easy for a human to grade, let's say.
Or alternatively, it might allow you to define LLM judges.
LLM judges can come in different flavors.
For example, I can have an LLM judge that
does a pairwise comparison.
So what the LLM is asked to do is here are two summaries.
Just tell me which one is better than the other one.
That's what the LLM does.
And that can be used as a proxy for how good
the summarization baseline versus the refined version is.
Another way to do an LLM judge is
if you do it for a single answer grading,
so here's a summary graded from 1 to 5.
And then you can go even deeper and do
a reference-guided pairwise comparison.
Or you add also a rubric.
You say a 5 is when a summary is below 100 characters.
I'm just making up.
Below 100 characters.
Mentions at least three key points
that are distinct and starts with a first sentence that
displays the overview and then goes into the detail.
That's a great summary, number 5 out of a 5.
0 is the LLM failed to summarize and actually was very verbose,
let's say.
And so you put a Rubrik behind it,
and you have an LLM as just finding the rubric.
Of course, you can now pair different techniques.
You can do a few shots for the rubric.
You can actually give examples of a 5 out of 5s, 4 out of 4s,
3 out of 3s because now, you multiple techniques.
Does that make sense?
Yeah.
OK.
So that was the second section on prompt engineering
or the first line of optimization.
Now, let's say you've exhausted all your chances
for prompt engineering, and you're
thinking about actually touching the model, modifying its weights
or fine tuning it in other words.
I was telling you, I'm not a fan of fine tuning.
There's a few reasons why.
One, it requires substantial labeled data typically
to fine tune.
Although now, there are approaches
that are getting better at fine tuning that
look more few shot prompting actually than fine tuning.
It's sort of merging.
Although one modifies the weight,
the other doesn't modify the weights.
Fine tuned models may also overfit to specific data.
We're going to see a funny example actually.
Losing their general purpose utility.
So you might fine tune a model.
And actually, when someone asks a pretty generic question,
it doesn't do well anymore.
It might do well on your task.
So it might be relevant or not.
And then it's time and cost-intensive.
That's my main problem.
And at Workera, we steer away from fine
tuning as much as possible.
Because by the time you're done fine tuning your model,
the next model is out, and it's actually
beating your fine tuned version of the previous model.
So I would steer away from fine tuning as much as you can.
The advantage of the prompt engineering methods we've seen
is you can put the next best pre-trained model directly
in your code.
It will update everything immediately.
Fine tuning doesn't work like that.
There are advantages though where it still makes sense.
If the task requires repeated high precision outputs
such as legal, scientific explanation
and if the general purpose LLM struggles
with domain-specific language.
So let's look at a quick example together,
which is an example from Ros Lazerowitz.
I think it was a couple of years ago, September 23,
where Ros tried to do Slack fine tuning.
So he looked at a lot of Slack messages within his company.
And he was like, I'm going to fine tune
a model that speaks like us or operates like us because this
is how we work.
This is the data that represents how people work at the company.
And so he actually went ahead and fine tuned the model,
gave it a prompt, like, hey, write--
he was delegating to the model.
A 500-word blog post on prompt engineering.
And the model responded, I shall work on that in the morning.
And then he tries to push the model a little further and say,
it's morning now.
And the model said, I'm writing right now.
It's 6:30 AM here.
Write it now.
OK, I shall write it now.
I actually don't what you would like me to say
about prompt engineering.
I can only describe the process.
The only thing that comes to mind for a headline
is how do we build prompt?
It's kind of a funny example for fine tuning because it's true
that it went wrong.
Like he was supposed to think like I want
the model to speak like us at work.
And it ended up acting like people
and not actually following instructions.
So one example why I would steer away from fine tuning.
Super.
Let's talk about RAGs.
RAGs is important.
It's important to out there and at least having the basics.
It's a very common interview question, by the way.
If you go interview for a job, they
might ask you to explain in a nutshell
to a five-year-old what is a RAG.
And hopefully after that, you'll be able to do it.
So we've seen some of the challenges with standalone LLMs.
Those challenges include the context window being small,
the fact that it's hard to remember details
within a large context window, knowledge gaps, cutoff dates,
you mentioned earlier.
The model might be trained up to a date,
and then it cannot follow the trends or be up to date.
Hallucinations.
There are some fields.
Think about medical diagnosis, where
hallucinations are very costly.
You can't afford a hallucination.
Even in education, imagine deploying a model for the US
youth education, and it hallucinates,
and it teaches millions of people something
completely wrong.
It's a problem.
And then lack of sources.
A lot of fields love sources.
Research fields love sources.
Education love sources.
Legal loves sources as well.
And so the pre-trained LLM doesn't do a good job to source.
And in fact, if you have tried to find sources on a plain LLM,
it actually hallucinates a lot.
It makes up research papers.
It just lists like completely fake stuff.
So how do we solve that with a RAG?
RAG integrates with external knowledge sources, databases,
documents, APIs.
It ensures that answers are more accurate, up to date,
and grounded because you can actually update your document.
Your drive is always up to date.
I mean, ideally, you're always pushing new documents to it.
And when you query, what is our Q4 performance in sales?
Hopefully there is the last board deck in the drive,
and it can read the last board deck.
And more developer control.
We'll see why RAGs allow for targeted customization
without actually requiring the retraining of the model.
In fact, you don't touch the model with RAGs.
It's really a technique that is put on top of the model.
So to see an example of a RAG, this
is a question answering application where
we're in the medical field, and a user is asking a query,
what are the side effects of drug X?
This is an important question.
You can't hallucinate.
You need to source.
You need to be up to date.
Maybe there is a new update to that drug that
is now in the database, and you need to read that.
So a RAG is a great example of what you would want to use here.
The way it works is you have your knowledge
base of a bunch of documents.
What you do is you use an embedding
to embed those documents into lower
dimensional representations.
So for example, if the document is a PDF, a long PDF,
you might read the PDF, understand it,
and then embed it.
We've seen plenty of embedding approaches
together, triplet loss, et cetera, you remember?
So imagine one of them here for LLMs
is embedding those documents into lower representation.
If the representation is too small,
you will lose information.
If it's too big, you will add latency.
It's a tradeoff.
You will store typically those representations
into a database called a vector database.
There's a lot of vector database providers out there.
I think I've listed a couple that are very common.
No, I haven't listed, but I can share afterwards.
A vector database is essentially storing those vector
in a very efficient manner, allowing the fast retrieval
with a certain distance metric.
So what you do is you also embed, usually
with the same algorithm, the user prompts.
And you run a retrieval process, which is essentially
saying, based on the embedding from the user
query and the vector database, find the relevant documents
based on the distance between those embeddings.
Once you've found the relevant documents, you pull them,
and then you add them to the user query with a system prompt
or a prompt template on top.
So the prompt template can be answer user query
based on list of documents.
If answer not in the documents, say I don't know.
That's your prompt templates where the user query is pasted,
the documents are pasted, and then
your output should be what you want because it's not
grounded in the documents.
You can also add to this prompt template.
Tell me the exact page, chapter, line
of the document that was relevant, and in fact,
link it as well, just to be more precise.
Any question on RAGs?
This is a simple, vanilla RAG.
Yes.
Do document embeddings still retain information [INAUDIBLE]
Question is do the document embeddings still
retain the information of the location of the information
within that document, especially in big documents?
Great question.
We'll get to it in a second.
Because you're right that the vanilla RAG
might not do a good job with very large documents.
So let's say, when you open a medication box
and you have this gigantic white paper with all the information,
and it's very long, maybe a vanilla RAG would not cut it.
So what people have figured out is a bunch
of techniques to improve RAGs.
And in fact, chunking is a great technique that is very popular.
So you might actually store in the vector database
the embedding of the full document.
And on top of that, you will also
store a chapter level vector.
And when you retrieve, you will retrieve the document.
You retrieve the chapter.
And that allows you to be more precise with the sourcing.
It's one example.
Another technique that's popular is HyDE.
Hypothetical document embeddings,
where a group of researchers published a paper
showing that when you get your user query,
one of the main problem is the user query
actually does not look like your documents.
For example, the user query might
be what are the side effects of drug X, when actually,
in the document in the vector database,
the vectors represents very long documents.
So how do you guarantee that the vector
embedding is going to be close to the document embedding?
What they do is they use the user query to generate
a fake hallucinated document.
They embed that document, and then they
compare it to the vector in the vector database.
That makes sense?
So for example, the user says what
is the side effect of drug X?
There's a prompt that this is given to another prompt that
says, based on this user query, generates a five-page report
answering the user query.
It generates potentially a completely fake answer.
You embed that, and it will be closer to the document
that you're looking for likely.
It's one example of a RAG approach.
Again, the purpose of this lecture
is not to go through all these three and explain
you every single method that has been discovered for RAGs.
But I just wanted to show you how much research
has been done between 2020 and 2025 in RAGs
and how many branches of research you now have
that you can learn from.
The survey paper is LinkedIn the slides, by the way,
and I'll share them after the lecture.
Super.
So we've made some progress.
Hopefully now, you feel if you were
to start an LLM application, you know how to do better prompts.
You know how to do chains.
You know how to do fine tuning.
You also how to do retrieval.
And you have the baggage of techniques
that you can go and read and find the code base,
pull the code, vibe code it.
But you have the breadth now.
The next set of topics we're going to see
is around the question of how could we
extend the capabilities of LLMs from performing single tasks,
and hence, with external knowledge,
to handling multi-step, autonomous workflows?
And this is where we get into proper agentic AI.
So let's talk about agentic AI workflows
towards autonomous and specialized systems.
Then we'll talk about evals.
Then we'll see multi-agent systems.
And we'll end with a little thoughts on what's next in AI.
So Andrew Ng actually coined the term agentic AI workflows.
And his reason was that a lot of companies use, say agents.
Agents, agents everywhere, agents everywhere.
If you go and work at these companies,
you would notice that they mean very different things by agents.
Some people actually have a prompt,
and they call it an agent.
Other people, they have a very complex multi-agent system,
they call it an agent.
And so calling everything an agent doesn't do it justice.
So Andrew says let's call it agentic workflows.
Because in practice, it's a bunch of prompts with tools,
with additional resources, API calls
that ultimately are put in a workflow,
and you can call that workflow agentic.
So it's all about the multi-step process to complete a task.
Also, calling it agentic workflow
allows us to not mix it up with what
I called agent, in the last lecture,
with reinforcement learning.
Because in RL, agent has a very specific definition,
interacts with an environment, passes from one state
to the other, has a reward and an observation.
You remember that chart, right?
So here's an example of how we move from a one step
prompt to a multi-step agentic workflow.
Let's say a user queries a product.
What is your refund policy on a chatbot?
And the response, using a RAG, says
refunds are available within 30 days of purchase,
and maybe the RAG can even look link to the policy documents.
That's what we learned so far.
Instead, an agentic workflow can function like this.
The user says, can I get a refund for my order?
And the response via the agentic workflow
is the agent retrieves the refund policy using a RAG.
The agent then follows up with the users and says,
can you provide your order number?
Then the agent queries an API to check the order details.
And finally, it comes back to the user
and confirms your order qualifies for a refund.
The amount will be processed in three to five business days.
This is much more thoughtful than the first version,
which is sort of vanilla.
So that's what we're going to talk
about in the next couple of slides,
is how do we get from the first one to the second one?
There are plenty of specialized agency workflows online.
You've heard, and if you hang out in SF,
you probably see a bunch of billboards, AI software
engineer, AI skills mentor you've
interacted with in the class through Workera.
AI SDR, AI lawyers, AI specialized cloud engineer.
It would be a stretch to say that everything works,
but there's work being done towards that.
I'm not personally a fan of putting
a face behind those things.
I think it's gimmicky.
And I think in a few years from now, actually,
very few products will have a human face behind it,
but it might be a marketing tactic from some startups.
It's more scary than it is engaging, frankly.
OK.
I want to talk about the paradigm shift.
That's especially useful.
Let's say you're a software engineer
or you're planning to be a software engineer.
Because software engineering as a discipline
is sort of shifting.
Or at least the best engineers I've
worked with are able to move from a deterministic mindset
to a fuzzy mindset and balance between the two
whenever they need to get something done.
So here's the paradigm shift between traditional software
and agentic AI software.
The first one is the way you handle data.
Traditional software deals with structured data.
You have JSONs.
You have databases.
They're pasted in a very structured manner
in a data engineering pipeline.
And then there used to be displayed
on a certain interface.
The user might feel a form that is then retrieved and pasted
in the database.
All of that historically has been structured data.
Now, more and more companies are handling free form text, images,
and all of that requires dynamic interpretation to transform
an input into an output.
The software itself used to be deterministic.
Now you have a lot of software that is fuzzy.
And fuzzy software creates so many issues.
I mean, imagine if you let your user ask anything
on your website.
The chances that it breaks is tremendous.
The chances that you're attacked is tremendous.
The chances-- it's really, really complicated.
It's more complicated than people make it seem on Twitter.
Fuzzy engineering is truly hard.
You might get hate as a company because one user did something
that you authorized them to do that ended up breaking
the database and ended up--
we've seen that with many companies
in the last couple of years.
So it takes a very specialized engineering mindset
to do fuzzy engineering, but also
know when you need to be deterministic.
The other thing I'd call is with agentic AI software,
you want to think about your software as your manager.
So you're familiar with the monolith or microservices
approaches in software, where you structure your software
in different boxes that can talk to each other,
and it allows teams to debug one section at a time.
Now the equivalent with agentic AI is you think as a manager.
So you think, OK, if I was to delegate my product
to be done by a group of humans, what would be those roles?
Would I have a graphic designer that then puts together a chart
and then sends it to a marketing manager that converts it
into a nice blog post, that then gives it to the performance
marketing expert, that then publishes the work, the blog
post, and then optimizes and A/B tests?
Then to a data scientist that analyzes the data
and then puts hypotheses and validates
them or invalidates them.
That's how you would typically think if you're building
an authentic AI software.
When actually, the equivalent of that in traditional software
might be completely different.
It might be We have a data engineer box
right here that handles all our data engineering.
And then here, we have the UI/UX stuff.
Everything UI/UX related goes here.
And companies might structure it in very different ways.
And here is the business logic that we want to care about.
And there's five engineers working on the business logic,
let's say.
OK.
Testing and debugging is also very different.
And we'll talk about it in the next section.
The other thing that I feel matters
is with AI in engineering, the cost of experimentation
is going down drastically.
And so people, I feel, should be more comfortable
throwing away code.
It's like in traditional software engineering,
you probably don't throw away code a ton.
You build a code, and it's solid, and it's bulletproof,
and then you update it over time.
We've seen AI companies be more comfortable throwing away
codes, which has advantages in terms of the speed at which you
move but also disadvantages in terms
of the quality of your software that can break more.
So anyway, just wanted to do an update on the paradigm shift
from deterministic to fuzzy engineering.
Oh, and actually, I can give you an example from Workera
that we learned probably over the last 12
months is like if you've used Workera,
you might have seen that the interface has asks you sometimes
multiple choice questions.
And sometimes, it asks you multiple select.
And sometimes, it asks you drag and drop, ordering, matching,
whatever.
Those are examples of deterministic item types,
meaning you answer the question on a multiple choice.
There is one correct answer.
It's fully deterministic.
On the other hand, you sometimes have a voice questions,
where you go to a role play or you
have voice plus coding questions,
where your code is being read by the interface or whatever.
Those are fuzzy, meaning the scoring algorithm
might actually make mistakes, and those mistakes
might be costly.
And so companies have to figure out
a human in the loop system, which
you might have seen with the appeal feature at the end.
So at the end of the assessment, you have an appeal feature where
it allows you to say, I want to appeal the agent
because I want to challenge what the agent said on my answer
because I thought I was better than what the agent thought.
And then you bring the human in the loop that
then can fix the agent, can tell the agent, actually,
you were too harsh on the answer of this person.
And that's an example of a fuzzy engineered system
that then adds a human in the loop to make it more aligned.
And so if you're building a company,
I would encourage you to think about what can I
get done with determinism?
And let's get that done.
And then the fuzzy stuff, I want to do fuzzy
because it allows more interaction.
It allows more back and forth, but I need
to put guardrails around it.
And how am I going to design those guardrails?
Pretty much.
OK?
Here's another example from enterprise workflows,
which are likely to change due to agentic AI.
This is a paper from McKinsey, I believe from last year,
where they looked at a financial institution, and they said,
we observed that they often spend one to four weeks
to create a credit risk memo.
And here's the process.
A relationship manager gathers data from 15
and more than 15 sources on the borrower,
loan type, other factors.
Then the relationship manager and the credit analyst
collaboratively analyze that data from these sources.
Then the credit analyst typically spends 20 hours
or more writing a memo and then goes back
to the relationship manager.
They give feedback, and then they go through this loop
again and again.
And it takes a long time to get a credit memo out.
And then run a research study, where they changed the process.
They said gen AI agents could actually cut time by 20% to 60%
on credit risk memos.
And the process has changed to the relationship manager,
directly work with the Gen AI agent system,
provides relevant materials that needs to produce the memo.
The agent subsidizes the project into tasks
that are assigned to specialist agents,
gathers and analyzes the data from multiple sources,
drafts a memo.
Then the relationship manager and the credit analyst
sit down together, review the memo,
give feedback to the agent.
And within 20% to 60% less time are done.
And so this is an example where you're actually not changing
the human stakeholders.
You're just changing the process and adding
Gen AI to reduce the time it takes to get a credit memo out.
It turns out that, imagine you're an enterprise,
and you have 100,000 employees, and there's a lot of enterprises
with 100,000 employees out there.
You are currently under crisis in terms
of redesigning your workflows.
It turns out that if you actually
pull the job descriptions from the HR system
and you interpret them, you also pull
the business process workflows that you
have encoded in your drive.
You actually can find gains in multiple places.
And in the next few years, you're
probably going to see workflows being
more optimized to add Gen AI.
Even if that happens, the hardest part is changing people.
What we know, this is great in theory, but now,
let's try to fit that second workflow for 10,000 credits,
risk analysts, and relationship managers.
My guess is it will take years.
It will take 10, 20 years to get to this being actually done
at scale within an organization.
Because change is so hard.
It's so hard to rewire business, workflows, job descriptions,
incentivize people to do different, and be different,
and train them.
And so this is what the world is going towards,
but it's going to take a long time I think.
OK.
Then I want to talk about how the agent actually works
and what are the core components of an agent.
Imagine a travel booking agent. that's
an easy example you've all thought about.
I still haven't been able to get an agent to book a trip for me,
or I was scared because it was going to book
a very expensive or long trip.
But in theory, you can have a travel booking
agent that has prompts.
So the prompts we've seen, we know the methods
to optimize those prompts.
That travel agent also has a context management system,
which is essentially the memory of what it knows about the user.
That context management system might
include a core memory or working memory and an archival memory,
OK?
What the difference is within memory
is not every memory needs to be fast to access.
Think about it.
You're onboarded on a product, and the first question is hi,
what's your name?
And I say, my name is Keon.
That's probably going to sit in the working memory
because the agents, every time he's going to talk to me,
he's going to want to use my name.
But then maybe the second question
is what's your birthday?
And I give it my birthday.
Does it need my birthday every day?
Probably not.
So it's probably going to park it on the long term
memory or the archival memory.
And those memories are slower to access.
They're farther down the stack.
And that structure allows the agent
to determine what's the working memory,
and what's the long term memory?
And that makes it easier for the agent to retrieve super fast.
Because think about it.
When you interact with ChatGPT, you
feel that it's very personal at times.
You feel like it understands you.
Imagine every time you call it, it has to read the memories.
And that can be costly.
It's a very burdensome cost because it happens
every time you talk to it.
So you want to be highly optimized with the working
memory.
If it takes three seconds to look
in the memory, every time you're going to talk to your LLM,
it's going to take three seconds, which you don't want.
Anyway.
And then you have the tools.
The tools can include APIs like a flight search
API, hotel booking API, car rental API, weather API,
and then the payment processing API.
And typically, you would want to tell your agent
how that API works.
It turns out that agents or LLMs, I should say,
are very good at reading API documentation.
So you give it the API documentation,
and it reads the JSON, and it reads,
what does a GET request look like.
And this is the format that I need to push.
And then it pushes it in that format, let's say.
And then it retrieves something.
Does that make sense, those different components?
Anthropic also talks about resources.
Resources is data that is sitting somewhere that you
might let your agent read.
For example, if you're building your startups, you have a CRM.
A CRM has data in it, and you want to do lookups in that data.
You will probably give a lookup tool,
and you will give access to the resource,
and it will do lookups whenever you want super fast.
This type of architecture can be built
with different degrees of autonomy,
from the least autonomous to the most autonomous.
And I'll give you a few examples.
Less autonomous would be you've hard coded the steps.
So let's say I tell the travel agent first identify the intent.
Then look up in the database the history
of this customer with us and their preferences.
Then go to the flight API, blah, blah, blah.
Then go to the--
I would hard code the steps.
OK.
That's the least autonomous.
The semi-autonomous is I might hard code the tools,
but we're not going to hard code the steps.
So I'm going to tell the agent, you act like a travel agent.
And your task is to help the person book a travel.
And these are the tools that you have accessible to yourself.
And so I'm not hard coding the steps.
I'm just hard coding the tools that you have access
to for yourself.
The more autonomous is the agent decides the steps
and can create the tools.
So that's where you might give actually access
to a code editor, to the agent.
And the agent might actually be able to ping any API in the web,
perform some web search.
It might even be able to create some code
to display data to the user.
It might even be able to perform some calculations.
Like oh, I'm going to calculate the fastest route
to get from San Francisco to New York,
and which one might be the most appropriate
for what the user is looking for.
And then I want to calculate the distance between the airport
and that hotel versus that hotel.
And I'm going to write code to do that.
So it's actually fully autonomous
from that perspective.
So yeah.
Remember those keywords.
Memory, prompts, tools, et cetera.
Now, I presented the flight API, but it does not
have to be an API.
You probably have heard the term MCP or model context protocol
that was coined by Anthropic.
I pasted the seminal article on MCP at the bottom of this slide.
But let me explain in a nutshell why those things would differ.
In the API case, you would actually
teach your LLM to ping an API.
So you would say this is how you ping this API,
and this is the data that it will send you back.
And you would have to do that in a one off manner.
So you would have to build or give
the API documentation of your flight API.
You're booking hotel API, your car rental API.
And then you would give tools for your model
to communicate with those APIs.
It doesn't scale very well versus MCP.
MCP, it's really about putting a system in the middle that
would make it simpler for your LLM to communicate
with that endpoint.
So for instance, you might have an MCP server, an MC client,
where you're trying to communicate
with that travel database or the flight API or MCP.
And your agent might actually just communicate with it
and say, hey, what do you need in order to give me more flight
information?
And that agent will respond by I would like you to tell me
where is the origin flight, where is the destination
and what you're looking for at a high level.
This is my requirement.
OK.
Let me get back to you with in my requirement.
Oh.
You forgot to tell me your budget, whatever.
Oh.
Let me give you my budget, et cetera.
And it's agent to agent communication,
which allows more scalability.
You don't need to hard code everything.
Companies have displayed their MCPs out there,
and your agent can communicate with them
and figure out how to get the data it needs.
Does that make sense?
Yeah.
[INAUDIBLE] rewriting any [INAUDIBLE]
I think it is, ultimately.
The question is, isn't it a shifting issue?
Because anyway, if an API has to be updated,
the MCP has to be updated, is what you say, right?
Yes, that's correct.
But at least it allows the agent to go back and forth
and figure out what the requirements are.
But at the end of the day, ideally, if you're a startup,
you have some documentation.
And automatically, you have an agent or an LLM workflow
that reads that documentation and updates the code
accordingly.
But I agree.
It's not something that is fully autonomous.
Yeah.
i I've seen some security issues.
Why is that possible.
Which security specifically?
[INAUDIBLE]
Yeah.
So are there security issues with MCPs?
So think about it this way.
MCPs, depending on the data that you get access to,
might have different requirements, lower stake
or higher stake.
I'm not an expert at the full range.
But it wouldn't surprise me that when you expose an MCP to--
I think you would a lot of MCC have authentication.
So you might actually need a code
to actually talk to it, just like you would with an API,
or a key.
Yeah, but that's a good question.
I'm not an expert at the security of these systems,
but we can look into it.
Any other questions on what we've
seen with the agentic workflows, APIs, tools, MCPs, memory?
All of that is under progress.
So even memory is not a solved problem by any means.
It's pretty hard actually.
Yes.
You don't need an [INAUDIBLE] The MCP just
makes it easier to access the API, but technically,
[INAUDIBLE]
Exactly, exactly.
Is MCP about efficiency or accessing more data?
It's about efficiency.
Let's say you have a coding agent, and it has an MCP client,
and there's multiple MCP servers that are exposed out there.
That agent can communicate very efficiently with them
and find what it needs.
And it's a more efficient process
than actually displaying APIs and the APIs on that side
and how to ping them and what the protocol is.
But it's not about the data that is
being exposed because ultimately, you control
the data that is being exposed.
You probably, depending on how the MCP is built,
my guess is you probably expose yourself to other risks
because your MCP server can see any input pretty much
from another LLM.
And so it has to be robust.
But yeah.
Super.
So let's look at an example of a step
by step workflow for the travel agent.
So let's say the user says, I want to plan a trip to Paris
from December 15 to 20th with flights,
hotels near the Eiffel Tower, and then an itinerary of must
visit places.
That's the task to the travel agent.
Step two, the agent plans the steps.
So it says, I'm going to find flights.
Use the flight search API to get options for December 15.
Search hotels, generate recommendations for places
to visit, validate preferences, budget, et cetera.
Book the trip with the payment processing API.
That's just the planning, by the way.
Step three, execute the plan, use your tools,
combine the results, and then proactive
user interaction and booking.
It might make a first proposal to the user
and ask the user to validate or invalidate
and then may repeat that planning and execution process.
And then finally, it might actually update the memory.
It might say, oh, I just learned through this interaction
that the user only likes direct flights.
Next time, I'll only give direct flights.
Or I noticed users are fine with three star hotels or four star
hotels.
And in fact, they don't want to go above budget or something
like that.
So that hopefully makes sense by now on how you might do that.
My question for you is how would you know if this works.
And if you had such a system running in production, how
would you improve it?
Yeah.
Lets users rate their experience.
So that's an example.
So let users rate their experience at the end.
That would be an end to end test, right?
You're looking at the user experience through the steps
and say how good was it from 1 to 5, let's say.
Yeah.
It's a good way.
And then if you learn that a user says 1,
how do you improve the workflow?
[INAUDIBLE]
OK.
So you would go down a tree and say, OK, you said 1.
What was your issue?
And then the user says the prices were too high, let's say.
And then you would go back and fix that specific tool or prompt
or, yeah, OK.
Any other ideas?
[INAUDIBLE]
Yeah, good.
So that's a good insight.
Separate the LLM related stuff from the non-LLM related stuff,
the deterministic stuff.
The deterministic stuff, you might
be able to fix it more objectively essentially.
Yeah.
What else?
So give me an example of an objective issue
that you can notice and how you would fix it
versus a subjective issue.
Yeah.
[INAUDIBLE]
So let's say you say there's the same flight,
but one is cheaper than the other, let's say.
It's objectively worse.
And so you can capture that almost automatically.
Yeah.
So you could actually build evals
that are objective, that are tracked across your users.
And you might actually run an analysis after
and see that for the objective stuff,
we notice that our LLM AI agent workflow is bad with pricing.
It just doesn't read price as well because it always
gives a more expensive option.
Yeah.
You're perfectly right.
How about the subjective stuff?
Do you choose a direct or indirect flight
if the indirect is a little bit cheaper?
Yeah.
Good one.
Do you choose a direct flight or an indirect flight
if the indirect is cheaper but the direct is more comfortable?
Yeah.
That's a good one actually.
So how would you capture that information.
Let's say this is used by thousands of users.
Could you feed something in [INAUDIBLE]
Could you feed something in?
Yeah, I mean, you could--
could feed something in about the user preferences?
Well, you could build a data set that
has some of that information.
So you build 10 prompts, where the user is asking specifically
for a direct--
saying that I prefer direct flights because I
care about my time, let's say.
And then you look at the output and you actually
give a good example of a good output,
and you probably are able to capture
the performance of your agentic workflow on this specific eval.
Does it prioritize?
Does it understand price conscious--
is it price conscious, essentially,
and comfort conscious?
Yeah.
What about the tone?
Let's say the LLM right now is not very friendly.
How would you notice that, and how would you fix it?
Yeah.
Have the test user run the prompt
and see if there's something wrong with that.
OK.
Have a test user run the prompt and see if there's
something wrong with that.
Tell me about the last step.
How would you notice that something is wrong?
So a couple of tests [INAUDIBLE] evaluates
the response and [INAUDIBLE]
Yeah.
I agree with your approach.
Have LLM judges that evaluate the response
against a certain rubric of what politeness looks like.
So here in this case, you could actually
start with error analysis.
So you start, you have 1,000 users.
And you can pull up 20 user interactions
and read through it.
And you might notice, at first sight,
the LLM seems to be very rude.
It's just super, super short in its answers,
and it's not very helpful.
You notice that with your error analysis manually.
Then you go to the next stage.
You actually put evals behind it.
You say, I'm going to create a set of LLM judges
that are going to look at the user interaction
and are going to rate how polite it is.
And I'm going to give it a rubric.
Then what I'm going to do is I'm going to flip my LLM.
Instead of using GPT-4, I'm going to use Grok.
And instead of using Grok, I'm using Llama.
And then I'm going to run those three LLMs side by side,
give it to my LLM judges, and then get my subjective score
at the end to say, oh, x model was more polite on average.
Yeah.
Perfectly right.
That's an example of an eval that is very specific
and allows you to choose between LLMs.
You could actually do the same eval not across LLMs,
but fixed the LLM, change the prompt.
You actually, instead of saying act like a travel agent,
you say act like a helpful travel agent.
And then you see the influence of that word on your eval
with the LLM as judges.
Does that make sense?
OK.
Super.
So let's move forward and do a case study with evals.
And then we're almost done for today.
Let's say your product manager asks you to build an AI
agent for customer support, OK?
Where do you start?
And here is an example of the user prompt.
I need to change my shipping address for order, blah, blah,
blah.
I move to a new address.
So what do you start if I'm giving you that project?
Yes.
We search online for existing models and [INAUDIBLE]
So do some research.
See benchmarks and how different models
perform at customer support.
And then pick a model.
That's what you mean.
Yeah.
It's true you could do that.
What else could you do?
Yeah.
[INAUDIBLE]
OK.
Yeah, I like that.
Try to decompose the different tasks that it will need
and try to guess which ones will be more of a struggle, which
ones should be fuzzy, which ones should be deterministic.
Yeah, you're right.
[INAUDIBLE]
Yeah.
Similar to what you said.
That's what I would recommend as well.
You say I would sit down with a customer support
agent for a day or two, and I would decompose the tasks
that are going through.
I will ask them, where do they struggle?
How much time it takes?
Yes.
That's usually where you want to start with task decomposition.
So let's say we've done that work, and we have this list.
I'm simplifying.
But the customer support agent, human, typically
would extract key info, then look up
in the database to retrieve the customer record.
Then check the policy.
Are we allowed to update the address,
or is it a fixed data point?
And then draft a response email and sends the email.
So we've decomposed that task.
Once you've decomposed that task,
how do you design your agentic workflow?
Yes.
[INAUDIBLE]
Exactly.
So to repeat, you're going to look
at the decomposition of tasks, get an instinct of what's fuzzy,
what's deterministic, and then determine
which line is going to be an LLM one shot, which one will require
maybe a RAG, which one will require a tool, which one will
require memory, which one--
So you will start designing that map.
Completely right.
That's also what I would recommend.
You might actually draft it and say, OK, I take the user prompt.
And the first step of my task decomposition
was extract information that seems to be a vanilla LLM.
You can guess that the vanilla LLM would probably
be good enough at extracting the user wants
to change their address, and this is the order number
and this is the new address.
You probably don't need too much technology
there other than the LLM.
The next step, it feels like you need a tool because you're
actually going to have to look up in the database
and also update the address.
So that might be a tool, and you might
have to build a custom tool for the LLM
to say, let me connect you to that database
or let me give you access to that resource with an MCP.
After that probably need an LLM again to draft the email,
but you would probably paste confirmation.
You would paste the confirmation that your address
has been updated from x to y.
And then the LLM will draft an answer.
And of course, just to not forget,
you might need a tool to send the email.
You might actually need to post something to
for the email to go out.
And then you'll get the output.
Does that make sense So exactly what you described.
Now moving to the next step.
Once we have-- we compose our tasks.
Then we have designed an agentic workflow around it.
It took us five minutes.
In practice, it would take you more
if you're building your startup on that.
You want to make sure your task decomposition is accurate,
your thing is accurate here, and then
you can have a lot of work done on every tool
and optimize it and latency and cost.
But let's say, now we want to know if it works.
And I'm going to assume that you have LLM traces.
LLM traces are very important.
Actually, if you're interviewing with an AI startup.
I would recommend you in the interview process to ask them,
do you have LLM traces?
Because if they don't have LLM traces,
it is pretty hard to debug an LLM system because you don't
have visibility on the chain of complex prompts that were called
and where the bug is.
And so it's a basic part of an AI startup
stack to have an LLM traces.
So let's assume you have traces.
How would you know if your system works?
I'm going to summarize some of the things I heard earlier.
You gave us an example of an end to end metric.
You look at the user satisfaction at the end.
You can also do a component-based approach
where you actually will look at the tool, the database updates,
and you will manually do an error analysis and see,
oh, the tool actually always forgets to update the email.
It just fails at writing.
And I'm going to fix that.
This is deterministic pretty much.
Or when it tries to send the email
and ping the system that is supposed to send the email,
it doesn't send it in the right format.
And so it bugs at that point.
Again, you could fix that.
Draft of the email.
The LLM doesn't do a great job.
It's not very polite at drafting the email.
So you could look at component by component,
and it's actually easier to debug than to look at it
end to end.
You would probably do a mix of both.
Another way to look at it is what is objective
versus what is subjective?
So for example, an objective example
would be a DLRM extracted the wrong order ID.
The user said my order ID is X, and the LLM,
when it actually looked up in the database,
it used the wrong order ID.
This is objectively wrong.
You can actually write a Python code
that checks that, checks just the alignment between what
the user mentioned and what was actually pasted in the database
or for the lookup.
You also have subjective stuff, which we talked about,
where you probably want to do either human rating or LLM
as judges.
It's very relevant for subjective evals.
And finally, you will find yourself
having quantitative evals and more qualitative evals.
So quantitative would be percentage of successful address
updates.
The latency.
You could actually track the latency component-based
and see which one is the slowest.
Let's say sending the email is five seconds.
It's too long, let's say.
You would notice component based or the full workflow.
And then you will decide, where am I optimizing my latency,
and how am I going to do that?
And then finally, qualitative.
You might actually do some error analysis
and look at where are the hallucinations?
Where are the tone mismatches?
Are the user confused, and by what they're confused?
That would be more qualitative.
And typically, it would take more white glove approaches
to do that.
So here's what it could look like.
I gave you some examples.
But you would build evals to determine
objectively, subjectively, component-based, end
to end based, and then quantitatively and
qualitatively, where's your LLM failing
and where it's doing well.
Does that give you a sense of the type of stuff
you could do to fix or improve that agentic workflow?
Super.
Well, that was our case study on evals.
We're not going to delve deeper into it.
But hopefully, it gave you a sense of the type of stuff
you can do with LLM judges, with objective,
subjective, component-based, end to end, et cetera.
Last section on multi-agent workflows.
So you might ask, hey, why do we need multi-agent workflow when
the workflow already has multiple steps,
already calls the LLM multiple times, already gives them tools.
Why do we need multiple agents?
And so many people are talking about multi-agent system online.
It's not even a new thing, frankly.
Multi-agent systems have been around for a long time.
The main advantage of a multi-agent system
is going to be parallelism.
It's like is there something that I
wish I would run in parallel, sort of independently,
but maybe there are some things in the middle?
But that's where you want to put a multi-agent system.
It's when it's parallel.
The other advantage that some companies
have with multi-agent systems is an agent can be reused.
So let's say in a company, you have an agent that's
been built for design.
That agent can be used in the marketing team,
and it can be used in the product team.
And so now you're optimizing an agent,
which has multiple stakeholders that can communicate with it
and benefit from its performance.
Actually I'm going to ask you a question
and take a few, maybe a minute to think about it.
Let's say you were building smart home
automation for your apartment or your home.
What agents would you want to build?
Yeah.
Write it down.
And then I'm going to ask you in a minute
to share some of the agents that you will build.
Also, think about how you would put
a hierarchy between these agents,
or how you would organize them, or who
should communicate with who.
OK?
OK.
Take a minute for that.
Be creative also because I'm going to ask all of your agents,
and maybe you have an agent that nobody has thought of.
OK.
Let's get started.
Who wants to give me a set of agents
that you would want for your home, smart home.
Yes.
The first is like a set of agents [INAUDIBLE]
OK.
So let me repeat.
You have four agents, I think, roughly.
One that tracks biometric, like where are you in the home?
Where are you moving?
How you're moving, things like that.
That sort of knows your location.
The second one determines the temperature of the rooms
and has the ability to change it.
The third one tracks energy efficiency
and might give feedback on energy and energy usage.
And might be, I don't know, maybe
it has the control over the temperature as well.
I don't know actually.
Or the gas or the water, might cut your water at some point.
And then you have an orchestrator agent.
What is exactly the orchestrator doing?
It passes instructions [INAUDIBLE]
OK.
Passes instructions.
So is that the agent that communicates mainly
with the user?
So if I'm coming back home and I'm
saying I want the oven to be preheated,
I communicate with the orchestrator,
and then it would funnel to another agent.
OK.
Sounds good.
Yeah.
So that's an example of, I want to say,
a hierarchical agentic multi-agent system.
What else?
Any other ideas?
What would you add to that?
Yeah.
[INAUDIBLE]
Oh, I like that.
That's a really good one.
So let me summarize.
You have a security agent that determines if you can enter
or not.
And when you enter, it understands who you are.
And then it gives you certain sets
of permissions that might be different depending
of if you're a parent or a kid.
Or you might have access to certain cars and not others.
Or your kid cannot open the fridge, or I don't know.
Something like that.
Yeah.
OK, I like that.
That's a good one.
And it does feel like it's a complex enough workflow where
you want a specific workflow tied to that.
I agree.
What else?
Yes.
[INAUDIBLE] So you can get more complicated.
So high energy savings with whether or not you
or someone else can be blind to those in the house or also
when you tap into the grid.
Yeah So another thought I have as well is much harder
to track in the grocery store.
But understanding what's in your fridge.
OK
Well, that's really good actually.
So you mentioned two of them.
One is maybe an agent that has access to external APIs that
can understand the weather out there, the wind, the sun,
and then has control over certain devices at home.
Temperature, blinds, things like that, and also understands
your preferences for it.
That does feel like it's a good use case because you could give
that to the orchestrator, but it might lose itself
because it's doing too much.
And also, these problems are tied together,
like temperature outdoor with the weather API
might influence the temperature inside,
how you want it, et cetera.
And then the second one, which I also like,
is you might have an agent that looks at your fridge
and what's inside.
And it might actually have access
to the camera in the fridge, for example,
and know your preferences and also has
access to the e-commerce API to order
Amazon groceries ahead of time.
I agree.
And maybe the orchestrator will be the communication line
with the user, but it might communicate with that agent
in order to get it done.
Yeah.
I like those.
So those are all really good examples.
Here is the list I had up there.
So climate control, lighting security, energy management,
entertainment, notification agent,
alerts about the system updates, energy saving, and orchestrator.
So all of them you mentioned actually.
And then we didn't talk about the different interaction
patterns, but you do have different ways to organize
a multi-agent system.
Flat, hierarchical.
It sounds like this would be hierarchical.
I agree.
And the reason is UI/UX, is I would rather
have to only talk to the orchestrator,
rather than have to go to a specialized application
to do something.
Like it feels like the orchestrator
could be responsible for that.
And so I agree, I would probably go for a hierarchical setup
here.
But maybe you might also add some connections
between other agents, like in the flat system
where it's all to all.
For example, with climate control and energy,
if you want to connect those two,
you might actually allow them to speak with each other.
When you allow agents to speak with each other,
it is basically an MCB protocol, by the way.
So you treat the agent like a tool, exactly like a tool.
Here is how you interact with this agent.
Here is what it can tell you.
Here is what it needs from you, essentially.
OK super.
And then without going into the details,
there are advantages to multi-agent workflows
versus single agents, such as debugging.
It's easier to debug a specialized agent
into debug an entire system.
Parallelization as well.
It's easier to have things run in parallel,
and you can earn time.
There are some advantages to doing that,
and I'll leave you with this slide if you want to go deeper.
Super.
So we've learned so many techniques to optimize LLMs,
from prompts to chains to fine tuning, retrieval,
and to multi-agent system as well.
And then just to end on a couple of trends I want you to watch.
I think next week is Thanksgiving, is that it?
It's Thanksgiving break.
No, the week after.
OK.
Well ahead of the Thanksgiving break.
So if you're traveling, you can think about these things.
What's next is in AI, I wanted to call out a couple of trends.
So Ilya Sutskever, one of the OGs of LLMs and OpenAI
co-founder, raised that question about are we plateauing or not.
The question are we going to see in the coming years LLM sort
of not improve as fast as we've seen in the past?
It's been the feeling in the community
probably that the last version of GPT
did not bring the level of performance
that people were expecting, although it did make
it so much easier to use for consumers because you don't need
to interact with different models.
It's all under the same hood.
So it seems that it's progressing,
but the plateau is unclear.
The way I would think about it is the LLM scaling laws tell us
that if we continue to improve compute and energy,
then LLMs should continue to improve.
But at some point, it's going to plateau.
So what's going to take us to the next step?
It's probably architecture search.
Still a lot of LLMs, even if we don't
understand what's under the hood or probably
transformer-based today.
But we know that the human brain does not operate the same way.
There's just certain things that we
do that are much more efficient, much faster.
We don't need as much data.
So theoretically, we have so much
to learn in terms of architecture search
that we haven't figured out.
It's not a surprise that you see those labs hire
so many engineers.
Because it is possible that in the next few years,
you're going to have thousands of engineers trying
to figure out the different engineering hacks and tactics
and architectural searches that are
going to lead to better models.
And one of them suddenly will find the next transformer,
and it will reduce by 10x the need for compute and the need
for energy.
It's sort of if you read Isaac Asimov's Foundation series.
Individuals can have an amazing impact on the future because
of their decisions.
Whoever discovered transformers had a tremendous impact
on the direction of AI.
I think we're going to see more of that in the coming
years, where some group of researchers that is iterating
fast might discover certain things that would suddenly
unlock that plateau and take us to the next step,
and it's going to continue to improve like that.
And so it doesn't surprise me that there's so many companies
hiring engineers right now to figure out
those hacks and those techniques.
The other set of gains that we might see
is from multi-modality.
So the way to think about it is we've had LLMs first text-based,
and then we've added imaging.
And today, models are very good at images.
They're very good at text.
It turns out that being good at images and being good at text
makes the whole model better.
So the fact that you're good at understanding a cat image
makes you better at text as well for a cat.
Now you add another modality like audio or video.
The whole system gets better.
So you're better at writing about a cat
if you know what a cat sounds like,
if you can look at a cat on an image as well.
Does that make sense?
So we see gains that are translated from one modality
to another, and that might lead in the pinnacle of robotics
where all these modalities come together.
And suddenly, the robot is better at
running away from a cat because it understands
what a cat is, how it sounds like,
what it looks like, et cetera.
That makes sense?
The other one is the multiple methods working in harmony.
In the Tuesday lectures, we've seen supervised learning,
unsupervised learning, self-supervised learning,
reinforcement learning, prompt engineering, RAGs, et cetera.
If you look at how babies learn, it
is probably a mix of those different approaches.
Like a baby might have some meta learning, meaning it
has some survival instinct that is
encoded in the DNA most likely.
And that's like the baby's pre-training, if you will.
On top of that, the mom or the dad is pointing at stuff
and saying bad, good, bad, good.
Supervised learning.
On top of that, the baby is falling on the ground
and getting hurt.
And that's a reward signal for reinforcement learning.
On top of that, the baby is observing other people
doing stuff or other babies doing
stuff, unsupervised learning.
You see what I mean?
We're probably a mix of all these methods,
and I think that's where the trend is going, is
where those methods that you've seen in CS230
come together in order to build an AI system that learns fast,
is low latency, is cheap, energy-efficient,
and makes the most out of all of these methods.
Finally, and this is especially true at Stanford,
you have research going on that you would consider human-centric
and some research that is non-human centric.
By human-centric, I should say human approaches
that are modeled after the brain and approaches that
are not modeled after humans.
Because it turns out that the human body is very limiting.
And so if you actually only do research
on what the human brain looks like,
you're probably missing out on compute and energy and stuff
like that that you can optimize even
beyond neuronal connections in the brain,
but you still can learn a lot from the human brain.
And that's why there are professors that are running labs
right now that try to understand,
how does back propagation work for humans?
And in fact, it's probably that we don't have back propagation.
We don't use back propagation, we only do forward propagation,
let's say.
So this type of stuff is interesting research
that I would encourage you to read if you're curious
about the direction of AI.
And then finally, one thing that's going to be pretty clear,
I call it all the time, but it's the velocity
at which things are moving.
You're noticing, part of the reason
we're giving you a breadth in CS230
is because these methods are changing so fast.
So I don't want to bother going and teaching you
the number 17 methods on RAG that
optimizes the RAG because in two years,
you're not going to need it.
So I would rather you think about what
is the breadth of things you want to understand.
And when you need it, you are sprinting and learning
the exact thing you need faster because the half life of skill
is so low.
You want to come out of the class with a good breadth
and then have the ability to go deep whenever
you need after the class.
And so that's sort of how that class is designed as well.
Yeah.
That's it for today.
So thank you.
Thank you for participating.