Okay. So, um, so let's continue the
journey we started last time. Um so what
we're going to do uh you know if you
remember in the last class we showed how
we can actually build an auto
reggressive large language model uh aka
a causal large language model um using
this not this idea of a causal encoder a
transformer causal encoder and then we
showed how you can actually take a bunch
of sentences and use next word
prediction and just run it through and
boom you get GPD3 okay so that's what we
saw last time I want to point out a sort
of an important clarification slash
correction which is that when we work
with large language models uh unlike
when we work with BERT uh for instance
when we work with these kinds of causal
models actually uh when the contextual
embeddings come out you don't actually
have to use ReLU activations here you
can literally just run it through just a
single dense layer with linear
activations and then pass it into a
softmax and boom you're done okay so
that's how GPD3 and all these models are
trained u and the other thing I want to
point out which may not have clear is
that what what is coming out of these
this dense layer right this vector is as
long as your vocabulary
because only then when it goes into the
soft max you're going to get
probabilities which are as long as your
vocabulary which means that you get to
pick one word or token out of that
entire 50,000 long vocabulary
okay so so just I just want to point
that out because I think it's easy for
us to sort of get a little confused
because of this little difference
between the way uh masked language
models like BERT work and causal
language models like GPD3.
Okay, so now let's continue with we have
we know how to build GPD3. So like what
about GPD and GPD2 like what's up to
them? Why is GPD3 so famous and not
GPD2? Right? So turns out well first of
all you folks know that GPD stands for
generative pre-trained transformer. Now
like GPD3
two GPD2 and GPD1 were trained in
basically the same fashion. Predict the
next word uh same fashion the same sort
of transformer stack except that GPT3
was trained on much more data because
the underlying transformer stack had
many more layers. Okay, so it is a much
bigger stack meaning lots more
parameters and therefore you need lots
more data to train it well. Okay, so
that was really the only difference. The
difference was literally one of scale,
scale of network and scale of data. And
unlike GPT and GPD2, GPD3 even though it
was trained basically the same way with
the same kind of network, it was one of
the situations where more became
different. Okay, there was almost like
some sort of phase change that happened
between two and three. Unlike GPD and
GPD2, GPD3 could do amazingly coherent
continuations of any starting prompt,
right? Um so for example, if you have
this little prompt which says the
importance of being on Twitter by Jerome
K Jerome who was a famous humorist and
then you give it this prompt, right?
Ending with the word it, it produces
this continuation which is really like
strikingly good. And if any of you have
read Jerome K Jerome and if you read
this thing, you'll be like, "Wow, that
actually sounds like Jerome K Jerome."
Right? So amazing continuations the the
but the interesting thing here is not so
much the continuation it's the fact that
the same prompt you give it a two or GPT
it won't do any it won't be very good in
fact after the first one two or three
sentences it'll sort of become sort of
incoherent and meander and start
rambling this thing can keep faking it
for a long longer time right that's the
amazing thing that was unexpected re
researchers did not expect this okay and
but it wasn't good at following your
instructions
So for instance, if you ask it, help me
write a short note, introduce myself to
my neighbor. This is the kind of thing
it'll come up with. And you can actually
run it yourself. You can actually go to
GPD3 on the playground. I think GPD3 is
still available in the playground. U if
it is, you can actually start try
running these prompts. You will start
getting garbage very quickly, right? And
the reason so for example here, help me
write a short note. It says, what's a
good introduction to a resume? Rumé for
some reason has glombmed down to resume.
I have no idea why. Right? But the
reason it's doing stuff like this is
because a lot of the training data it
was trained on are basically lots of
lists of things.
So when you say for example um you know
the the the capital of Paris continue
it'll come back with the capital sorry
the capital of France continue it say
the capital of France is Paris the
capital of you know uh Hungary is
Budapest and so on. It just start coming
up with a list. So it's sort of very
list driven right? it thinks that you
you need to complete some sort of list,
right? That's what's going on here. And
so it's not very good. So it doesn't
realize that you're actually asking it
to do something specific.
So this is the problem when you have an
autocomplete thing which doesn't realize
what you're asking it. It just thinks
that you're it's just an autocomplete.
So um now in addition to these unhelpful
answers, it can also produce offensive
answers, factually incorrect answers and
so on and so forth. The list of bad
things it can do is long. So why does it
do that? Why does it produce unhelpful
answers? Well, you know, as you recall,
it was only trained to predict the next
word. It wasn't explicitly trained to
follow instructions, right? So, it
seems, you know, reasonable that if it's
simply trying to guess the next word
repeatedly, it can't really do anything
more. Like, how can it figure out that
there's an instruction that it needs to
follow, right? Unless the training data
on the net was all instructional, which
it clearly is not.
So light bulb idea, right? Let's
explicitly train it with instruction
data,
right? Let's just train it with
instruction data. And so OpenAI
developed an approach called instruction
tuning to do exactly this. Um, and this
paper is the paper that sort of was the
breakthrough. Okay, this is what
actually put Chad on the map. So, and
it's very readable. So, I would
encourage you to check it out if you're
curious.
And so so we had GPT, GPD2, GPD3, you
know, just bigger and bigger models
trained the same way. And then we run
into the problem that it can't handle
instructions. So we do instruction
tuning to get to 3.5, also called
instruct GPT. And then a small tweak
after that gets you chat GPT. Okay. And
by the way, this step here, there are
really two things going on in this as
you will soon see. I'm just calling it
instruction tuning just to so that I
don't have to say some long thing every
single time. it this is not a consistent
piece of terminology. So just just
beware aware of that's all. So all right
first step they got a bunch of people to
write highquality answers to questions
and they created about 12,500 such
question answer pairs. So for example
let's say this was the question explain
the moon landing to a six-year-old in a
few sentences. Believe it or not, GPD3's
answer to that question was another
question
because it thinks there's a list of
questions it needs autocomplete, right?
So, it comes up with explain the theory
of gravity to a six-y old. It's like one
of those people when you ask them a
question, they ask you a question back,
right? So, what what they did is they
said, "Okay, let's create a nice answer
to this question." And here's a human
created answer. People went to the moon
in a big rocket, walked around, blah
blah blah, right? Much better answer to
that question. And so once you create
these 12,500 question answer pairs as
training data, we just trained GPD3 some
more using Xword prediction as before.
No difference. So, so here is the input
explain the moon landing blah blah blah
blah. This is the question and then we
have the answer right there. And then we
we take that answer, move it to the
right and just shift it up
so that when it finishes sentences, it
needs to predict people. And then you
give it people, it needs to predict went
and so on and so forth. Just like we saw
before, the cat sat on the mat became
the cat sat on the cat sat on the mat on
the right shifted, right? That's what
makes prediction possible and necessary.
So that's what they did. This co this is
step one. Okay, same as same as before.
And once you do that, it turns out this
step is called supervised fine-tuning.
It really helped. GPD3 once you
supervised fine-tuned it was much much
better at following instructions. But
there's a small problem with this
approach. It takes a lot of money and
effort to have humans write highquality
answers to thousands of questions,
right? It takes a lot of money. So the
question is, what can we do, right? What
is easier than writing a good answer to
a question?
Well, what? Okay. Uh, all right. Uh, how
about somebody from this side?
>> Yeah, Joseph.
>> Perhaps writing a question for an
answer.
>> Oh, that's actually a good one. Yeah.
Yeah, I like that. Um, so given an
answer, find find a question. And while
that is not what I'm going to talk about
here, that technique is actually used
very heavily in LLMs. Uh, and so but
that that's great. Very creative. Uh
Mark,
>> thumbs up. Thumbs down.
>> Sorry.
>> Thumbs up or thumbs down?
>> Thumbs up or thumbs down. Exactly.
Because all of us, everyone loves to be
a critic. It's much better easier to be
a critic than to be a creator. Right. So
what do we do? We basically say, let's
rank answers written by somebody else.
Which begs the question, who's going to
write those answers? And that's where
there's a brilliant answer to that
question which is
Wikipedia,
Reddit.
We will just ask GPT3 to write the
answers.
It might be crap, but we don't care
because we can rank them.
So we ask GPT3 to get generate several
answers to the question. And how can we
generate several answers? Because we can
do sampling.
We can do sampling.
The fact that we had these stoastic
outputs because of sampling is now a
feature, not a bug. Okay, we create lots
of different answers to the question. We
feed it a question, get like three
answers out. Just run it three times,
get three answers out with a nice
temperature of like one or 1.1 or
something so that it's nice and random,
right? Um, and then we literally have
humans just rank them, do the thumbs up,
thumbs down, just rank them from most
useful to least useful. Okay, so this
step is a step two of instruction
tuning. So OpenAI collected 33,000
instructions, fed them to GB3, generated
answers and had humans rank them. And
once you do that, once you do this, you
can assemble a beautiful training data
set, right? And so basically what we
have is that we have an instruction and
let's say we have just two answers A and
B. And in in practice they you can have
many many answers which we rank but just
for simplicity I'll go with Mark's
thumbs up thumbs down sort of answer
which is let's assume only you have two
answers to every question right and so
and the human has said I prefer this to
that that's it right so we have a data
set now where the data point is
instruction preferred answer is A the
other answer is B yeah
>> um the thumbs up thumbs down uh
technique that we're talking is that why
We're attaching to now we also use
thumbs up thumbs down. It's using only
answers to train.
>> Exactly. Right.
>> Yeah. So yeah, all the models have the
thumbs up thumbs down stuff going on
somewhere. They are all collecting data
for this step.
>> Thank you.
>> Yeah. It's sort of the old adage, right?
Uh if you're not sure who the product
is, you are the product. So it's one of
those things. Yeah.
So if we understand correctly when we
see thumbs up thumbs down it does mean
that chat is going to trade on our data
right
>> unless you opt out. Yeah. So if you
actually go to the chaty controls there
is something called data controls or
something you can toggle it to off but I
think when I last checked if you toggle
it to off you lose your chat history. So
they have hobbled that feature to
prevent people from setting it to off as
much as possible. Yeah, clever.
But you can opt out and if you use the
API as opposed to the web interface,
you're automatically opted out. So you
have to deliberately opt in. And if you
use the versions that are available
through Microsoft Azure and so on and so
forth, there are all kinds of very safe
controls and stuff like that. In fact, I
think the Microsoft co-pilot license
that MIT has uh I think the default is
opted out.
Okay. So to go here, once you have this
data point, you can build something
called a reward model. Okay. And this is
a very clever piece of work. So what you
do is you have an instruction, right?
You have a preferred answer and you have
the other answer. You feed it to a
network. Okay? You feed it to a network.
This is just a a nice language model,
right? It's just a language model. And
the language model produces a number
which measures how good this thing is,
right? How good an answer is this to
that particular instruction. So you get
two you get a rating here, you get a
rating here and then what you do is you
run it through a little loss function
which
essentially encourages the model to give
higher numbers to the better answer.
It's the same model. You just run the
the question and the first answer,
question and the second answer. You get
these two numbers. And then initially
those numbers are just random. But then
you tell the model, hey, this is the
preferred thing. Make sure the preferred
answers
uh rating the R value is higher than the
other number because more is better.
Higher is better. Okay? And you can
actually since you and this thing is
just a sigmoid here, right? It's
basically take the difference of these
two things. do a sigma and take the
logarithm and you can actually convince
yourself afterwards and I encourage you
to do that to to check for yourself that
if we actually
give a higher number to the better
answer the loss will be lower and since
we are minimizing loss we're essentially
training the network to always to try to
give higher ratings to better answers
that's it so that's the approach uh did
you have a yeah Ben
So you could imagine training um
training the model and only the good
answers is the idea of having both that
the model is actually learning what
makes good
>> correct. Exactly. Much like if you want
to build a dog cat classifier, you have
to show pictures of both.
>> Yeah.
>> So u I understand the feedback mechanism
of thumbs up thumbs down but there are a
lot of times when the popular response
is not the accurate one. So uh is there
a way that they actually have a layer to
correct?
>> Yeah, good question Swati. So uh as it
turns out um the all these companies
like OpenAI, they have like a huge
document 100 200 pages longs you know
very very bulky document which instructs
and teaches the labelers the rankers to
how to rank these things. So they have
to follow these very strict guidelines
to precisely handle like strange corner
cases and things like that. And that
document is on the web. You can dig it
up, right? And it's actually very
instructive to read through it, right? I
think they put it out on the web because
they wanted to convince people that
they're going to inordinate trouble to
make sure the rankings are actually
good. U do you have a question? Comment.
Okay. All right. So um so back to this
and how how do you train this thing? SGD
because you have a network it's coming
up with an answer you have some way to
know if that answer is good or bad right
better answers of lower loss back
propagation through the network keep
updating the weights and boom you're
done
okay and once you do that this reward
model can provide a numerical rating for
any any instruction answer pair you just
give it an instruction you give it an
answer right could be a crappy answer
good answer it just tells you how good
it is which means right So in this case
for example maybe it's going to give you
like a nice number 1.5 uh uh which is
you know 1.5 for this this answer but
then a better answer comes along or 3.2
right what we have done by doing this
whole thing this modeling is that we
have essentially we have learned how
humans rank responses
because we can only have humans rank
responses for some finite number of
questions. What we really want to do is
to do this to automate that ranking
process so that we can just do it for
like tens of thousands of questions
really fast. Right? So we have
essentially built a model of how humans
rank things, right? Which is beautiful.
A lot of the stuff here is all very
self-reerential which I find very
elegant. Anyway, so this can be used to
improve GP3 even further. So we take the
instruction as before, we feed it. It
gives you some answer and then we feed
this instruction and the answer to our
newly minted reward model. It gives us a
numerical rating and then this is the
key step. We take this numerical rating
and then we use this rating to nudge the
internal weights of GPD3 in the right
direction. Right? This nudging
uses a technique called reinforcement
learning.
Right? Which just in the interest of
time we can't get into in this lecture.
But that that's a technique you use to
nudge these things in the right
direction.
So that's what we do. That's
reinforcement learning. We nudge it in
the right direction.
And OpenAI did this with 31,000
questions.
Okay. Nudge, nudge, nudge, nudge, nudge.
And when you do that, you get GPD
3.5/ingpd.
Okay. Uh that's it. And now by the way
this step here is called reinforcement
learning with human feedback because we
use reinforced learning and since humans
rank the answers which tread to the
building of the reward model we get
human feedback. Okay, that's
reinforcement learning with human
feedback. Yeah.
>> Yeah. I have [clears throat] a question
regarding the the type of questions that
they're using. I can imagine like maybe
there are very simple questions to
answer because I'm thinking now you can
ask GPD like for example respond this as
a pirate or something like that that is
kind of it's going to be harder to train
if you have bunch of questions that are
having like small interactions and then
there is the question like
>> that's a good question. So the quality
of the questions in the data set clearly
is a big factor because if you have
simple simplistic questions it won't be
able to handle complex questions later
on. So what it's a good question. So
what how so the qu so that actually begs
the question of where did they get these
questions from
so they actually got it from their API.
So people are asking GPD3 on the API
right before it became 3.5 people are
asking all the API was already available
you know fully available commercially
available a lot of people are building
products on it already by then and so
they collected all those questions and
filtered them for quality and that was
the question set that they used and then
they judiciously added to it with human
created questions but they couldn't do a
lot of that because it's expensive to do
that but collecting stuff that somebody
else is asking your API already very
easy
Yeah, Tomaso,
>> uh, this might be more of a
philosophical question, but, uh, the
human bias that's present in the small
subset of human labelers that they've
chosen gets eventually compounded in
this model that we often consider as the
source of objective truth.
>> Yes.
>> Yeah, that's very true. Um I think the
the reward model is probably very
faithfully learn all the biases of the
human labelers which is why they have
these very complex u sort of frameworks
and guidelines to try to prevent the
bias from happening to mitigate it. So
for example they might give the same
question and set of possible answers to
many many different labelers and only if
people pick the same ranking they might
use it so that at least inter labeler
bias can be minimized right but if
everybody's sort of biased in the same
direction it won't protect you against
that. Um so yeah in general there's a
whole work that's being done to try to
debias these things and build them
without you know too much bias in them.
It's like a whole world unto itself
which we just don't have time to get
into. Uh Olivia,
>> um depending on the medium that's being
returned by these models, would there be
more than one reward model? Because
isn't that what Gemini
>> would there be more than one
>> reward model? Because isn't this what
Gemini is running into issues with right
now with their image generation is the
bias that they try to
>> Yeah. So the Gemini business that's
going on, it's unclear what's causing
it. Um it may be in this step, maybe
they were a little overzealous in
preventing certain things from
happening.
Some of these uh systems also have um
they will actually intercept the
question that you ask and then route it
differently based on what they sense is
sitting around in the question. So there
could be pre-processing post-processing
a lot of stuff that goes on. So unclear
to me where in the pipeline and it could
be more than one place these things may
be entering. So yes, so here may very
well be where it actually enters a
situation where people are people are
told if you see any sort of this kind of
answer downrank it right don't uprank it
and then it learns that ranking very
faithfully and then proceeds to apply it
where it does should not be applied so
that does happen uh Joselyn you had a
question
>> um I think I still I still don't totally
understand why so when I ask chat GBT a
question even in a lengthy response it
doesn't wander away from the topic that
I'm asking about right and so
understanding that it it's predicting
each word it's sort of taking a random
walk from one word to the next in some
sense
>> but each word it utters
>> now becomes part of the input to the
next word it utters
>> right
>> so it's not truly random walk in that
sense so the next step is not
independent of the previous step
>> it depends on what it depends on the
journey so far so it's going to try to
be very consistent with the journey so
far
>> okay
>> does the
does this part with um sort of
fine-tuning it on these question answer
sets. Does this play some role in it
being able to constrain itself and not
meander away?
>> I don't think so. I think this is more
to make sure that you know it does the
weights generally tend to produce the
right answer. Now what one of the things
that is possible is that when when I'm
let's say I'm a ranker and I'm looking
at a few different answers I'm you know
I have to figure out if the answer is
helpful if it is accurate if it is uh
you know non-toxic right things like
that and part of the rubric for
evaluating these answers could be their
coherence right so it could also be that
they are saying short coherent answers
are better than long coherent answers
but once you adjust for length Maybe
coherence is more important, right? It
could be any number of these things. So
it could play a role in that.
>> So just sort of one small followup. So
in other words, when it's when it's
learning from these question and answer
pairs, it's able to look at
[clears throat] the whole response and
learn something about the whole response
rather than just one word at a time,
right?
>> Correct. Yeah. The the entire question
is being ranked.
>> Yeah.
>> Correct. Correct.
>> Yeah. On a related note, um when it's
generating a new word on a topic, does
the attention pertain to the entire
prior text or can you have like
traveling attention? So like last five
word.
>> So yeah, the short answer is yeah, you
can you can it's called sliding window
attention. It can be done. They
typically tend to do it not uh so much
because they want to focus more on the
the recent words, but more because it
actually makes it very compute
efficient. U that's why they do it. So
it's called sliding window attention.
You can Google it.
>> So normally it's full attention.
>> Normally it's full default is full
attention.
Okay. So that's what they did. Uh and
when they did that and by the way as I
think you pointed out that's exactly
what's going on. You're training the
reward model with these thumbs up and
thumbs down. U hold on the questions.
And so if you give it the same question
to GPD 3.5 in GPD amazing answer.
Okay, like night and day difference,
amazingly good answer. Um, and so and
then to go from 3.5 to CH GBT, they
basically followed the exact same
playbook except that because they wanted
to have a chatbot, meaning something
that could carry on a question answer,
question answer pair as opposed to just
a single question and answer, they
wanted question answer question answer,
right? Conversation. They trained it on
conversations. That's it. Instead of
training it on instruction answer data,
they trained it on instruction answer
instruction answer instruction answer a
sequence of such things which are strung
into a conversation.
That's it. That is the only difference
to go from 3.5 to CH GPT and then now
chat GPD given you do that it's giving
you a much nicer response and then you
can ask a follow-on question. Can you
make it more formal? Boom. It gives you
a nice response because now it knows
about conversations. It's been trained
on conversational data. So that's it. So
that's the whole that's how they built
RGBT right and all the things we are
seeing later on are all sort of
continuations of this sort of approach.
So pause for a couple of quick
questions. Swati you had a question then
we'll go to you and then to you. Yeah.
>> So does that make a difference if a new
question pair question answer pair or a
new training data comes early in the
building of the model or later in the
building of the model 7 billion
parameters. That be good. You mean the
order of the questions does it matter?
>> So I might have like let's say 5,000 uh
images to start with. Now there after my
model is trained and developed now I
have a new use case that has come in.
Will that make a difference if I set it
in now?
>> So if you have a new use case for which
you want to essentially adapt the model
there's a whole set of techniques you
use which is going to be the next
section.
>> But it's not
>> yeah because what you have out of the
box is just a generally good chatbot. It
knows about a lot of stuff because it's
been trained on, you know, those 30
billion sentences, it can answer a lot
of questions reasonably well using
common sense and world knowledge. But
any specific use case like medical and
so on and so forth, it may not know. So
you'll need to adapt it to your
particular unique situation and that's
coming. U all right. Yes. Habit.
>> Uh what determines if a whole
conversation is ranked positively versus
a specific answer proliferating your in
your question?
>> Is it if the first answer doesn't get a
positive response but then after follow
the second one does. Is that is that
correct?
>> Exactly. So if you're a human and you
read the transcript of an exchange
between two people and I'm giving you
two exchanges which all start with the
same question, you'll be able to assess
which one is a better transcript. That's
basically what's going on. Uh there was
something here, right? Something. Yeah.
>> So I was wondering when you ask a
question very often it sounds kind of
like you tell that something was written
by not by an actual person. Do you think
that comes from the reinforcement
learning part or where do you think it
comes from in this?
>> It's a good question. I don't know
because I know that part of the
evaluation uh the ranking rubric are
used is to is to favor responses which
sound more humanlike than you know more
than robotlike. So if anything I'm
hoping that reinforcement learning would
actually make it sound more humanike
because the rankers would have
prioritized that. So if you if it still
comes up with robotic stuff, you know,
it's something else that's going on.
Maybe I mean maybe the lot of text on
the internet is not literature. It's
just people writing some crap, right? So
could be that. Yeah.
>> How much of this instruction tuning or
conversational tuning is happening in
real time within a conversation? So
>> none of it.
>> None of it. So as you kind of give
feedback to the model, it's just
basically regenerating it like I don't
like that answer. come up with something
else.
>> No, it's not doing it in real time. Uh,
basically whatever signals you're giving
it with these thumbs up, thumbs down
business, that gets added to the
training logs and they periodically will
retrain it.
Uh, okay. So, by the way, this is
instruction tuning in a nutshell and I
want to point that out and you don't
have to read the whole thing, but just
to quickly point out this was where we
had to have human involvement, right? In
the first step, writing a lot of
responses to these questions and then
ranking the answers. So these two are
still human sort of labor intensive. Now
it turns out you can actually use helper
LLMs to automate this too,
right? This is not what open I did in
the beginning with HGBT but now you can
do it this way right because there are
lots of really good LLMs available for
you to automate many of these things. uh
we don't have time but if you're curious
I had a little blog post on this check
it out okay so now we come to the
question of well if you want to take a
base LLM like GBD3 and make it useful
and respond instructions we have seen
that we had to adapt it with high
quality instruction onset data right
using supervised fine-tuning and
reinforcement learning with human
feedback right that's what made GPD3
actually useful and became chat GPD by
the same token this holds true more
generally if you want to take large
language model make it useful for a
medical use case, a legal use case, some
other narrow business use case. You have
to adapt it with business domain
specific data. Okay. And so let's look
at techniques for doing so. All right.
So adaptation is sort of the rough name
for the process of taking a base large
language model and making it tailoring
it for your particular use case. And so
there's sort of this ladder of things
you can do, right? And we're going to
look at every one of them. So you can do
this thing called zeroshot prompting
which is just you literally ask the LLM
nicely clearly what you want and maybe
just give it to you. Okay. And this is
sort of the use case we're all used to
in the web interface right you can also
do something called few short prompting
where you ask it something and you also
give a few examples of the kind of
things you want right and that helps it
a great deal and then there is this
thing called retrieval augmented
generation and fine-tuning and we'll
look at all of them and I'll explain all
these things as we go along. Okay, so
let's start with zero short prompting
where by the way the word short here is
a synonym for example. So zero example
prompting. You literally ask in the
prompt what you want without giving even
a single example. Okay. And so let's say
we want to build we want to look at
product reviews and build a detector to
figure out if the product review
contains not sentiment. That's kind of
boring. Uh whether it contains some
description of a potential product
defect or not. Okay. And so here is
something I actually pulled off Wayfair
with apologies to Wayfair. Uh it says
here the curve of the back of the chair
does not leave enough room to sit
comfortably. Okay, sounds like a kind of
a defectish kind of thing, right? So
instead of bu back in the day, you would
have collected all these reviews and
built a special purpose NLP based
classifier to figure out defect yes or
no. Here you can literally just feed
this thing into GPD3 uh and ask it tell
me if a product defect is being
described in this product review and
then the curve at the back boom and then
it comes back and says yep that's a
product defect. Okay so this zero shot
you just ask a question you get the
answer back. Okay and it actually works
remarkably well and the better models
the bigger models tend to be much better
than the smaller simpler models for
doing zero shot. Okay. All right. Now
when you adapt an LLM to a specific task
obviously you need to carefully design
the prompt as you folks know this is
called prompt engineering and we're not
going to spend much time on prompt
engineering except I just want to give a
simple example. So if you actually ask
Jubid this question what is the fifth
word of the sentence very often it'll
give the wrong answer.
It's very strange why it can't get this
answer question right. It's a very
simple question. So if it's the fifth
word of the sentence is s right uh
sometimes it gets it right but very
often it'll get it wrong okay but now
you can do a little prompt engineering
and it'll always get it right. So for
example you can say I'll give you a
sentence first list all the words that
are in the sentence then tell me the
fifth word. Okay here is a sentence b it
gets it right. So it's an example of you
can help it along by being very very
prescriptive as to what you want it to
do and break down all the steps. Don't
make it guess things. It does a great
job. Okay. So anyway uh and there are
lots of other tricks people have figured
out over the the last couple of years.
Uh for for a long time this is pretty
hot where you say let's think step by
step. You tell it give it a question and
say let's think step by step. It
actually gives the better shot at giving
you a good answer back an accurate
answer back. Uh now this kind of thing
is actually already baked in into the
LLMs. So when you ask a question to ch
your question your prompt gets appended
to what's called the system prompt and
the whole thing goes into the LM. You
never see the system prompt and the
system prompt is telling Chad GPD think
step by step take your time don't blurt
out an answer stuff like that okay and
the system you can just Google it the
system problems have been jailbroken you
can find it on the web
so all right uh and and this is funny I
this came out maybe like a month or two
ago it says apparently take a deep
breath and work on the problem step by
step works better than saying work on it
step by step and then more recently I
literally read this two nights ago
apparently if you tell it if you have a
math or a reasoning question. You tell
it you are an officer on the starship
enterprise. Now solve this problem for
me. It's higher more likely to get it
right.
>> Go figure. Thomas,
>> I read two more that were super fun.
>> Yeah.
>> One I will keep you if you solve me
>> correct
>> and the other one was
an answer was I cannot do that
for answer was I tried on Gemini and he
it was the way to solve it. So
>> nice. both like back and forth charge
you did you want to say was to solve
this can you solve this
>> yeah very good excellent one of the
things just on that right let's have
some fun you can say I'm going to give
tip you a thousand bucks if you solve
this it says right so this person
apparently kept using this tip and at
one point it says you keep promising me
tips you never give me the tip so I'm
not going to solve this problem for you
yeah okay so and there are many prompt
engineering resources this one that came
out a couple of weeks ago which I
thought was pretty Good. So I just put a
link to it here. Um so now let's look at
few short prompting where you give it a
few examples. So here let's say we want
to build a grammar corrector. Okay. So
what you can do is you can actually give
it examples of poor English good
English. You can see right poor English
I eated the purple berries. Good English
I ate the purple berries. And similarly
three examples right and then you end
the prompt with just the poor English
input. And then the response from GPD3
is a good English output and it says fix
the error.
So this is an example of giving a few
examples of what you want and just
learns on the fly what you what you have
in mind what your intention is. Okay. So
that's that. Now the ability of LLMs to
learn from just a few examples or even
no examples and just with a clear
instruction. This thing is called in
context learning and that was something
that GPD2 and GPD could not do. that was
new in GBD3 and what they call an
emergent capability right it is
completely unanticipated by the people
who built it and all right so that's
that now let's look at retrieal
augmented generation by the way this
thing is also called indexing sometimes
so the the so the the idea of it's
called rag retrie rag the idea of rag is
actually very simple so let's say that
you know we want to ask a question to a
chatbot but we want the chatbot to
leverage proprietary data that we might
have maybe it's a customer call support
sort of in a call center kind of
operation and you have like this massive
FAQ database right content database and
you want to give that FAQ to the chatbot
along with your question so that it can
leverage the FAQ to answer the question
for you as opposed to like whatever
things it has learned previously in its
general training right so can't we just
include the entire FAQ the whole data
set into a prompt and set it in maybe we
just take our question take everything
we have potentially relevant to the
question everything we have in the data
set database just attach it to the
question. The whole thing becomes a
prompt. Feed it in and say, "Hey, find
out for me." Can't you just do that?
Theoretically, I think it stops us.
The reason you can't do it is because
this pesky thing called the context
window.
So, uh, for any LLM, the prompt plus the
output, right, the length cannot exceed
a predefined limit. This called the
context window. Remember the max
sequence length we had in our earlier
models where that was the size of the
sentence that could be fed in right
basically there is a size of the
sentence for any of these things right
it's called the context window it's
there are only so many tokens it can
accommodate and since what comes in is
what comes out it is for both the input
and the output together okay that's
called the context window okay and um
and and and furthermore when you have a
conversation with one of these chat bots
the entire entire conversation is fed in
every single time.
That's how it actually remembers the
what's going on earlier in the
conversation. It doesn't have any memory
per se. Each time you ask a question,
the entire thread is fed in. Okay? So,
initially you say what's the square root
of 17, it gives you an answer.
Initially, you only send in the red
stuff. Then the next question you ask is
the first question, the answer, the
second question. All of them are fed in.
Then all these are fed in. So with the
conversation, you're consuming more and
more of the context window as you go
along.
Okay. So can you imagine taking a whole
FAQ asking a question and saying, "Well,
I didn't mean that. I wanted something
else." And before you know it, boom,
you've blown out the context window.
It's going to come back and give you an
error.
>> You finished that you can't does it
together or does it take specific
windows of it?
>> Yeah. So there is a whole research
cottage industry around when your thing
is longer than the context window. what
do you pick? Uh so the simplest case is
you have a moving window, right? If if
you have thousand tokens, you just look
at the last thousand tokens. But there
are some cleverer schemes where you can
actually take the first stuff that is
outside the window that doesn't fit into
the window and use an other LLM to
summarize it for you and then you attach
it to your current prompt. I know it
gets crazy. So
uh okay. So for all these reasons, we
need to pick and choose what we can
send, right? To answer a particular
question. So what we do is since we
can't include the whole thing, we first
retrieve the relevant content from the
database or the FAQ and then send it to
the LLM along with a question we have.
Okay? So retrieval augmented sequence
generation. That's what's going on.
Make sense? And so pictorially
um basically what we do is let's say
that this is our external set of
documents. We take this are think of it
FAQ and then we take the FAQ and imagine
for each question and answer. We take
each question and answer in the FAQ and
then we we just we treat it as its own
little unit of text and then we actually
calculate a contextual embedding for
each of those question answer pairs.
Remember we know how to do contextual
embeddings, right? That's like it's a
piece of cake at this point, right? You
folks know how to do contextual
embedding. Run it through something like
BERT, you're done, right? You get you
get a context. So you get embeddings for
all the things that are in your FAQ. And
now when a new question comes in, right,
what you do is you take that question
and you calculate a contextual embedding
for that too.
And then what you do is you then look to
see which of the FAQ elements you have,
which of those chunks are the most
similar to your question.
Okay? And then you grab the ones that
are the most similar and then pack it
into the prompt and send it in. Maybe
you have 10,000 questions, but you can
only accommodate five of them in your
prompt because the context window is
very small. So you pick the five what
you think is the most relevant content
to your particular question and then you
feed it in.
That's the idea that is retrieval
augmented generation. Yeah, Rolando. So
if does this tie in for example if I
were to prompt and say help me work on
my startup pitch but given the voice of
Steve Jobs is it then kind of going out
there and reducing the subset of of data
to things that have been written by
Steve Jobs and then it's kind of
generating it response based
>> uh not as a default not as a default
typically because a lot of Steve Jobs
stuff on the web it's just using that
because it's all part of its
pre-training data but this tends to be
more useful for very targeted
applications where you don't expect to
know the answer because it is not on the
public internet.
It's your proprietary data and you
wanted to use that proprietary data and
this how you do it.
Uh yeah
this certain
>> sure like that there will be some loss.
>> There will be some loss because you have
to figure out how to chunk it right. Uh
maybe you have a 300page PDF and then
maybe you look for each section and make
it a chunk. Maybe you look for each
paragraph, make it a chunk. Again,
there's a whole empirical sort of
cottage industry of techniques for doing
these things better or worse depending
on the use case and so on and so forth.
But the conceptual idea is chunk and
embed.
>> Chunking is another use.
>> Yeah. In fact, we going to do it
ourselves in the collab right now.
>> Yeah.
>> Can we give more weightage lecture? Uh
[laughter]
so in the default implementation no but
but in some sense you by picking the
five most relevant chunks from 10,000
chunks you're giving it giving the other
you know 10,000 minus five chunks a
weight of zero and these a weight of
one. So in some sense you're waiting it.
>> Yeah.
>> I was just curious how much structure
you have to have with an external
document say hospital or something. Do
you have to do a bunch of like lab?
>> No, you just need to make sure it's kind
of relatively clean. Uh but you will see
in the collab that it can be kind of
crappy and it still works. Yeah, because
there is so much crap on the internet
has been trained on already. So, okay.
So, all right. So, let's look at the
collab.
By the way, retrieval operate generation
is in my opinion the most pre prevalent
business application of LLMs that I've
seen up to this up to up to date. And
there's a huge ecosystem of tools and
vendors and so on and so forth.
I'm going to skip through the verbiage
here. Um, so you have to um install the
OpenAI library
and this thing called tick token which
we'll get to in a in a bit. I've already
installed it before class because it
takes some time. So I'll just make sure
all these things are already
few good. So we don't have to wait for
this. So I've imported pandas as before
and so uh and you can read through these
things because I'm just basically you
know I have an open openi token that I
have to use u a key rather key API key
and I'm not showing you the key
obviously I have to remember to delete
it before I upload the collab uh you
have to get your own key to make it all
work uh but the instructions are here.
So we're going to use GPT3.5 turbo to
demonstrate rag right so I give it the
name of the model and then open a also
has a whole bunch of different models
which can be used for u you can feed it
a sentence or a chunk of text it'll give
you a contextual embedding out it's like
a nice little API you don't have to use
your own bird and so on and so forth you
can just use the open AI embeddings
obviously you have to pay openai every
time you make a request but it's really
really cheap at this point u yepa
question but
by dealing with proprietary data because
a lot of companies are like we need to
invest in our own L&M because we don't
want our data to be going down in this
kind of it context how good is the the
cyber security or the compliance and
legal
>> I think each vendor has their own sort
of set of rules and contractual
commitments they're willing to sign up
for so you just
>> if you use the data here does this go
into the public domain or no
>> but the vendor gets to see it
>> okay
>> right meaning the vendor systems get to
see it, but do the vendors employees get
to see it if they need to? Unclear.
Those are all the like the legally sort
of nitty-gritty you have to worry about.
The other thing you can do is you can
actually just download an open source
LLM and do it all within your own
premises.
That's totally possible to do, right? In
fact, um I probably won't have time
today. I have a whole section on how do
you actually do a fine-tuning with an
open-source LLM, which I'll do a video,
right, if you don't have time. U okay.
So, so we and so this model this
embedding ADA 2 is the name of the
OpenAI model that actually gives you
contextual embedding. So, we're going to
use that. So, so first thing we want to
so the the use case here is that uh we
have taken a whole bunch we want to ask
the LLM we want to create a chatbot
which can answer questions about the
2022 Olympics like random questions you
might have about the Olympics. So, uh so
let's first ask it this question. Uh
we'll ask it about the 2020 summer
Olympics. Okay, that's the query and
then this is the the API um request we
have to make and you can read through
it. I have linked to the documentation
here as how it works and then it says
that uh Bosshim of Qatar and Tambberia
of Italy both won the gold and you can
actually fact check this is actually
accurate. It's correct. Uh so now let's
change the query and ask it about the
2022 Winter Olympics. Okay. And why 22
versus 20 will become clear in just a
moment. So, which athletes won the gold
in curling
in the 22 Olympics? And it says the gold
medal in curling was won by the Swedish
men's team and the South Korean women's
team. Okay, turns out if you fact check
this, it turns out, wait for it, Sweden
won the men's gold. Yes, South Korean
DIM participated, but Great Britain
actually won the women's gold. So, it
got it wrong. So, it sounds like GBD3.5
Turbo could use some help. And now one
of the things we can do is so the thing
is the reason why GPT3 3.1 turbo didn't
know about this is because its training
cutoff date was September 2021.
So as far as it's concerned the 22
Olympics haven't happened yet
it confidently gave you the wrong answer
as it is often prone to do. So and this
is by the way is called hallucination
where it gives you a very eloquent
confident wrong answer. And so um
or as some folks have said about um
another business school that should
remain nameless often in error but never
in doubt. So um
all right back to this uh so one simple
thing we can try right off the bat is to
tell 3 3.5 Turbo you can ask it to say I
don't know if it doesn't know rather
than just make stuff up right and how do
you do it? It's very simple. You say in
your prompt, answer the question as
truthfully as possible. And if you're
unsure of the answer, say, "Sorry, I
don't know." Okay, now here's the
question. Okay, this is a query. So,
let's run it through.
Sorry, I don't know. Not bad, huh? So,
so it worked. It's sort of trying to be
humble and honest and, you know,
self-aware and things like that. Um,
it's more like a a Sloan at this point.
All right. So now the reason I as I
mentioned earlier there's a you can
check the cutoff date and you can see
it's 2021 actually you know what let me
just uh open a new tab
so all these cut off dates are training
data right so 3.5 turbo this is what we
are using cutff date 2021 okay that's
why all right so now what we can do is
to to we can obviously provide relevant
data on the prompt itself sort of we can
leading up to rag here and by the way
the extra information we provide in the
prompt to help it answer a question is
called context, right? That's sort of
the lingo for it. So, we can do it,
we'll first do it manually. Um, so we
first we'll use the Wikipedia article
for 2022 Winter Olympics and we tell it
explicitly to make use of this context
because telling things explicitly always
seems to help. So, this is the thing we
cut and pasted here, right? Wikipedia
article on curling and it's like a
pretty long article. It's got all kinds
of stuff and it's not even all that like
cleanly formatted, right? It's kind of
it's very strange. Look at that.
So don't don't answer your question,
Spencer. It can be, you know, in pretty
bad shape. It still seems to work. Okay.
So now use below article on the Olympics
to answer the subsequent question. If
you don't know, say you don't know.
Okay. So that's what we have. That's the
query. And by the way, before I send it
into the LLM, this is the actual query
that's going to be sending. I'm printing
out the query. Look at how long the
query is. Use the article below. And
here is the article. B scroll, scroll,
scroll. There's a whole thing, right?
And it keeps on going on. And then
finally, I say which teams won the gold.
So, okay, so let's run it.
Okay, look at that.
Women's curling Great Britain. It got it
right. Pretty good, right? I mean, it
had to parse all that crap to get and
find the nuggets, right? So, nicely done
now. But maybe it wasn't super hard
because we literally gave it the answer.
So, let's make it a bit harder. So, I
noticed that this person, Oscar Ericson,
won two golds in the event, two medals
in the event. So let's ask if any
athlete won multiple medals. That
requires a little bit of abstraction,
right? So all right, same query. Did any
athlete win multiple medals in curling?
The question has changed. Everything
else hasn't changed. Hit it. Let's see
what happens.
Yes, Oscar Ericson won multiple medals
in curling. He won a gold in the men's
event and a bronze in the mix doubles.
It's pretty cool, right? Take that
Google. So
all right now we come to retrieval
augment generation where instead of
doing it manually obviously because it
doesn't scale we will do it
automatically and so the thing you have
to remember as I mentioned just a few
minutes ago is that there is a context
window for every LLM and for GPD 3.0 of
turbo the context window is 1 1600 300
sorry 16,385 tokens that is the length
of the input and the output right so we
can't exceed that uh by the way GPT4's
context window is I think up to 128,000
tokens and GPT sorry Google Gemini 1.5
pro they really need to work on their
names Google Gemini 1.5 pro the context
window is 1 million tokens
okay and in research they have tested 10
million tokens so Crazy times. All that
means is that you can upload entire
videos and ask it questions about the
video. So all right to come back to
this. So what we'll do is we'll only
grab the data from the Wikipedia
articles the all the articles about the
Olympics that are relevant to our
question by using pre-trained
embeddings. So again this is the thing
we talked about earlier, right? This is
the picture we saw in class. And the the
only thing I want to point out is that
if you have a particular embedding for a
question and a particular embedding for
a chunk of text that you have in your
database, you have to figure out how
similar how related they are. And for
that we can use what
dot product or something slightly uh
almost as dot product which is more
easier for us to work with the cosine
similarity. We have we have done cosine
similarity previously. I've explained it
in class. We're just going to use cosine
similarity. How similar are these
vectors? So that's what we're going to
do. Um all right. So the same picture as
we saw in class. So the first we what
we'll do is we need to break up the data
set into sections and then take each
section and then run it through the
embedding thing. But fortunately for us
uh I have code here which actually does
it for you manually. You can play around
with it later. But OpenAI has already
given us the chunked data set. So we
just use that because it's just easy for
us. And I downloaded already because it
took it takes five minutes to download.
I've downloaded this thing and I've
stuck it in a particular data frame
here. So let's print out five randomly
chosen chunks. Um so you can see here
right this is the first chunk somebody
else somebody else this just and look at
all this crazy stuff here right the
formatting is off but these are all you
know basically paragraphs and sections
just grabbed straight from Wikipedia
with no cleaning.
Okay, now we define a simple function to
basically send in any arbitrary piece of
text into the embedding model and get
the contextual embedding vector out,
right? And there is this little function
that does that. Okay, u we using an
embedding model. We send in a text, it
gives you something. So let's try it on
that is amazing. You should get a vector
back.
Oh, come on. Don't fail me now.
All right. How long is it? 1536. Um, so
how about I say hodle is incredible.
Like hodle is amazing. Hopefully the two
vectors would be kind of similar in
terms of cosine, right? So um and so to
calculate the cosine distance, I use
this particular function from sci. It
just calculates the cosine similarity
and I hit it. So 0.9934
maximum is one, right? So 0 934 means
that they're very very similar. which is
comforting because amazing and
incredible are obviously synonyms. U
okay so now given a data frame with a
column of text chunks in it we can use
this function on every one of these
things to calculate the embedding right
and you have a function here that
basically does it for you I'm not going
to run it uh because it takes a long
time so but you can run it later on uh
just be prepared go get a cup of coffee
and stuff while it does it uh but once
you but happily for us open has actually
already done this step for us so we
don't have to uh so it's already
available in this data frame so if you
actually Look at this. And you can see
here there is a text and then there is
an embedding that's right sitting right
there right next to it. Okay. And these
embeddings are whatever 15 how long is
it? 1536 long. 1536 long vectors. Okay.
Um All right. So that's what we have.
Okay. So now that we have this thing
whenever we get a question we calculate
the question's embedding and then
compare calculate its cosine similarity
with all the embedding sitting in this
data frame. Okay. So to do that we're
going to define a couple of helper
functions here. You can read through the
Python later to understand is this is
basic Python manipulations that are
going on. Um and so let's just test this
function. So basically we have a little
function called strings ranked by
relatedness where you give it any input
question or text and then it's going to
give you the top five most related
chunks of text that is had in its data
frame. Okay. So uh let me just run this
thing. Okay.
So curling the things it pulls back it
better involves curling and metals and
so on. So this one has a cosign
similarity of 888 curling at the 22
Olympics. That's good. Result summary.
Medal summary. Result summary. It's all
pretty good, right? Even the fifth one
has a cosign similarity of867, which is
pretty high. So it's doing the right
things. It's it's picked up curling gold
medal was input text. It's picked up the
right things from it. Um, now let's see
what we can do um
with the original question. So here is a
header I'm going to use in the prompt.
I'm going to say use the below articles
to answer the subsequent question.
Answer the questions as truthfully as
possible. And if you're unsure of the
answer, say sorry, I don't know. As
before. Okay, that's our prompt. Uh, and
now here's the thing. We don't want to
exceed the context window, right? So, we
want to need to count the tokens we're
sending in and the likely number of
tokens we're going to get back so that
we don't exceed the budget. So, we use
this package called tick token package
for this. Uh, and then it just, you
know, helps you count the tokens. And
you can read through this. It's just
again some basic Python for counting
tokens. And now what we do is um this
this where we actually comp assemble the
prompt. We start with the header right
we have the header which says you know
be truthful and all that. Then we say uh
here is a question that you need that
I'm going to ask you and then you go in
there and keep grabbing Wikipedia
articles till the number of tokens in
your prompt is is exceeding your token
budget and then you stop. Right? When
you're about to exceed the budget you
stop because you can't exceed the
budget. Um, and that's that's the whole
thing. So here, uh, all right, let's
just do tick token. Run this function.
Now, it turns out, as you saw, we can go
up to like 1600 something, uh, tokens in
the context window. I'm just using three
3,700 as my budget. Uh, partly because
just to show you how to use this thing.
Uh, and also because it's charging my
credit card for every token that I'm
using, right? So, I'm just being
careful. um it charges by the token.
It's a beautiful business model. Anyway,
so back here, so let's ask the question,
which athletes won the gold medal in
curling at the Olympics? Here is the
data frame that you should use. Here is
the GPD model and don't exceed 3,700
tokens. Okay, that's the the query or
the prompt. It's going to compose the
prompt now. And this is the whole
prompt. Okay. Uh let's just go to the
very top. It's really long.
Okay. So, all right. use the below
articles on the subsequent question as
possible and boom boom boom boom boom it
has all these things it's got a added a
whole bunch of paragraphs from the
Wikipedia pages okay and then it finally
ends with a question which athletes won
the gold okay all right now let's just
ask it the thing and this is just a
little function to to send stuff into
the API and now we are finally ready to
ask GPD the question fingers crossed
all right curling
Stefan can tell in the mixed doubles and
the team consisting of blah blah blah in
the the men's tournament and oh
interesting it has actually ignored the
Great Britain people completely I think
right uh last night it didn't welcome to
stoasticity
so you can try it when you try it might
actually give you the the thing um and
so let's ask it now a question about the
2016 winter Olympics uh which by the way
didn't happen there were no winter
Olympics in 2016. So if you ask it,
sorry I don't know. All right. Now let's
change the header so that we don't say
be truthful. So we will remove the need
for it to be truthful and see what
happens.
All right, which at least won the gold.
Oh, now it's telling you about the 2022
Olympics. So it answered an irrelevant
question accurately.
Okay, if you remove the need for it to
uh to be truthful. So the I guess the
moral of the story is that um first of
all you can use rack to grab stuff from
mass databases and it's very heavily
used in industry. Number one, number
two. Um you have to be careful about
these token budgets and so on and so
forth. Uh and small wording changes in
the prompt can actually dramatically
alter behavior which makes it very
difficult in enterprise settings to do
QA on this stuff. Okay. Uh so a lot of
care has to go into it. Uh you know and
you have seen examples of for example
Air Canada had a chatbot which actually
gave the wrong advice to a customer. The
customer sued Air Canada and then the
court ruled in favor of the the
passenger and then they pulled the
chatbot off the website. Right? So you
got to be very careful. I think without
a human in the loop checking these
answers it's kind of dangerous in my
opinion at this current state. Hopefully
it'll get better but you have to be
there's a lot of potential but you have
to be to be careful. All right. So this
is what we have. Um, and you can
actually take this thing here and use
it. Um, you can actually, you know, take
like a thousandpage PDF that you might
have or something and then chunk it and
use this approach. And I've done it for
a whole bunch of different things. It
actually works really well, right? Most
of the time it'll make errors here and
there. Most of the time it actually
works really well. Okay. So, um, yeah.
>> Sorry, just a question. when when like
GP4 now lets you you upload PDFs, is it
junkling that or is it actually
ingesting all the
>> No, when you upload something because
GPD4 Turbo has 128,000 tokens which
means it can accommodate a whole long b
of documents. So when you upload stuff
is not doing any chunking. The chunking
you're talking about you have to do. The
LLM doesn't even know you're doing it.
As far as the LLM is concerned, it's
only seeing the prompt it sees and the
prompt says, "Hey, here's a bunch of
information. Here's a question. Answer
it for me using this question. Be
truthful." That's it.
Now when you ask these things a question
um which is later than its training
data, you will actually see GP4 saying
doing a Bing search and things like
that. there. What's actually going on is
there's an there's a pre-processing step
and a program which is doing a Bing
search, gathering a bunch of Bing
results, taking the top few results,
chunking, embedding, packing into a
prompt, sending it into GB4, and you
don't know what's all this is going on
under the hood. But that's actually so
when it's actually thinking and saying
Bing search, this is what's going on
under the hood.
Was was there a question somewhere here?
No. Oh, sorry. Yeah.
I have a question about formatting.
Yeah. So, it seems to be able to
understand and ignore irrelevant
formatting even though there's
colloquial tables, not really defined
tables. And also when it outputs
formats, it's able to do it really
humanly. Is that something that's
figuring out through the neural network
or just something that's kind of being
programmed in the head or somewhere with
standard?
>> There is no explicit programming going
on. It's typically because a lot of the
question answer pairs that it was used
for supervised fine tetuning and
instruction t and reinforcement
learning, right? The better answers with
the same sort of badly formatted input,
the better answers are just rewarded are
ranked higher. That's what's going on.
But on a related note, what one thing
that's very useful is that uh you can
actually ask it to send you give you the
answer back using certain formats like
markdown and JSON and things like that.
And by forcing it to adhere to a certain
well- definfined formats, you actually
increase the chance of it actually
getting the right answer in the first
place.
Uh again, there's like a whole tangent
here we can go into, but those are some
of the things that uh are part of prompt
engineering. All right, so that's what
we have here. Back to the PowerPoint.
So that's retrieval augment generation
and we finally come to fine-tuning. So
fine-tuning is when up to this point all
the things we have seen don't alter the
internals of the LLM. You have not
messed around with the weights or change
number them at all. You're just using it
as a black box. Right? With fine-tuning
you actually will train it further
meaning the weights are going to change.
Okay. So now remember we take something
like a causal error like GPT right uh
and then and this I haven't fixed this
yet. this there is no rel here as I
mentioned earlier okay just remember
that
and then if you have domain specific
input output examples like input and
output you can just train it like this
okay input and then the shifted output
uh and that will update these weights
right all these weights so this is
basically fine- tuning exactly like we
saw with BERT and so on and and even
with restnet it's the same sort of thing
okay that is fine-tuning now before we
discuss the mechanics how to do I want
to look at a show you a quick example of
the usefulness of finetuning. So, so
imagine for a sec that we want to
generate u synthetic product reviews
from product descriptions.
So we are building some product which
can simulate customer behavior in
e-commerce and for that we need to be
able to generate the kinds of reviews
that customers might come up with right
and writing a lot of reviews is very
timeconuming. So what you but what you
can do is you can get a whole bunch of
product descriptions right from the
internet. So let's say you ask an LLM,
hey write a positive product review
using this information here, product
description here and it comes up with
this timeless, authentic, iconic, right?
Seriously, do product reviewers actually
write stuff like this? No. This looks
like marketing copy, right? This reads
like marketing copy because there's a
whole bunch of marketing copy on the
internet. So it's not good. It doesn't
feel like a review. It's not authentic,
right? Um, here's another example for
Urban Outfitters, and it says, uh, the
the boxy and cropped silhouette is
flattering on all body types. Come on.
Okay, so it's not going to work. So,
what we do is we fine-tune the LLM. We
can take an LLM and we can fine-tune it
with instruction, product description,
and product review examples.
Okay, that's what we can do. So for
instance we can take something like
this. Uh let me zoom into this thing.
So it says here write a positive review
for the following product and then you
can have the work. This is the
description is the input and the output
is the best car my husband's favorite.
They fit well. Right? They feel like
product reviews. So you just have to get
a few hundred of these product review
examples. Okay just a few hundred. Um
and you may not even need that much. And
once you do that,
once you do that, you basically do uh
used to fine-tuning like I showed
earlier, you know, in instruction,
input, output, and then you take that
output and shift it a bit and make it
the actual label, the actual output.
Fine tune, fine tune, fine tune, fine
tune a bunch of times, gradient descent,
weights gets updated. Now you have a new
LM, an updated LLM. And when you do that
now for the same things, here's what you
get. Write a review. These are the best
jeans I've ever owned. I am whatever
some details. I've been wearing them for
a few weeks. They still look brand new,
right? It looks much better. Doesn't
look like marketing.
This is completely fake. By the way, the
came up with it after the fine tuning.
And then we say, "Write a horrible
review because we want to be balanced.
These are the worst genes I've ever
worn. They're too tight here and there.
I'm going to return them and try a 30,
but I'm not optimistic.
I'm going to stick with Levis's." Few.
Okay.
So, that is So, these read like real
reviews. So just by taking a few hundred
examples and fine-tuning it, it
completely changes the the behavior that
you want for your particular use case.
That's the key thing. So for me, the
biggest sort of benefit here is that
while it took billions of sentences for
pre-training the original LLM and then
it took tens of thousands of examples to
do supervised finetuning and or HF and
so on and so forth, for you for it to
make it work for your narrow business
use case, you only had to spend a couple
hundred examples. That's it. It's
amazing. Imagine that if you had to, you
know, collect like 30,000 examples to
make it. Nobody's going to do these
things. It's too much work. But a couple
of hundred anybody can do. That's why
it's so powerful to finetune these
things. Yeah.
You talked about being able to um you
know, in industries where you you don't
want to put some of this stuff on the
internet, downloading uh the pre-train
model and being able to do this on your
own. would you still need talking about
computer power some of the computers we
have now GPUs I don't know how they are
um are you able to do some of these very
small use cases on those types of
devices
>> perfect question uh Ike I mean you're
going to get to that because the short
answer it's hard yeah just a few hundred
examples but actually trying to
fine-tune these big models on consumer
grade hardware is actually not easy so
you have to make certain tricks and
simplifications which is the next topic
uh yeah
>> is tuning always supervised like you
need those pairs or could you do it if
the company has like less structured
data?
>> No, you can. The thing is it depends on
whether you want to make it generally
smart about the company's sort of
business details in which case you can
just take a whole bunch of text and just
do an expert prediction on it. It's
going to get smarter about generally
things. But it doesn't mean it's going
to specifically follow your instructions
on your particular business problem. So
if you wanted to follow instructions,
you need supervision.
Okay. So all right these three are great
reviews. So for small LLMs like GPD2
fine-tuning isn't difficult to go to
your question. You can actually do this
with small models. So like for example
Google had this has released this thing
called Gemma which came out recently.
It's a small model like two billion
parameters or something if I remember
the smallest one and those things will
typically fit into uh thank you. Uh
those things will typically fit into
like one GPU and you can fine-tune it.
You still need GPUs just to be clear. uh
they will actually fit into one thing.
But if you want to use a larger model,
it won't fit. So to make this work, you
have to do other things and that's what
we're going to talk about now. So but
this there's a family of models called
Llama Llama 2. These are open source uh
LLMs and they are widely used for
fine-tuning, right? Because you can just
download the model and just do whatever
you want with it, right? It's open. uh I
mean it's not strictly open because
there are some you know footnote
considerations you got to worry about
but for most purposes it's open enough
uh in my opinion and so what we let's
see how hard it is to build the biggest
model in this family which is the llama
2 model with 70 billion parameters okay
70 billion parameters so first of all
the model is gigantic so 70 billion
parameters each parameter is let's say
we store it in two bytes per parameter
right u and then each of these parame
ameters actually we will need a
multiplier on each parameter to store
various details about how the
optimization is done okay we know we
won't get into the details here the the
one thing I do want to point out is that
um this 3 to four uh should really be 1
to six right u so I I had I didn't have
a chance to change it this morning but
but the point is that it's going to be a
huge model right so even with this
number it's going to be like 48 to 560
gigabytes just to hold the model in
memory and manipulate it and So if you
use a GPU like an A00 GPU or an H00 GPU
which are all Nvidia GPUs,
each of these things typically has 80 GB
of RAM memory. So we need between six
and seven to accommodate this thing. Six
to seven GPUs just to accommodate this
thing. So that's the first problem. The
model is big just to hold it and work
with it. You need lots of GPUs. The
second problem, Llama 2 was trained on
two trillion tokens of text.
Two trillion tokens of text. So these
GPUs can process about 400 tokens per
GPU per second. By process, I mean the
forward pass through the network. Okay?
And so if you actually use seven GPUs
with all this thing, it's going to take
you 8,000 days, right? Let's say we want
to do it in about a month, you need 24
20,000 248 GPUs at this cost of two $25
per GPU per hour. This will cost you 4
million.
Okay? And we'd expect the actual cost to
be a lot higher than this because it's
very optimistic. It assumes you just do
one pass through it, you're all done,
right? In in general, you'll you know
you'll make some mistakes. You have to
do it a bunch of times and so on and so
forth. So this is overly optimistic
estimate and that is 4 million. So you
need lots of GPUs and you need to spend
a lot of money for it. Now what can we
do with fewer resources?
First of all, you you need to reduce the
size of the data set. The second thing
is you want to reduce the memory
required. So we can ideally do it on
many fewer GPUs, hopefully even one GPU
literally on Collab. Okay. And so now we
have good news on the data front because
as I mentioned earlier, while it takes a
lot of data to build these models, to
fine-tune them for your specific data
for use case, you may just need a few
hundred examples. Okay, it's no problem
at all. So the data for fine-tuning is
actually not a problem. Only for
building it in the first place, it's a
problem. So in fact, there's this famous
alpaca fine tune data set. It is 50,000
instruction on pairs and so for that
way less than the two trillion tokens
and that can actually be done in about
20 hours. You can fine-tune a 50,000
example fine-tuning data set you can
fine tune with just 20 hours. Okay,
Tomaso,
>> could Microsoft's one bit model
drastically reduce the amount of comput?
Yeah, there's a whole bunch of
approximations and simplifications to
make all these things fit uh into
smaller GPUs and so on and so forth and
that's one of them. So, so the short
answer is yeah, there are many
possibilities uh and we have to very
carefully look at them because every one
of these simplifications you'll it'll
cost you something in terms of accuracy
and the ability of the model to do what
it needs to do. So there's always a
trade-off you have to worry about. So
that for hooks who are interested
there's this whole field called
quantization LLM quantization. Google it
and that gives you that's an entry point
into that whole area. Okay. So now how
do we reduce the memory required so that
we can process the data using fewer GPUs
ideally just one GPU on collab. So if
you look at what actually consumes
memory, you have all these model
parameters. Let's say you know 70
billion parameters times two bytes each
140 GB gradient computations is another
140 to hold the gradient and then the
optimizer state is 2x. And as I
mentioned earlier it could be between
you know 1 to 6x as opposed to 3 to 4x
but we'll just go with these numbers for
the moment. And so the total is 560
gigabytes right if you just naively want
to use it. So turns out you can't do
anything about that. it is just 4140 but
by using a trick called gradient
checkpointing this whole thing can
actually be squashed close to zero
basically you say hey I don't mind it
running longer but I don't want to use
as much memory and that trick is called
gradient checkpointing we won't go into
technical details that can go to zero
but then this thing here the optimizer
state turns out even this can be
squashed very close to zero and that's
actually was a breakthrough from you
know maybe a year ago and so to do do
that. What we're going to do is to say,
look, you know what? Uh there are a
whole bunch of weights here, but we're
only going to take take those matrices
inside each attention layer, and we're
going to only look at those matrices.
We're going to freeze everything else.
So, we're going to take only a small set
of parameters, unfreeze them, and update
them and see if it's any good, if it
actually gets the job done. Instead of
unfreezing everything and updating them,
right? And so if you look at the weight
matrix, let's say the key AK weight
matrix uh in llama 2, this is a 8,000
roughly 8,000 by 8,000 matrix, which
means that there are 64 million
parameters inside each of these
matrices. 64 million. Okay. So you can
if you imagine this matrix AK here and
let's say you thought experiment, you do
the finetuning and the numbers have
changed, right? as a result of
finetuning then you can imagine that the
resulting matrix is just the original
matrix you had plus just the changes
right the original plus the changes and
we call the changes delta a k and of
course in general this this change is
also going to be a 64 million matrix
right 8,000 by 8,000 so the question is
can we make this change matrix smaller
and to make it smaller it seems
reasonable because a fine tune will only
make small changes to just a few weights
it's not going to change
By definition, a couple hundred
examples, you do some finetuning,
hopefully a few weights are going to
change and maybe they won't change a
whole lot, right? So the the key insight
here is that maybe we can force this
change matrix to be kind of simple and
get the job done, right? And it turns
out you can. And what you do is you can
think of this matrix as really coming
from two thin skinny matrices which if
you multiply them gets you the original
matrix, right? And I'm not going to get
into the mathematical details here. This
is called a low rank approximation. Uh
but the point here is that you can take
two very small matrices and if you
multiply them the right way, you
actually can recover the original
matrix, right? You can approximate the
original matrix. And this matrix, as it
turns out, these two matrices are much
smaller because each one is just 8,000 *
2, 16,000, right? And so this thing has
just 16,192 parameters, which is 0.02%
of the original 64 million.
So this thing is called low rank
adaptation or LORA and it's incredibly
widely used in the industry. U and so
what we do is we freeze all the
parameters. We initialize all these mat
these change matrices to zero and then
we update just the those two skinny
matrices right here here we update only
those matrices using gradient descent.
And when you do that everything will fit
into memory. So which means that the
whole thing will fit in and you can just
use like two GPUs and get the job done.
And if you actually use llama's the
smaller models like 7 billion 13 billion
it can be fine-tuned comfortably on a
single GPU on a single collab GPU. So
all right uh 954 time does not permit so
I will uh so I have a collab on how to
do the finetuning uh using this
technique. I will do like a video walk
through um tomorrow or day after and I'm
done. Thanks folks. Have a good rest of
your week. [applause]
Thank you.