Um, so let's start with a quick review.
Last week we looked at BERT, how BERT
was created, and we learned about this
technique called masking, which is a
kind of self-supervised learning. And
the idea of masking was very simple. We
asked ourselves the question we have
seen ways in which people can take
images and pre-train models like restnet
on a vast you know vast uh body of
images but then for each image somebody
had to go and label them right so for
text we asked the question well what
does it mean to label a piece of text
when we don't actually have a clearly
defined end goal in mind except the
general goal of pre-training things
right and then we said oh well what we
can do is we can actually replace some
some of the words in every sentence with
a what you call like a mask token and
then we just train the network to
recover the blanks to fill in the blanks
right and this technique which is one of
many ways of doing what's called
self-supervised learning is called
masking and we and we described how if
you essentially take all of Wikipedia
and for every sentence you mask it like
this and then train a network to recover
to fill in the blanks the resulting
network becomes really good at doing all
kinds of interesting things and that in
fact the first such network or one of
the first such networks was called BERT
u and in fact in your homework you've
been you've been looking at BERT and so
on and so forth right that's masking now
we're going to switch gears and talk
about a different kind of self-s
supervised learning which is different
from masking which turns out to be
weirdly more interesting and powerful
okay so we are going to look at another
technique and this technique is called
next word prediction so now it is
actually in some some sense a special
case of masking where you're basically
saying take a sentence and instead of
randomly picking a word and and making
it a blank. You're saying, "I'm just
going to take the last word and make it
a blank." Okay? And then you send the
sentence in and then you have the the
machine just fill in the blank on the
last word. Predict the next word. Okay?
And you don't have to use full sentences
for it. You can use parts of sentences
for it. Sentence fragments as well. So
if you take the same sentences before
the mission of the MI loan school, you
can literally divide it into well you
can give the and ask it to predict
mission. If you can give it the mission
and ask it to predict off. You give it
the mission of ask to predict the you
get the idea. So every sentence fragment
you can take and literally just give it
the first few and then predict the next
one. First few next one first few next
one. Okay. So this is next word
prediction. And
so the let's what we're going to do now
is we're going to actually take the
transformer encoder architecture that we
used to build bird in the last class and
we're going to try to use it to solve
next word prediction to build a model
that can do next word prediction. Okay.
So this is what [clears throat] we have.
So what we're going to do is uh if you
take the phrase the cat sat on the mat.
So the phrase was let's say the cat
sat
on
the mat.
So what you might want to do is to say
okay this is the input
output
the cat.
Then maybe you have the cat
then the output is sat.
The cat sat on and so on. Right, you get
the idea. And then finally, we have the
cat sat
the mat. Right, this is basically what
we have all these inputs and outputs.
But we're going to very compactly
express it as if it's just coming in
through as as one sort of data point in
one batch. And that's what we're doing
here. So what we're going to do is we're
going to stack it up like this where we
have the cat sat on the on the left
meaning everything but the last word and
then we're going to take that same
sentence and just shift it to the left
one right so the cat sat on the mat we
cut off the mat right and that becomes
the input then we cut off the first word
and that becomes the output so when you
look at it that way you can see here
right the you will want the to be used
to predict cat you will want the to be
used to predict SAT and so on and so
forth.
Okay, so this is just a little sort of
manipulation so that we don't have to
have you know like dozens of sentences
or sentence examples just for one
starting sentence.
So if you have something like this, what
you can do is you can run it through
positional input embeddings like we have
done before with BERT. Uh then we can
run it through a whole bunch of
transformers, right? It's like a
transformer stack. Then we get these
contextual embeddings. Then we run them
through maybe one or more ReLUs if you
want because it's always a good idea to
stick some ReLUS at the very end. U and
then we basically attach a softmax to
every one of the things that are coming
out. Okay. And then that soft max is
actually going to be a soft max whose
range is the entire vocabulary.
Okay. For now, let's assume that the
vocabulary is just a vocabulary of
words, not tokens. We'll get into tokens
a bit later on in the class. For now,
just assume it's words. And roughly
speaking, let's say there are 50,000
words in our vocabulary. So each of
these soft maxes, and this is exactly
what we did for BERT, by the way. Each
of these soft maxes is like a 50,000 way
soft max.
Okay. But what we're going to do is here
when we look at it this way
since we are fundamentally bothered
about next word prediction as you will
see later on we are actually going to
ignore all these predictions because who
cares? We are only going to look at the
last one to figure out okay what is the
last prediction? What is it? Because the
last prediction is going to be based on
everything that came before it here. So
this is really the next word that's
actually being predicted. All the things
before we don't care so much.
Okay. And all this will become slightly
clearer because you're going to make a
couple of passes through it. Yeah.
>> How do we
>> uh so um the notion of a sentence has
disappeared at this point. What we're
going to do is when we look at how we
tokenize the input for these kinds of
models, we're actually going to take
punctuation into account. So we're going
to take periods into account,
exclamation marks into account and so on
and so forth. And that that'll answer
your question and we'll come back to
that. U okay so this what we have. So um
all right. So just to be clear the
embedding that's coming out of the final
dense layer is passed through its own
softmax with the number of softmax
categories equal to the cap size. Okay.
All right. Um okay. So
first of all, s let's say we train
models a model like this with a lots of
inputs and outputs. Okay, this just
looks like bird, right? It's not that
different except that there's no notion
of a mask.
Do you notice any problems with the way
this thing has been set up? Uh
>> like for some words like the you're
going to have a lot of potential output
pairs that come out of that.
>> True. Which means that if you have a
word like the the next word
>> hard to predict.
>> It's true. So some words may be hard to
predict depending on the last word of
the sentence that was the input. Right.
That's what you're getting at. Yeah. U
concerns.
So I want you Yeah. Uh
>> since you're using contextual
like the output of the first word is
going to have access to the second word
and so it's kind of like cheating.
>> Bingo.
So remember for bingo is a technical
term in deep learning which means great.
So um so if you go to this right as she
points out if you look at the self
attention layer note remember the self
attention layer is the key building
block of the transformer block right and
so in the self attention layer every
word we calculate its contextual
embedding by waiting weighted averaging
of its relationship to all other words
in the sentence. So the last word can
see the first word, the first word can
see the last word and so on and so
forth, right? But when you're doing next
word prediction, this feels problematic
because you're peeking into the future,
right? So
so let's say that you want to predict
the next word. If you look at this
architecture, what it can simply do, it
can simply copy it from the input
because it can see the whole sentence.
So if I tell you, hey, the cat sat on
the mat. If I just gave you the cat sat
on the can you predict the the next word
for me? You'll be like yeah duh it's cat
it's Matt.
The whole thing becomes challenging only
if I say the cat sat on the dash. Now
predict the dash.
So to put it another way let's say that
you want to predict right you have fed
in the first two words and you want to
predict this. This is the right answer
for the prediction. The network should
only use the first two.
However, but because self attention can
see SAT, it can see this next word,
it'll trivially learn to predict the
next word to be SAT,
right? There is no challenge for it.
So, this is the key problem, right? This
is the key problem. We're just using the
transformer as is.
>> What's our loss function here?
>> The loss function in all these things is
actually the same as before, which is
that for every output that's coming out.
So imagine you have just a traditional
classification problem uh in which you
have one output uh let's say dividing
you're classifying things to uh 10
categories like we did with the fashion
mnest right 10 digits so you have 10
outputs right and that goes through a
softmax and then you have 10
probabilities and there we use cross
entropy right so here for every one of
these things we use cross entropy so we
take this output and there's a cross
entropy for just for that plus cross
entropy for that and so on and so forth
So we we minimize still cross entropy
but the sum of all these cross
entropies.
>> And does it get complicated at all by
the fact we have a large vocabulary size
now?
>> I mean it it gets complicated just
because there are more things to worry
about compute and so on and so forth.
But conceptually no difference whether
you have 10 or 50,000 it's the same
thing. It's just that instead of
classifying an input into one of 10
categories you're take the inputs
themselves are as long as the number of
words in your sentence. So each word
that comes into your sentence is being
classified in one of 50,000 ways, right?
So essentially you have as many
classification problems as you have
number of words in a sentence. But at
the end of the day, the loss function is
just a sum of all those things or to be
more precise, the average of all those
things.
Actually, I think I may have a slide
about this which I may have hidden
because I wasn't sure if I would have
time. Uh let's unhide it.
and B I did not agree ahead of time that
we're going to set this up like this.
Okay. So, all right. So, yeah. So, we
still use the cross cross entropy cross
cross entropy loss function. So, each
word that comes in. So, the cross
entropy is actually minus log
probability of the right answer. And you
may recall this from earlier in the
class. So, we just do the same thing for
for cat sat on the everything. And then
we just take the average 1 / 7. Boom.
That's it.
So let's so to go back to this problem.
So this is the issue. The issue is that
we can't allow words to be predicted
knowing the future. They should only
know about the past words. Okay. So what
do we do? Right? We have to make a
change to the transformer to make it
work for next word prediction. So what
we're going to do is when we are
calculating the contextual embedding for
a word, remember the contextual
embedding for a word is going to be a
weighted average of all the other words
embeddings. We will simply give zero
weight to future words.
If you give zero weight to future words,
it's almost as if they don't exist.
Okay? And this will become clear in a
second. So imagine that this is the the
thing we are going to calculate. These
are all for every word in the sentence
we are calculating the uh the pair-wise
attention weight and you will remember I
went through this you know with like an
iPad thing last week we calculate all
the weights. So for example to find the
um so all these weights in every row
will add up to one and so you take the
contextual embeddings of the cat sat on
the multiply them by the respective
weights that add up to one which is the
first row of this table and that gives
you the contextual embedding for the
word the and so on and so forth. And
since we can't look at the future words
all we do is we go take this table and
we just zero everything out in red.
Okay, we just zero everything here out
and then we renormalize so that the
remaining cells the nonzero dot cells
will still add up to one in each row. So
what that means is that if you're
actually only looking at the only this
thing is going to play a role for cat
only this thing is going to play a role.
So let's let's let's give an example. So
um to calculate
to predict uh on you'll only look at the
words for the cat sat.
Okay. The rest of it will not be
considered at all. Now the effect of
doing all this is that by the way this
is called causal self attention. This
tweak is called causal self attention.
Uh is also called masked self attention.
Right? Just different labels for the
same thing. And so what that means is
that when you're looking at the input
for the only the is going to be used to
predict cat.
When you look the cat only these two are
going to be used to predict sat and so
on and so on and so forth.
Okay. So this thing here this so all we
do is we go into a transformer and we
just change each attention head to be a
causal attention head
and the way it's actually done under the
hood is actually very elegant for
computational efficiency purposes but I
won't get into it because it gets a bit
you know involved but the key idea is
replace basic plain vanilla attention
with causal attention aka pay mass
attention
and you do that boom suddenly it it
starts you know working for an expert
prediction it can't cheat anymore
and when we do that we get the
transformer causal encoder
and by the way the word causal here
there's no connection to causality so
it's just a it's just a term
so if you look at the original
transformer paper um
it was created for translation for
machine translation you know English to
German right those kinds of use cases so
it had something called an encoder which
we are very familiar with from last week
and then it had something called a
decoder right and it is called the
encoder decoder architecture and we are
not going to cover the encoder decoder
architecture because we are not covering
machine translation in this class but
I'm mentioning this because the this
part of the the architecture is called a
decoder
because it uses see here there is a
masked attention business going on here
because it is using this masked
attention it's called a decoder so
the transformer causal encoder is also
referred to sometimes as a transformer
decoder but the word decoder has two
meanings
right it's a synonym for the causal
encoder like we have seen today it's
also used to refer to sequencetosequence
translation problems for the second part
of its architecture so you just have
keep it it'll become clear from context
what we're talking about in this course
of course there is no confusion because
we're not going to be looking at
translation right we may say decoder
causal encoder it's the same thing so I
thought there were some transformers
that use birectional
package like is it different from
>> no the um the birectional all all
birectional means is that I can see
everything so the encoder we looked at
last week the the basic self attention
thing is birectional
Basically all it means is I can look at
both in both directions to see what
other words are there in causal. You're
not using the one in the future.
Correct.
All right. So,
so in to summarize where we are. This is
what we looked at last week for BERT and
this is a transformer encoder and we
take the same thing and instead of
multi-head retention we would do causal
multi retention. We get the decoder aka
causal encoder.
Okay. And we use the left for masked
prediction. We use the right for next
word prediction.
All right. So now if you have instead of
having an encoder, if you have a causal
encoder, a TCE here, now we can train
models for expert prediction using the
same exact approach as before,
right? We set up the inputs and the
outputs like I described earlier. We run
it through a bunch of stacks, a stack of
causal encoders, dens, relu, softmax and
so on and so forth, right? Otherwise the
details don't change but the all
important changes go into the attention
layer and make it masked or causal.
Any questions so far?
>> Uh yeah,
this would only apply when we're
training the model, not when we're
validating and testing, right?
Uh so if I if you give me a sentence
after training right the final
prediction is only is the only thing you
care about and by definition the final
prediction will use everything that came
before it. So we are okay.
Was that your question? No, I think the
fact that we're
uh we're zeroing out the weights in the
future words I thought would apply more
when we're training the model and we're
trying to minimize the loss as opposed
to when we're as a chance to the next
set
>> right but the point is when we actually
use them what is the objective like what
do we want to do when we actually use
them for inference once we finish
training our objective is given a
particular string get me the next word
right and to find the next word you can
in fact use everything that came before
it
>> and therefore without any change to this
model it'll just work for your intended
purpose you don't have to go in there
and change it to you don't have to
unmask it for inference because you
don't need to
>> yes
>> uh I have one question is regarding like
when we do the puzzle transformers we
are putting certain weights to zero for
the words which are to be predicted and
then we
>> no word the the words that are in the
future
>> future Yeah.
>> And then we normalize it.
>> Correct.
>> And we have trained a transformer
earlier on the all the words packed all
the words together. So won't there be
difference in weights between both the
things
>> between the two ways of training? The
weights are going to be very different
and they are two different models. Bert
is used for certain things and this kind
of model which is the basis of GPT is
going to be used for other things.
>> We are training it as well like that. I
mean with while putting the by moving
some of the rates to
>> correct correct. So what I'm talking
about here is the what we're trying to
do here is to say let's say that we want
to do next word prediction as the as the
task as a self-supervised learning task
and and we want to train such a model on
a vast amount of text data right well we
can't just use what we did last week
because it's not going to work because
of the fact it can see the future
therefore we make a tweak and then we
build this model now the question
becomes okay what can you do with this
such a model right we have basically
trained two different kinds of models
that the one that can see everything
Bert and that one that can't see the
future which is actually GPT. So what
can you do with it? And we're going to
come to that.
Okay. U all right. So now once you train
such a model u right given any input
sentence um let's say that the sentence
is it was a dark and it was a dark and
right it goes through all these things.
And remember what I said earlier the
fact that it's predicting something
after just seeing it. We don't really
care.
All what we're really curious about is
what is the next thing it's going to
say? And the next thing it's going to
say is going to be is basically going to
be what's coming out of this softmax.
Does it make sense? We don't care about
anything that went before it
because we already have like a half form
sentence and we want to just find the
next thing here. So we only care about
this. We I mean these things will come
out of the of the architecture of the
model, but we don't we throw them out.
We don't even pay any attention to them.
Okay, we only look at what's coming out
in this one here. And what comes out of
the soft max, remember, is a 50,000 way
table of probabilities. That's what a
soft max is, right? It's a whole bunch
of probabilities that add up to one. And
so it's going to and let's say, for
example, that you know you have starting
with oddwark all the way to zebra,
right? Right? And these are the
probabilities.
So it was a dark and you know just for
kicks I put star me as the most highest
probability number but these numbers
will add up to one. We have this table.
Okay. And then what we do is we choose a
token from this table. We get we get to
choose right. There's a whole bunch of
numbers in this table that we we get to
choose a token. the the simplest thing
one can think of is just choose the the
word that is the most likely, right? And
we choose the word that's most likely
here. And we we're going to have a whole
section on how to choose these things
coming up. Okay, for now let's go with
the simple option. We're going to just
choose the one that's most likely 6. And
then we we attach it to the input. So
now the input has become it was a dark
and stormy. We run it through and we
again we only care about the last one
softmax.
Okay,
we do that. We get another table and
this table turns out the table keeps
changing because the softmax is
different for each time you run it
through because the input has changed.
So you get a new table and it turns out
the most likely one is knight. Okay. And
then we attach so night comes out the
other end. We and we attach knight here
and we keep on going right. We can keep
on going maybe till we basically we tell
the model okay generate up to 100 tokens
and stop. It might stop after 100 or you
or it might decide the model may decide
in fact that when it sees a punctuation
like a period or exclamation mark or
something it's going to stop. Okay. And
we have control over this when it stops
and how it stops. But this is this is
sort of the the basic process and you
folks are all very used to it because
you've all been playing with chat GPT
and the like right? So the but the basic
building block is next word prediction
feed it back to the input next word
prediction keep on doing it right you
keep on doing it and suddenly you know
it's writing entire novels for you
um yeah
>> that mean that the longer the initial
input is better you get a better
prediction
>> um it depends on your objective so
fundamentally you have some task you
want the thing to do for you right and
that task may and you need to give it
all the information it can puzzle we
find useful. Yeah. So the long the the
more helpful the input the better. Maybe
that's how I would say it.
Uh yeah.
>> Would this also apply to something like
Google search? Uh or does they also do
next letter prediction too? But would
this just be a deeper
>> Yeah. So the Google autocomplete for
example, I don't know if they actually
use uh this kind of model under the hood
or not. I just don't know. Um these
things tend to be kept tightly under
wraps. uh you know if they were to do if
they were using it you know my guess is
that
they so I don't know if you folks have
seen recently over the last few months
they have there is there is a generative
AI panel that opens up when you do a
Google search that panel I suspect uses
this uh but I don't know if the default
Google autocomplete actually uses it or
not because it's very compute heavy
right so I don't know what they do
um so yeah this is what you do other
questions on this on the mechanics of
Yeah,
>> for our vocabulary list, I'm assuming
it's static.
>> Yeah, correct. Uh, and as you will see
here, it's not really a word vocabulary.
It's a token vocabulary, but yes, it is
static for a given model.
>> And so for I guess I'm assuming for
Google or any other sort of like search
engine that wouldn't necessarily be
static. And so when it comes to I guess
I guess I'll leave it like because the
model would be different
sort of thinking about uh what happens
to like new words and things that are
formed and how does it handle it if the
vocabulary is static. There's a very
elegant solution that's coming up.
Okay. Um
all right. So now in other words we have
learned how to do sequence generation.
We already saw that we can do
classification with BERT. We can do
labeling with BERT B like models which
are trained on mass prediction. And for
generating sequences now we know how to
do it. We just need to use a transformer
cosal encoder.
Okay.
Now
these kind of models, sequence
generation models trained on text
sequences using next word prediction are
called auto reggressive language models
or causal language models. Okay. And of
course the GPD family is perhaps the
most well-known uh example of an auto
reggressive co language model. auto
reggressive because people who have done
econometrics and some regression know
the notion of auto reggression means
that you predict something and then you
you use sort of you know the past
predictions as inputs into the next time
you predict right so this is the notion
of auto reggression you feed you predict
you feed the prediction back get the
next prediction and keep on cycling
through yes
>> so when you you're kind of putting an
input into GPT for example and it has
that um you know it shows you like the
next words as as it's coming. Is that an
indication of it doing this
recalculation that you described here?
>> Correct. That's exactly what's going on.
Uh in fact, if you use the API, there is
the thing called the streaming API where
it'll actually stream each token that's
coming out through the through every
pass and you can actually see everything
very clearly. But when you actually work
with the web interface and you see the
thing almost as if it's typing like a
human, what I've heard from people, I
don't know if this is true, what I've
heard from people is that they can
actually do it much faster. They slow it
down intentionally to give you the
feeling that it's actually coming from a
human.
So it's like a UX trick to slow it down
to make it feel as if someone is
actually typing something on the other
end. So when you're interacting with a
chatbot, for example, sometimes you see
it actually typing like slowly you can
see the bubble and you can see the
typing. It's actually intentionally
slowed down. Uh because you know it's a
bot otherwise, right? So there's a
little bit of UX
creepiness maybe going on. Uh I don't
know to what extent this is 100% true
and how pervasive it is, but folks who
work in the field have told me that this
actually is not uncommon. So
okay, so that's what's going on here.
These are language models and of course
GPD3 is an auto reggressive language
model and the reason why we have an L in
front of the LM because it was trained
on lots of data with lots of parameters
right some someone does this at some
point it's not a small language model
anymore it's a large language model so
yeah so it's LLM nothing more momentous
than that so so as it turns out uh GPT3
uses 96 transformer blocks 96 blocks and
each block has 96 six causal attention
heads.
Okay. And you can see you can read the
GPD3 paper. It gives you all the details
of the architecture. That is interesting
because GPD4 they didn't publish the
architecture from GPD3 after GPD3
everything became closed. So we actually
don't know what the architecture is even
though there's a lot of speculation on
Twitter. So uh but GP3 we know exactly
what happened right 96 blocks each has
96 causal attention heads. Um and then
the data was actually they scraped 30
billion sentences um from a whole bunch
of sources, web text, Wikipedia, a bunch
of book databases. Um and um and then
they basically just took those 30
billion sentences and just trained it
exactly next word. That's it.
Now when they trained GBD3, I think it
cost them a lot of money um because
things were not as we hadn't figured out
how to do as efficiently as we know now.
uh but it was still pretty amazing and
I'll talk about you know what is so
special about GBD3 in just a minute or
two. So, so this is what we have here
and as you folks have seen the notion of
generating text right is very powerful
right uh because we can obviously
generate text but we can also generate
code because code is just text uh we can
generate documentation for code we can
summarize text we can answer questions
we can do chat I mean the list goes on
all the excitement we see around genai
from the time chat GBD came out is
precisely because the simple idea of
text in text out is just so flexible
It's so versatile. It can handle all
sorts of use cases. That's why there's
so much excitement.
Um, by the way, um, if you're really
curious, I would actually recommend
seeing this video where this this guy
Andre Karpathi builds GPT from scratch.
Okay, it's a fantastic video. If you if
you have even like a little bit of
curiosity about how these things are
actually built, I would strongly
recommend checking it out. Um and
there's also a little blog post where
this person you know basically if you
know numpy you can actually create GPD3
GPD using numpy without any using any
frameworks and things like that. So um
I I found it super interesting and
helpful to understand what exactly is
going on. So if you would like to do
this. Okay. So now we're going to talk
about um decoding sampling strategies
which is I said that when we produce uh
when when when we come up with the
softmax for that last token right we
have 50,000 choices. What do we pick
right as it turns out to actually get
really good performance out of uh genai
systems like charge you need to be quite
thoughtful about the how to decode right
how to actually sample from that table.
So we'll talk about that for a bit. So,
so the first of all definition the
process of choosing a token from the
probability distribution from the coming
out of the softmax right I'm sticking
this table right here this is the
softmax right this process of choosing
it is called decoding that's a technical
term for it right we have to we get this
table we have to decode meaning we have
to pick something from this table okay
that's called decoding now
there are two sort of extreme cases of
very highly simple ways to do
The first thing of course is just pick
the one just pick the word with the
highest probability.
This is called greedy decoding.
Okay.
So in this case for example if stommy is
6 the highest probability in this whole
table we just pick stommy. Okay. So that
is the obvious extreme simple case. The
other thing we can do which is also
super simple is that because we have a
probability table here, we can just
reach into the table and sample a word
out of it, right? In proportion to its
probability, which means that if you if
if you have this table and you're
sampling from it, if you sample from it
100 times, 60 times you probably get
Stormy because the probability is 6. But
some small fraction of the time you may
get strange things like oddwark and
zebra and so on and so forth,
right? you're just literally doing
random sampling.
That's a fine way to do it too, right?
There's nothing wrong with that. So
these these are both options. So the key
thing you need to remember is that the
which one you pick and there are some
variations on it which we'll get to in a
moment. What you pick, which way to
decode you pick really depends on what
your task is, what you're trying to use
the the system for, right? The LLM for.
So the the the broad thing to remember
is that if you're working on questions
for which the factual accuracy of the
response is really important
and or you want the output to be
deterministic meaning every time you ask
it a particular question you really want
the same answer back right you can
imagine a customer call support agent
where there two different customers ask
the same question and they get different
answers right you don't want that so you
want determinist IC outputs. So in those
situations, you should use greedy
decoding is a good starting point
because you will get you know you won't
get any random stuff because for any
given input sentence the softmax that
comes out of that table is not going to
change. It's the same table and if
you're always picking the highest number
in the table that's not going to change
either. So guaranteed determinism
and I found that for reasoning questions
and things where you know you're asking
questions, math questions, reasoning
questions, logic questions, you should
really sort of keep it as sort of greedy
as possible in my experience. Okay. Now
there are other situations where random
sampling is actually a better option. If
you're doing creative things, right?
write a poem, write a highQ, write a
screenplay, things like that. You do
want a lot of creativity in which case
you actually randomness is your friend,
right? You get a lot of different
varieties of responses, diversity of
responses, all that is really good. The
price you pay for it is that you lose
determinist determinism. The outputs are
going to be stoastic. They're going to
be random. They're going to vary from
the same question. The answer is going
to vary again and again. But in many
cases, maybe it's okay. You don't care.
Okay, so that's sort of how roughly how
you think about. Other one I want to say
is that the diversity of response also
important because you if you imagine a
chatbot um if you ask questions if the
chatbot always responds in the same
stilted robotic fashion right it kind it
starts to get annoying you want some
variation in the output right because a
human will never give you the same thing
back though I must say that when I
interact with call center agents I think
they're just cutting and pasting from a
text library so it does look kind of
robotic u so maybe we are already kind
of used to this but anyway Okay, so
those are some of the things to keep in
mind. Yeah,
>> if you're using random sampling, do you
end up with a better estimation of the
uncertainty and probability are more
calibrated in the sense that the table
that you end up at the end is the real
probability that you observe from the
words in your corpus.
>> The table doesn't change regardless of
how you sample it. The table is a
starting point for sampling.
The all of all decoding is about what
token from the table you're going to
pull out.
>> Oh, so it doesn't impact the loss
function.
>> No.
>> Yeah. It's all those things are fixed.
You literally get the table and then you
literally can forget how you got the
table and now decoding starts.
>> Is there a reason why would generate a
different answer given the same prompt
if we run it again and again? Because
they are using random sampling.
>> Correct. That's exactly why. And we'll
see I'll see do a demo of it very very
shortly because you can actually
manipulate it. Uh
>> if you do the prediction word by word,
is there a way to make it resilient to
mistakes? Like if you say the night was
dark and hard work, that can mess up the
next word, right?
>> It can totally mess it up.
>> So how does it can it get itself back on
track?
>> It cannot. And so great question. And
we'll look at an example of things going
off the rails in just a second. Yep.
Is this how Bing works where you can
slide between being more creative, more
accurate?
>> Yeah, exactly. So, Bing has creative,
balanced, precise something, right? Uh
they're basically under the hood,
they're manipulating some of the par
we're going to look at some of those
parameters in just a moment. They're
just manipulating it for you. But if you
use the API, you can manipulate it
directly.
Okay. Um All right. So, so here's sort
of the basic thing to remember about
random sampling.
So, our hope is that the, you know, for
any given sentence, we think that there
is probably some set of good answers for
the next word and a whole bunch of bad
answers, right? Intuitively. So, we want
the probability of the good stuff,
right? We we want like a you can imagine
a distribution is going like that. There
is the head of the distribution, the
first few words in the distribution. if
you sort them from high to low
probability and then there's all the
long tale of you know kind of you know
inappropriate not inappropriate
irrelevant words right so our hope is
that the model is so good that for any
given input phrase it it basically
concentrates the output probability in
the softmax to just a few good words and
sort of kind of zeros out everything
else that is the ideal scenario because
in that scenario if you do random
sampling you by definition you'll pick
something from the high quality head of
the distribution and life is good. Okay.
Now, we want random sampling to sample
from the head and not from the tail,
right? That's the key point. And what do
I mean by head and tail? Let's be very
clear.
So, um imagine you have
take the table that we looked at the
softax table which went from whatever
oddwalk to zebra right and let's say we
sort the table based on high to low
probabilities. So maybe what's going to
happen is that star me
is going to have a probability of I
don't know 6 and I think if I remember
right a knight had a probability of.3
and then a there was a whole bunch of
other words
all the way to the 50,000th word right
from highest low probability so this is
what I so this is you can think of this
as like a probability distribution
okay and So basically what we are saying
here is that these this is the head of
the distribution
while this long tail is the tail of the
distribution and we want our system to
grab something from the head and not
from the tail because the head is the
stuff that's actually the relevant
useful good stuff. Okay, that's really
what we're trying to do here. Does it
make sense? Okay. So,
so to come back to this um
and here is like the most important
point to remember about this slide.
While the probability of choosing any
individual word in this long tail is
pretty small. For any one word, it's
pretty small. The probability of
choosing some word from the tail is
high.
Some word from the tail is high. So to
go back to this thing here. Yeah. Uh so
in this particular example
6 +.3 there is a 0.9 probability it's
going to be either stormy or night but
there is a 10% probability it's going to
be one of these words
and who knows what that word might it's
going to be it might be some random
nonsense word right so what that means
is and this goes to um
this goes to point from before if the
LLM happens to sample a token from the
tail which is not good it won't be able
to recover from its mistake it'll just
go off the rails
Which is why every word that gets
generated is really important to get it
right because book it can't recover very
often.
>> Is there a technical way to define the
difference between the head and the
tail? No,
it's sort of like this common thing
people use and the reason why it's not
is because uh it's so problem dependent
as to what like the you know like
basically you're saying that for any
particular problem I think depending on
the question the right number of words
is probably 20 for the same for a
different question maybe it's 40 for a
totally different model for the same
question maybe 10 so because of that
variability we just can't figure it out
okay so um all All right. So, and I'll
show you this how to do this in just a
moment. So, just for kicks, um I went in
to GPD 3.5 U and then I said students at
the MIT Sloan School of Management are
and I said predict the next word. Okay,
so it turns out invited is the most
likely next word followed by given,
expected, required and able. These are
the top five words.
Okay. And the probability is 3% 2% you
see the you know pretty small
probabilities but then the words that
are below it right the remaining
whatever 50,000 odd words are even
lower. Okay. So here the most likely
word is invited. So what I did is I went
in there and said okay let me try again
now with students of this loan school of
management or invited. And now
autocomplete that find me the next
thing. So it comes back with see now
this is my new prompt. student the M
school invited to submit their original
white papers to the annual MIT
something. It seems reasonable. Doesn't
seem bad, right? It seems reasonable.
Okay. Now, let's mess it up a bit. So
now I go in there and I noticed that the
word masters and the word spending were
much lower probability than these top
five words. Right? I just mucked around
till I found these things. So this is
only 0.05%. This is.1%.
So these are clearly in the tail, right?
They're not the most likely. So I said,
what's going to happen if I actually
force it to use masters and then I force
it to use spending? Okay, this is what I
what you get. Students MID school of
management are masters of chaos.
They routinely blow past deadlines
fracture and then I couldn't take it
anymore. I stopped it.
a single word
and then I said students school of
management or spending which is the
other unlikely word the semester
learning life skills so far it looks
promising through knitting socks
I'm not making this stuff up but this is
GP3.5
so yes it will go off the rails you have
to be super careful um and so
so the way we sort of tame random
sampling to make it work for us uh
Do you think that these sentences refers
like the past like the master of chaos
blow past deadline like is something
that it was in the training sense?
>> Yeah. I mean that is the thing is it's
basically doing rough it's doing some
very rough and approximate pattern
matching from all the training data it
was trained on. So it doesn't mean for
example that on on the mit.edu edu
website right on the collection of sites
that actually there were text saying
that yeah MIT Sloan students were doing
all this crazy stuff it's probably more
like a whole bunch of you know u college
university websites probably had some
content like that maybe there was a
bunch of Reddit people posting stuff
like that so you're just doing some
rough pattern matching it's basically
looking the thing is you have to
remember always with large language
models what it's trying to give you it's
giving you a response that is not
implausible
There is no guarantee of correctness.
There's no accuracy. Nothing like that.
It's giving you a probabilistically
plausible response. That's it. Okay.
Now, usies being Sloan, uh we look at
stuff like this and we get offended. So,
we are we are imputing our values onto
its generation, but it doesn't know and
it doesn't care.
So in fact if I when I typed in
something like list all the awards that
professor Ramak Krishna has won it gave
me an amazing list of awards apparently
I won this and I won that I won none of
it is true to which a student said not
yet.
So I had the tea I made a note of that
fine person's name. So [laughter]
>> so yeah so that's what's going on.
Yeah
>> I get the sense like Maybe there's
>> Could you use the microphone, please?
>> I get the sense that maybe there's some
sort of sliding window that's somehow
waning later words more strongly than
earlier words given how far out because
I feel like the context of students at
MIT, right, should have steered in a
certain direction even with the presence
of the word masters. So, is there
something like that happening?
>> No, it is just the thing is think about
the training process, right? In the
training process, uh, we gave it
sentence fragments and we asked it to
predict the next word. Now, clearly the
more you know about the input that's
coming and the longer the input, the
more clues you have to figure out what
the right next prediction is going to
be. Right? If I say the capital uh the
capital of you'll be like, I don't know,
it's got to be a country, I guess, or a
state, but I don't know anything more
than that. But if you if I say the
capital of France is dramatic narrowing
of the cone of uncertainty. So that's
basically what's going on. And in fact
some there's a very beautiful expression
I've heard which is that what what the
LMS do they call it subtractive
sculpting. So what I mean by that is
it's sort of like when you start it's
like this big block of marble and then
every word chips away at the marble and
then when you're done it's kind of
pretty clear it's David inside the
marble. Right? That's sort of what's
going on.
All right. So to come back to this, uh
what can we do? We can there are three
ways in which you can tune random
sampling to make it work for you. The
first way and and the the idea of all
these things is that you have some
probability distribution. We are now
going to sort of manually
focus on the head and then we're going
to kill everything else and only focus
on the head and sample from that head.
Okay, which immediately begs the
question, how will you decide what the
head is? Right? And that was sort of
Alina's question from before. How will
you decide what the head is? So, one way
we do that is to say, you know what, I
know we have 50,000 words in the
vocabulary. I don't care. Each time, I'm
only going to pick the top K words,
right? K could be 10, 20, 30, 40, 50.
This very problem dependent. I'm going
to pick the top 20 words and I'm going
to ignore everything else and only
sample from the top 10 or the top 20.
That's called top K sampling. And so the
way it works is that let's say this is
your whole distribution and I just
stopped at wet instead of going all the
way to 50,000, right? And then you see
and you decide let's say that you want k
to be two. So you just grab the top two
words k equals 2 and then you reormalize
the probability so they add up to one.
So 6 and2 reormalize it becomes 75 and
0.25.
And now just imagine that this is the
new softmax table that you're sampling
from and you grab a number from I'm
sorry a word from here and you're done.
Okay, that's this called top K sampling
very commonly used
but there's it has a small shortcoming
which is that it basically assumes that
this K that you have come up with let's
say 20 every input sentence the right
number of words in the head is 20 which
seems obviously it's not a you know well
supported assumption it's just an
assumption so then the question becomes
can we do better right because what you
really want is you want the words that
you pick to have the bulk of the
probabilities,
right? As much probability as possible.
You don't really care how many words are
inside it as long as together they have
a lot of probability. Which brings us to
something called top p sampling also
called nucleus sampling where instead of
deciding on the number of words we're
going to pick every time, we decide you
know what we're just going to
choose all the words such that the
probability of such words that we have
chosen is at least P.
Sometimes it may be just two words.
Sometimes it may be 20 words. We don't
care. And then we sample from it.
Okay. So here, same thing here. Let's
say you go with P equ= 0.9. So you 6
+2.8 plus.1.9. Boom. We have hit 0.9. We
stop and then we grab these three words
and then we renormalize them to get this
thing and then boom, we sample from it.
So this actually is even more effective
in my opinion because it sort of it
fluctuates. It doesn't hardcode the
number of words you think is important.
Uh was there a question? Yeah.
>> What if like let's say 0.9 ended up like
if foggy was 0.12 will it only take 0.1
from foggy?
>> Yeah. What it does is it so you give it
a give it a 0.9. What it's going to do
is it's going to keep adding words till
it just crosses that number.
>> Yeah. I was thinking, can't you just set
a threshold for the word slap? Don't
pick a word below probability. This top
B, what if was like 0.89
and then the other one is just 0.1. So
you pick two words.
>> Yeah, you can do that. Um and in fact in
what you can do is you can always say I
want to pick a word which is the most
likely word, right? You can do that. But
if you say I want a word um I want only
consider words whose probabilities are
at least something then basically what
you're saying is that I'm just going to
keep on doing and then we draw a line
here right but the problem is you don't
know how many words have crept over your
threshold
right you might for example find that to
to go to your example maybe you said 0.9
as a threshold may maybe there are a
whole bunch of there was a word at 089
that you just missed because you didn't
make the threshold you'll be like oh no
I should have made it 089 so there's No
right answer unfortunately. But these
are exactly the this is exactly the kind
of thinking that brought us these kinds
of ways to tune these things
all sort of you know the foundation here
is that the realization that we cannot
pro sort of a priority decide what the
right number of words is. So we have to
find huristics to try to do do these
things. So in practice people try all
these methods. In fact you can do both.
You can do you can set up so that you
can do top p and top k at the same time.
Basically you're saying grab words uh
till you cross the probability uh or you
cross k whichever is earlier.
Okay. So those are two methods people
use heavily.
The third method is called distribution.
I'm sorry temperature. And the idea of
temperature is that in top K and top P,
it sort of we have to decide on a number
up front K or P and then we just draw
the line and look at the words that pass
the threshold. Temperature is like a
softer way to do the same thing. It it's
a softer way to emphasize the head more
than the tail. So um I think iPad. All
right.
So the idea of temperature is remember
uh when we have this um oops soft max.
So you know oddwark
all the way to zebra
you have all these probabilities right
now remember where did we get these
probabilities these properties came from
a softmax. So what is a softmax? We
basically had you know all these nodes
say 50,000 nodes in some output layer
and these were just numbers let's just
call them a1 through a 50,000
and then we ran it through a softmax
function and what did it do it basically
did e ra to a1 e ra to a2 all the way to
e ra to a let's call it n and then we it
divided it by the sum of all these
things to get the probabilities. So this
number became e ra to a1 divided by the
sum of all the e ra to a
okay so e ra to a divided by e ra to a1
plus e to a2 and so on and so forth. So
this how softmax works. I'm just
refreshing your memory from a few weeks
ago. Okay. Now what temperature does is
that let me just write it a little
easier.
So e ra to a1 plus e ra to a2 is all the
way
and
what it does is it introduces a new
parameter here called temperature which
is that we divide everything here by t.
And the effect of adding this little
knob called temperature here, right, is
very interesting. So let's assume for a
second that t is a very very small
number.
Assume that t is pretty close to zero,
very small number. So if t is close to
zero,
what's going to happen is that since
it's in the denominator here, all these
numbers,
all these numbers are going to become
really big because t is really small.
Right? If if a1 happens to be a positive
number, it's going to become really big.
If a1 is a negative number, it's going
to be a really really small negative
number. Okay? Now in particular, what's
going to happen is the biggest of all
the a numbers, it was already big. Now
it's going to get massive
which means that its probability is
going to dominate everything else
because you're taking a really big
number and doing e ra to that number.
So what's going to happen is that wait
what what did this
okay so if t is close to zero
the biggest a
Uh, hold on.
The word corresponding to the biggest A
will have a probability of one or close
to one.
And since all the probabilities have to
add up to zero, which means that
everything else is going to be zero. So
the biggest A will have a probability of
one. Everything else is going to have
zero. So reducing temperature close to
zero means that the probability
distribution is going to peak at the
biggest word and everything is going to
become zero. So in practice what that
means is that if you look at something
like this if you apply um
temperature here
what's going to happen is that stormiest
thing is going to get something like.999
and everything else right it's going to
get wiped out
right it's going to get really small
it's going to get even smaller and so on
and so forth and so when t is exactly
zero basically what that means is that
this is going to be exactly nine uh one
and everything was going to just get
zero. So when one of them is one and
everything else is zero when you do
sampling from it you're just picking the
the big number right which means it sort
it becomes greedy decoding.
So that is the value of having
temperature as a knob. Conversely, if
you take temperature T and make it
bigger and bigger, right, as opposed to
smaller and smaller, this distribution
is going to become flat. Meaning all the
words are going to have the same
probability.
So a any one of these words becomes
equally likely. So t close to zero, the
biggest biggest word gets picked. T
close to say exceeds one goes to 1.52
any word becomes likely. It becomes
truly random. So that is the effect of
temperature.
And this knob, you can actually tune it.
Um,
all right. So, uh, this is called, uh,
I'm at
platform.openai.com.
It's called the OpenAI playground. And
in this playground, you can actually put
in all the sentences you want. You can
choose the model and then you can it'll
actually tell you what the softmax
output is. Okay, it's very handy. So
this is where I said oh so here are a
few things I want to draw your attention
to. The first one is you see temperature
here the default is one. If you make it
zero it becomes greedy decoding but you
can make it more than one if you want.
It'll give you all kinds of crazy stuff
as you will see in a second. Okay. Um
and then they don't have top K. They
don't have support for top K openai but
they do have support for top P. You can
put P here in this thing. And I'll
ignore these things. You can read the
documentation uh to understand those
things. But you can actually ask it to
show the probabilities. So I'm going to
ask it to show all the probabilities.
I'm also going to tell it um don't go
nuts. Just give me like a few outputs.
Let's just call it 30. Okay. And now I'm
going to enter some sentences for us to
see what's going on. So let's enter the
same sentence as before. students
at the MIT
Sloan
School of Management
or I think that's what we had right so
submit
so okay this is what it's filling out
now you go click on this word you get
all the probabilities
pretty cool right so you can see invited
given expected these are all some of the
things we had u and so what you can do
is you can go in and say here clearly uh
aching. What is that?
That's very weird. So I'm going to again
I'm just going to check to make sure
that I use the same sentence as before.
It's very brittle. Students MD school
management are okay. Uh are
oh I know what it is.
Okay.
Okay. So, let's try that again.
Okay. So, invited 3.18. That's what we
had, right? Invited 3.19. 3.8. Okay.
Close enough. So, this is what we have.
And now, if you wanted to force it to
choose invited here, you just go in
there and make the temperature zero.
Temperature zero means it's always going
to pick the best one. Greedy recording.
So, you can hit it again.
And it better give you invited. See it
has given you invited.
So that's how you manipulate it using
temperature. Um you can also ask it you
can also manipulate top P. You can do
all these things right but so it's a
it's a people actually use it very
heavily for debugging right and for when
they're playing with a bunch of data
with a model for that particular use
case. You just play with it to get a
sense for what kinds of probability
distributions you see and then you can
fine-tune it using that using that
knowledge. Um so yeah check this out.
Oh, uh, I I said that if the temperature
goes above one to a higher number, every
word in the 50,000 becomes sort of
equally likely, which means it's going
to produce garbage, right? So, let's
actually see garbage production in
action.
So, all right, let's just nuke this.
Okay, and I'm going to take the
temperature and max it. I'm going to
call it two. Okay, which means that
literally anything is possible.
Submit.
Ladies and gentlemen, I present to you a
modern large language model.
Isn't it like shocking
>> because when we work with these language
models we have, we always when we see it
doing some smart things, we always
ascribe some level of, you know,
interesting abilities and intelligence
and so on and then you realize all I had
to go in go in there and change one
parameter and it's garbage.
So you can see the amount of garbage
right it's showing just by twiddling one
parameter. So you have to be in
production use cases when you're
building applications on top of these
large language models you got to be very
very careful with these parameters. So
pay attention. All right. So um what did
I have next?
Okay. So that brings us to the uh sort
of the end of the decoding section.
Oh, see now I'm going to switch gears
and talk about tokenization, right?
which is that um when so far in all the
the the things we have done including
the homeworks and so on we looked at
this tokenization the standard process
right for taking a bunch of text and
vectorizing it which was the stie
process standardize tokenize um index
right and then encode and the
standardization I had mentioned earlier
uh strips out punctuation lower cases
everything uh sometimes removes stop
words like a and the things like that it
also does these things called stemming
But turns out if you actually work with
uh something like GPT, you know that
it hasn't stripped out punctuation. The
punctuation is really good, right? It
uses case, uppercase, and lower case.
And in fact, even better, you can
actually make up a word as part of your
question and it'll use the word
consistently in the output. So just for
fun,
um I made up a word.
I just did this yesterday, a day before.
I said, here's a new word and it
definition. The word is relo
backwards.
I said the definition a student who
understands deep learning backwards
please use his word in a sentence. And
here is a sentence it's coming up with.
Um
I was like a little shocked during the
advanced neural network seminar. It
became evident that Jane was a true relo
effortlessly explaining even the most
complex deep learning concepts in
reverse order.
Okay. So it clearly knows how to use
anything you may make up with. Right? So
it has the ability to compose things
from scratch as opposed to just looking
up stuff. So where is the thing coming
from? Right? That's the question. And
the answer is this very beautiful thing
called bite pair encoding which we'll
look at next.
So all right. So what here um when we
look at this process the adv
disadvantages are some of the things we
have discussed which is that we want to
be able to preserve punctuation. We want
to be able to preserve case. We want to
be able to handle new words and so on
and so forth. So uh the new like the the
sort of the modern models like BERT and
so on they use different tokenization
schemes. They don't actually do the STIE
thing and the GPD family uses bite pair
encoding BPE. Uh BERT uses something
called wordpiece. All of these ways of
encoding, the fundamental idea is to
say, well, you know what? Why don't
whatever language you're working with,
why don't we start first of all with all
the individual characters? Because if
you could actually work with individual
characters, you can clearly compose any
word that comes up, right? Reo is just R
E L D O H, right? Six tokens. If you're
working with characters at the character
level, but working only with characters
is not great, right? because that means
that the model you're giving it no
information about the world. It has to
learn every word from scratch, what the
word means and so on and so forth. So we
it would be nice if we can actually give
it words as well. But we don't we don't
want to give it infrequent words because
infrequent words by definition are not
worth adding to your vocabulary. We're
just going to you know take up another
embedding vector and things like that.
For infrequent words, we'll just make
we'll just compose them. we'll we'll
actually construct them on the fly
because we can always use characters.
Okay, so we don't want to put every word
in there. We only want to put frequent
words. But to give this thing the
ability to compose new words and not
always have to go to characters, we will
give it parts of words. These are called
subwords. So the key idea is that let's
come up with a way to build a vocabulary
which has characters full words that are
frequent enough to be worth adding and
subwords or word fragments that occur
frequently enough to be worth adding. So
for example the word standardize
right normalize standardize and so on
and so forth. I is going to show up a
lot in many places. So you don't want to
have standardize and normalize and so
on. You just want to have eyes. you can
just attach it to all kinds of words,
right? And make it all work, right? So
that's the basic idea of all these
tokenization schemes. And BP is one such
way to figure out how to actually
construct this vocabulary from a
training corpus, right? And by the way,
when I say characters, this will include
not just you know uppercase lowerase
alphabets and numbers, it may it will
also include punctuation.
So that all these things just become
atomic units.
All right. So uh so what we're going to
the way BP works is that uh we're going
to uh start with each character as a
token and I'll talk about the rest of
the thing on the page in just a moment.
Don't worry about it. We'll start with
each character as a token. So let's say
that your training corpus is just a
single sentence. The cat sat on the mat.
Okay. And even though GPT does not
actually do any lower casing, it'll just
actually use like TH uppercase is
different than TH lowerase. Uh just for
simplicity, I'm just going to
standardize it here. So it just becomes
a cat sat on the mat. And then I'm going
to write it in this form where I
basically put a comma after every word
and then I put a little underscore to
show the space between the words. Okay,
I'm going to write it in this format.
And it'll become clear why I'm writing
it in just a second. Okay. Now my
starting vocabulary is just all the
individual letters in the training
corpus. So the starting is just whatever
all these letters. Okay, that's it. And
this is a starting point. And now what
we do and this is the key step.
We merge tokens that most frequently
occur right next to each other. So if
two characters or two tokens are
occurring right next to each other a
lot, let's just merge them because they
seem to be occurring a lot together,
right? May as well merge them. And so
here, for example, I've I've listed the
frequency of the adjacent token. So for
example, if you look at th
shows up right after each other here, it
also shows up here. So therefore, it
shows up twice.
Now H E again is showing up here. It's
also showing up here. So that also shows
up twice. CA on the other hand is only
showing up here. It's not showing up
anywhere else. So it shows up once. A
shows up three times in Matt, SAT, and
in CAT and so on and so forth. You get
the idea. So you're just looking at
pair-wise adjacent tokens. And you pick
the most frequent one that's showing up,
which in this case happens to be a t.
And then you take a and t and you merge
them. So it becomes 80.
Okay. So when you do that when you when
you you merge them and then you add that
new token that you've just literally
created to your vocabulary list and then
you update the corpus to reflect the
merge you've just did. So now the corpus
becomes the cat sat on the mat. But in
this case there is no a and t
separately. There is just the at combo
com combo token here.
Are we good with this step so far?
take the most frequent things and merge
them.
It's a way to compress the data. In
fact, the algorithm came from someone
trying to figure out a way to compress
data.
You know,
think of it this way, right? Suppose I
tell you uh I'm I want you to compress a
message I'm going to send to you and
then you look at all the past messages
you've had to deal with and it turns out
you're finding that u certain characters
are occurring next to each other all the
time right maybe just for argument let's
say ABC shows up ridiculously often in
the messaging and then you'll be like
you know what's if it's always showing
up all the time together why treat it as
three things let me just call it one
thing ABC that's it you send a single
token called ABC every time you send
need ABC not a B C that's the basic
idea. So here if you come here that's
what we have and then what we do is now
we do again this calculation of
adjacency tokens on this updated corpus
and you can see here th shows up once TH
shows up here twice so you get two every
H shows up twice everything else shows
up once and yeah when many things are
showing up with equal frequency just
pick one randomly from this. So we pick
up th right and we merge that which
means that we add th to our vocabulary
and once we do that we update the corpus
and now we have th is now one thing
fused together along with the previous
thing 80 that had been fused together
that is a corpus after the second merge
and then we do the same thing we find
the frequency adjacent tokens turns out
th and e are showing up twice everything
else is showing up once so we take th
merge it to get the boom the and now we
have the cat sat on the mat. So this
process continues
till we reach a predefined limit for our
vocabulary. Now as it turns out when
they built GPT2 and GPT let me just see
I think I did some digging around on
this thing. Yeah. So GPT2 and 3 they set
the vocabulary size to be roughly
50,000. So it basically kept on doing
this till it hit a limit of 50,000 then
it stopped. GPD4 on the other hand
actually went goes all the way to
100,000 vocabulary size.
Okay, so this is BP in action. U and so
what's going to happen is once you
finish all this thing and you have
vocabulary and you have all these things
that you have merged when a new piece of
text comes in right the merges remember
here we merged a to get a this th became
this and so on. When a new piece of text
arrives the tokenization apply the
merges in the exact same order. So if
the new text that comes in is the rat,
it's first going to apply the 80 to 80
to become fuse this here and then going
to fuse th to get this and then it's
going to fuse th and e to get that. And
the final list of tokens that goes in to
your model is going to be the token for
the the token for space and the token
for r and the token for at.
So let's see this in action.
uh GP I mean OpenAI has a has its own
thing but I found this uh site to be
really good. So let's uh tokenize
hands-on
deep learning.
So you can see here
look at this.
So H uppercase H is its own token. It's
token number 39
and
it's it own token. dash is its own token
on is its own token and then space deep
is its token and space learning is its
token okay note one thing suppose you
had said
let's just say you just had deep deep
deep learning
deep has a different token than space
deep
okay what they have realized is that
most words are actually going to show up
after the space after a space right much
more likely so having a space attached
to the beginning of the word saves you a
lot of sort of you know saves you a lot
of compute and so on and so forth
because they will in fact arrive almost
all the time with the space before it
right that's why they have attached the
space to the word itself um and note
that deep learning deep and uh deep
actually let's call it this way
so deep and deep are different
right there is deep there is so clearly
it's taking case into account then I put
an exclamation here. Boom. That and so
ultimately what goes in when you have
have a phrase like um
sat on the mat.
So the cat sat on the mat. And you can
see here uppercase the um and then
let's just do another thing here.
So uppercase the with a space is 383.
lowerase the is 262. Uh and then that's
distinct from just the without any
space. That's a different thing. So
these are all the tokens. Now um let's
try something.
Let's try
Jane.
So Jane is one token which is great and
is another token. Let's see. Rama. Ah
darn. My name wasn't worthy enough to be
its own token. Okay. But strangely
enough
this I was very surprised by this. So if
I put Rama in lower case is its own
token.
I have no idea what they were scraping
which websites. Uh and if I put Jane
here
now J has become its token with space
and A has become different.
So the tokenization is like very it's a
very interesting thing and it works in
very interesting ways. But that's the
basic idea of what's going on under the
hood. I would encourage you to like
check out your names to see if it's
actually been tokenized. So all right,
I'm done. Thanks folks. I'll see you on
Wednesday.