Okay. Uh, all right. So, we'll continue
with transformers today. Part two. Uh,
we're going to do the second pass. Uh,
this is going to be a deeper pass
through the transformer stack. Um and I
think maybe the next 30 minutes it's
potentially the most demanding 30
minutes of the entire course. Okay, with
that motivational speech, let's get
going. Okay, so quick review. Why do we
want transformers? Because we want u we
want an architecture that can generate
output that has the same length as the
input. Same length. Oh, there it is. Uh
number two, we want to take the context
into account and we want to take the
order into account. And as you saw last
time, the transformer architecture
delivers on those three requirements.
And so uh just a quick review, if you
have a phrase like the train liftation,
we have all these little arrows which
stand for the the standalone or
uncontextual embeddings. Uh and then
sometimes this works. So I'm going to
put it close to me here.
Okay.
All right. So um so if you here if you
we start with either standalone
embeddings i.e. the contextual
embeddings uh which have been
pre-trained or random doesn't really
matter. If you look at the collab we did
uh the other day we actually just start
with random weights for the embeddings
and then we add positional embeddings to
them. And so you know each embedding
each word here we take it standalone we
take its positional embedding we just
literally just add them up element by
element then we get a total embedding
and that's called the positional
embedding of each word. Okay. And then
uh that's what we have position input
embeddings. So this whole thing goes
into this transformer encoder stack and
what pops out the other end is
contextual embeddings. Okay. So that's
the overall flow. Now
we applied this uh the transformer stack
to the word to slot classification
problem where we basically took every
incoming natural language query that
comes in. We calculate its positional
embeddings and then we run it through
the transformer stack. uh and then we
get contextual embeddings and then at
this point uh since each word that comes
out each embedding that comes out needs
to be classified into one of 125
possibilities we run it through a ReLU
and then we and when we attach a softmax
to each embedding right this is
basically what we did last class
um so this is the transformer encoder
okay now actually
any questions on this before I continue
I was wondering why when how do you
decide where to add more self attention
and where to add transformer layers? You
mentioned that chart has 96 of them.
>> Yeah. So right so GPD3 has 90 96
transformer blocks. Each one is a block.
Um, so I think the question goes to do
you add more attention heads within a
single block or do you add lots of
blocks? And both are good things to do.
Um, what increasing the number of
attention heads in a block does for you,
it allows you to pick up more patterns
at that level of abstraction.
But if you add more blocks, much like
later convolutional filters can build on
earlier convolutional filters, you're
going up the levels of abstraction. So
to go to vision for instance you have
the notion of lines and so on in the
beginning and then you have a notion of
edges which are two lines then you have
you know nose eyes face and so on and so
forth. So both are worth doing. So
typically that's what you you typically
find that people typically have you know
maybe a dozen heads or you know five six
a dozen heads. We'll see examples of how
many heads in a couple of architectures
later on today. And you can the more you
go up the more uh more capable the model
becomes. as long as you have enough data
to train it well. So the perennial
question of do we have enough data to
train this large model because if you
don't have enough data we might run into
overfitting problems and so on. That's
always the trade-off.
So okay so here I just want to quickly
switch to the collab because we didn't
get have a chance to finish it. I'm not
going to run it because it's going to
take some time. So where we left off
last time.
Okay. So here we we basically took this
architecture that we just saw on the
slide and then we essentially wrote it
as a keras model and I went through this
model in the last class so I'm not going
to go through it all over again. What we
did not do last class was to actually
run it. Um and so uh so if you actually
run it right you can just run it for 10
epochs just like we normally do. Give it
data give it a bunch of epochs choose a
particular batch size. I just
arbitrarily chose 64. You run it for 10
epochs and then you evaluate it on the
test set. You get a 99% accuracy on this
problem. One transformer stack. That's
it. One one block rather. One block.
That's it. And uh of course here there's
a little trickiness going on here
because a naive model can literally say
every word that comes in is other. O.
And since the O's are the majority of
the words, it's not going to do badly,
right? It's like having a classification
problem in which one class is very
predominant. So the naive way to
actually do well is to just say every
time something comes in, oh it's that
majority class. The same thing happens.
But if you then adjust for that, it
turns out that the accuracy on the nono
slots, which is really what you care
about, is actually 93%.
Which is actually pretty good. Okay. Uh
and then I had some examples of, you
know, lots of fun queries you can do,
including queries where I try to break
stuff like cheapest flight to fly from
MIT to Mars and see what happens, you
know, things like that. So have fun with
it. Okay. Um, all right, back to
PowerPoint.
So, this is what we had. Now, what we're
going to do in today's class, we are
actually going to take the encoder we
built last time and introduce three new
complications into it. And when we
finish introducing these three
complications, we will actually have the
actual transformer that was invented in
the 2017 paper. Okay. All right. Um, the
first tweak is the hardest tweak. So
we'll slowly work our way to it. U so
the thing to remember is let's review
self attention. What is self attention?
You have a bunch of words and we further
said that for any particular word like
station we want to take its positional
embedding and then make it contextual.
And the way we do that is by taking each
word's embedding and then calculating
these dot productducts between all the
other words. And then since these dot
products can be positive or negative we
want to make them all positive and
normalize them so that they nicely add
up to one. So we then exponentiate them
and then divide with the total, right?
Which is basically soft max. And when
you do that, you have nice fractions
that add up to one. And then we said,
well, the contextual embedding for W6 is
just all these weights S1, S2 all the
way to S6 multiplied by the original W's
and then you get the context for W6. So
this is the basic logic we covered last
time. Now it is obviously the case that
we explained it only for one word but we
have to do the same exact operation for
every one of the other words too so that
we could calculate W5 hat, W4 hat, W3
hat and so on and so forth right so
there's a lot of computations that are
going on and they all look kind of
similar where you got to do a bunch of
dot products you got to like you know do
some soft maxing on it and stuff like
that so the natural question is is there
a way to organize it very efficiently
And the short answer is yes. In fact, if
you could not do that, there wouldn't be
any transformer revolution. Okay,
because there is that ability to package
it up into a very interesting and
efficient operation that allows you to
put the whole thing on GPUs.
Okay, so now I'm going to switch to iPad
uh and give you some iPad scribblings of
mine which were concocted last night
because I was very unhappy with the
slides that follow. So, we're going to
do iPad. Okay. U All right. So if it
works, you folks are lucky. If it
doesn't work, last year's huddle class
is luckier.
So let's shift to that.
All right. So we're going to go here.
So let's assume we have a simple thing
like uh oops.
Okay, instead of you know train left the
station which is a long sentence, let's
just say you have a simple sentence like
I love hodddle. Okay, and so I love
hodddle is what you have and then you
have these standalone embeddings W1 W2
W3. Okay, so it comes into the self
attention layer and let's assume that
these W1's, W2, W3, they're already
positionally encoded, right? We have
already added up the position encoding,
all that stuff also. It's all behind us.
That all happens outside the
transformer. So you you you get it here.
Now what you do is you actually make
three copies of this thing.
Okay? And let's call this whole thing as
just X. Okay? I'm just giving it the
name X. It's a matrix of these three
vectors. And so the first copy goes up
here, the second copy goes straight, and
the third copy goes down. And don't
worry about the third copy just yet. So
if you look at the the first two copies,
here is the key thing to focus on. Okay,
this whole thing here. Remember that we
want to calculate dotproducts between
all these vectors. And basically we want
to calculate the dot product of every
pair of vectors, every pair of words.
The whole point of self attention is
that every pair of words we figure out
how attracted or related they are.
Right? Which means that we have to
calculate all pairs of dot products. And
so what you do is you take this vector
right there W1 WW3. You take this other
copy that went up. Okay? And then you
transpose it. So when you transpose it,
it all becomes nice and vertical like
that.
Right? All the vectors come in came like
this. When you transfer, it becomes
vertical. And now what you do is you
take each one you take W1 and then you
multiply it by W1. Here you take W1 W2
W1 W3. You calculate all those dot
products like that. And when you do that
you have these nice cells where every
pair of words their dot products have
been calculated in this grid. Okay. And
the key thing to see here and folks with
a matrix algebra background will see
this immediately. All we are doing is we
are taking this x which is the matrix
that came in
and then xrpose which is the matrix that
we went sent up and then brought back
down. We are basically doing a matrix
multiplication of x * xrpose. That's all
we doing. And when we do that we're
getting this nice uh grid of where in
which every pair of words their dot
products have been calculated for you
with one matrix multiplication. Boom.
Done. Okay. Okay, so if you have three
words, there are nine multiplications,
right? So if you have a million words,
that's a lot of multiplications, right?
One trillion multiplications on the
order of all trillion. And the reason to
say order is because you know W1 * W3 is
the same as W3 * W1. So there's some
duplication here. So you get this grid,
okay, in one shot is one multi
multiplication. And then we because each
of these numbers is just a dot product
which can be negative or positive, we
need to softmax it.
And so what we do is we take all these
numbers and we put it into a softmax
function where for each row it
calculates a soft max. And what do I
mean by that? It takes each number here
does e raised to the top e ra to the
number. It does it for each of these
numbers and then divides by the sum of
those numbers for each row. And when you
do that okay you can think of this
operation as soft max applied to x *
xrpose you get this nice little table of
numbers.
This table of numbers basically says
that for the first word right W1 for the
first word take 0.1 of the of the first
one 7 of the second.3 of the 2 of the
third and add them up. We do a weighted
average. So we have this table here. We
have now the third copy shows up here.
Okay is right there. So we do this times
that which is just a matrix
multiplication again. And when we do
that we get the final contextual
embeddings. So this for example is just
0.1 * w12
* w2
point sorry 7 * w2 and then2 * w3 right
there. And you can see the same logic
here as well. Okay. And you can read it
later on. I will post this thing uh to
make sure you understand exactly how it
flowed. But the larger point I want you
to focus on is that the entire sol self
attention operation we just looked at
here basically is this this beautifully
little compact matrix formula.
Okay X comes in you do XRpose you do a
matrix multiplication you do a softmax
on top of it and then multiply by X
again and boom you're done.
So that is the magic of taking the
transformer stack and representing it
using matrix operations because then
lightning fast on GPUs.
Okay. All right.
That was the warm-up.
Now let's crank it up a notch.
So recall that in the last class um I
talked about the fact
the self attention operation the W's are
coming in and we're doing all this stuff
with the W's right and then we're
getting some W hats out but there are no
parameters
there's nothing to be learned inside the
transformer self attention layer right
there are no there are no weights there
are no biases there are no coefficients
so well okay What are we learning then?
Right? So what we now do is we going to
make the self attention layer tunable.
We're going to inject some weights into
it so that when we train it on an actual
system, it'll the weights will keep
changing to adapt itself to the
particularities of whatever problem
you're working on. Right? So that takes
us to the tunable self attention layer.
Okay? Tunable self attention layer. So
this is the key thing to keep in mind. U
any questions on this before I continue
with the tunability thing.
Okay.
Is this picture working out by the way?
Okay.
Uh all right.
So what we now do is we have the same
exact logic as before where we have this
thing that comes in. Okay. We have this
input that comes in the same we call it
X again. this whole this matrix of
embeddings and then before we just send
three copies instead of doing that what
we're going to do is we'll take each
copy X and then we will actually
multiply it by a matrix
okay this matrix is called the key
matrix
okay and this matrix this matrix of
numbers are weights that will be learned
by Brack prop
so basically what we're saying is that
when this thing comes in let's see if
there's a way to transform this X into
some other set of embeddings which may
be useful for your task. We don't know
if they're going to be useful, but
surely giving it a bit more ability to
have weights which can be learned means
that it giving it more expressive power,
more modeling capacity. And whether it
actually uses the capacity will depend
on how much data you have and how well
you train it. And maybe if it's not
useful, it won't use it. In what I mean
is if transforming X actually doesn't
really help at all, then this matrix A
is going to be what?
it's going to be the identity matrix
because you take basically one and
multiply by X you'll get one X again. So
in the worst case maybe it just says I
have nothing to learn here but maybe
there is something you can learn. So so
that's what we do. So we multiplied by
this matrix A K and then we come up with
the same you know some embeddings
transformed embeddings and we call these
things K
okay K. Now this KQV as you will see has
its origins in the in this field of
information retrieval but I personally
find that that interpretation is not
super helpful because transformers are
used for lots of applications outside
information retrieval. So I'm not going
to go with that kind of interpretation.
I'm going to go with interpretation of
let's make each of these things tunable.
Okay. And tunability means we need to
give it weights. All right. So that's
what we have here. Now the second copy
we did this with the first copy. Well,
let's do the same thing with the second
copy. We'll take the second copy and
multiply it by some other matrix called
AQ.
And when we are done with that, we get
these embeddings. And we will call these
embeddings as Q.
Okay. Now, just like before, we will
take this this thing here and we'll
transpose it.
So, it all becomes nice and vertical
like that. And then we'll do exactly the
same as before. We'll calculate all
these pair-wise dot productducts using
one one shot one matrix multiplication.
And because we are calling this Q and we
are calling this whole thing as K. This
thing just becomes Q * KT.
Okay. At the end of it you come up with
a grid of numbers just like before.
Okay. And these numbers could be
negative or positive. So we need to do
the softmax on them to make sure they
are well behaved fractions that add up
to one. So we take this Q KT business
and then we do we just run a we put it
through a softmax function for each row
and when we do that we we'll get
basically the the like a table like the
ones we saw before by the way the
numbers here are the same just because I
duplicated it because I'm lazy in
reality given it has gone through all
these transformations the numbers are
not going to be the same right uh you
have these numbers and then you take the
final copy which is x * av Right? Each
copy is getting multiplied by its own
matrix. Right? And this copy is being
multiplied by AV. And let's call this X
A. Okay? Which is here as just V.
And so what you have here is this soft
max QT * V is exactly the same kind of
dot product as we saw before matrix
multiplication. So we have these
contextual embeddings and that's what's
coming out of the of the transformer
block. So now the whole thing we did
here the whole thing can be represented
as soft max of Q KT * V. Okay. So if we
zoom in a bit. Come on. Okay.
Okay.
So X came in.
Three tracks went here. The first track
X * A K X * AQ X * A V. And this thing
is called K. This thing is called Q.
This thing is called V. And then we do
the same transpose as before. We do the
dotproduct thing to calculate the
pair-wise dot products for everything
which is just Q KT. We run it through a
soft max. We get soft max of Q KT. We
multiply it by one to do the final
waiting and then boom the output comes
and that's this function. That's it.
Okay. So what we have done is we have
introduced three matrices learnable
matrices into the self attention layer.
Okay. Now,
okay. Let me just stop there for a sec.
Questions.
Yeah.
[clears throat]
>> Is there a relationship between AK, AQ,
and A
>> independent independent matrices?
>> Yes.
>> Like we have
>> could you use the microphone please?
>> Here we have three set of parameters K,
Q and P. If there are let's say if there
were 100 the total length was let's say
the number of total totals were let's
say 50. So you would have uh 50 for a
set of parameters like you'll have to
>> so if you have a 50 if the dimension is
50 long what is coming in the W's are 50
long then the key the what comes out of
it if you want it to be 50 as well so
this matrix needs to be 50 * 50 2500
>> U Luna
>> what are the different things the three
the three matrices are trying to
Sorry,
>> what are the different things that the
matrices are trying to learn?
>> We don't know. All we are saying is that
we have a self attention layer which can
pay attention to every pair of words.
But we need to give it some ways to
transform what is coming in into
potentially useful things. Right? As to
their actual usefulness, we'll have to
figure out if if it actually helps or
not. And of course, as you know, the the
punch line is that yeah, it helps
massively. That's why we do it. In
general, what you will find in the deep
learning literature is that whenever you
want to increase the capacity, the
modeling capacity of a particular model,
you just take a small piece and inject a
little matrix multiplication into it.
You take a vector that's showing up in
the middle and then you make it run
through a matrix to get another vector
and then further after you run it
through a matrix, you run it through a
little ReLU as well. Even better. So
that's how you inject modeling capacity
into the middle of these networks. Okay?
And that's what these people are doing
here. Yeah.
>> In the last step, you had the matrix V.
So on the previous example, you had used
the original matrix X. So could you just
say for why is it not using X? What does
that mean?
>> So what we're saying is that the in the
initial version we had three copies and
we treated them all identical. Now we
said well there are are there ways to
transform each copy into some other
representation which could be useful. So
we may as well use three different
matrices for it. Why stop with two?
There are three opportunities to make
them more expressive. We'll use all of
them.
>> Yeah.
>> You mentioned that these are kind of
you're tuning it. You're kind of
fine-tuning it. Is there any risk?
>> We're not fine-tuning it. Uh just to be
clear on the on the vocabulary here. So
we have added more weights to make them
tunable. What that means is that we when
we finally train this entire model,
remember all the weights are going to be
updated using back propagation, right?
In particular, these matrices will also
get updated using back propagation.
>> So there's no risk of is there a risk of
>> there's always the risk of overfitting
when you add more parameters to a model
>> which means that you have to look at the
validation set and all that good stuff.
We are basically adding more parameters
in a very interesting way because we
want to add more capacity to the self
attention layer. We want to give it a
more of an ability to learn things from
the data. Before it could not learn
anything. It could only do dot products.
So we we want to solve that problem.
All right, I'm going to continue and
we'll come back to this. Okay. Um
so uh all right, let's just just for
fun, I'm going to do this. Um the the
original paper is called attention is
all you need. This is a transformer
paper.
You folks should read it at some point.
Just want to show you something.
Uh
You see that? So that is the famous
transformer formula. Okay. And the only
thing we ignored is this root of DK
business in the back under it. I
wouldn't worry about it. The reason they
have it is because these soft maxes when
you have lots of numbers and some
numbers really really big what's going
to happen is that all the other numbers
are going to get squashed to zero. Okay.
And so to make sure the gradient flows
properly, they just divide it by a
particular number to make sure no number
is too big. Okay, that's a small
technical important but bit of a
technical detail which is why I ignored
it in my iPad. But the rest of it you
can see this is exactly the formula we
derived qt * v softmax.
Okay, so this is the famous transformer
formula
and congratulations now you understand
it.
You seem less than fully convinced.
Okay.
Yes. Hi iPad.
Now I have a bunch of slides which I had
but actually I'll come back to this. I
had a bunch of other slides. This is
from last year uh which actually
explains what I did in the iPad in a
very different way without using any
matrices and so on. I was looking at it
last evening and I was getting very
annoyed by these slides for some reason
because I felt that it wasn't really
conveying the core matrix sort of the
matrix uh the ability of using matrix
algebra to to actually do this so
efficiently and compactly which is why I
decided to like handdraw this thing on
the iPad. Okay, but you should read it
afterwards to make sure that whatever
you saw on the iPad actually matches
this. Okay, because two different ways
of understanding something always helps.
Um okay so this what we have here now to
just to recall
the by making self attention tunable we
get a very interesting benefit which is
that when you have these different
attention heads before
you could have two attention heads but
because there were no parameters inside
their outputs would have been identical
because the inputs are the same for both
therefore the outputs would be identical
but now by since each attention head
will have its own aq
matrix
the outputs are going to be different.
That's why it makes sense to do the
tunability thing because that's what
actually makes multiple attention it's
actually useful. Um
is is there actually any relationship
between AK AQ and AV or is the A just
for like a notation standpoint?
>> Just notation. The thing is we want to
use QV for the resulting matrix and so I
had to find something else to use for
the first one and I was like okay aqaq
and we at MIT we do subscript super
subcripts right so yeah
>> what what is the the size of the
matrices are there like square matrices
or
>> yeah so typically what happens is that
um there's a whole bunch you can think
of it as a hyperparameter in some ways
um typically what people do in most
implementations is that they will
actually just preserve the size so if
the incoming embedding is and they'll
make sure the the thing coming out of
thing is also 10. So you just do a 10x10
matrix to transform it. Uh but the the
the value v av matrix on the other hand
there's a bit more technical stuff going
on where it often tends to be smaller.
Um so for example let's say that your
incoming is 100 you do 100 to 100 for
the key 100 to 100 for the query. But if
you have say five attention heads, you
may do 100 to 20 for the W's because
ultimately all the V's are going to get
concatenated into another 100 again. So
I can tell you more offline but fun
broadly speaking these things tend to
get transformed. They don't they
preserve the dimension 10 and 10 out.
Yeah.
>> So this uh aq uh these numbers are
random when you start with it and then
allow it to back.
>> Exactly. Exactly.
So all right um
yeah so the values in these matrices are
weights learned through optimization
using SGD. Uh and then what that means
is that
each of these attention now has its own
copy of these matrices. It has its own
matrices and over the course of back
propagation these matrices will look
very different. Okay. So important each
attention head will have its own mat set
of three matrices. So if you have 10
attention heads 30 matrices will be
learned.
So by the math it seems like it's
creating essentially a relationship
between all of the content being
ingested and if you're creating if
you're ingesting all the content for
each attention head are there different
categories of attention head type that
you're trying to go after?
>> Yeah. So basically what we're trying to
do is to say a particular attention
head. So in any particular sentence it
may turn out to be the case that one
pattern could be about the meanings of
these words right like the word bank and
what it means the word station train
things like that. That's what really
we've been talking about. But there is a
whole other pattern to do with grammar
and tense and things like that. There
could be another one in terms of tone.
All those things are very important. And
a priority we don't know how many such
patterns exist. Much like in a
convolutional network, we don't when
we're designing how many filters to
have, we don't know how many kinds of
little things we have to detect, you
know, vertical line, horizontal line,
semicircle, quarter circle, stuff like
that. So, you just give it a lot of
capacity so that it can learn whatever
it wants.
All right. So, um so that that is the
transformer encoder. So, we have done
one the first of the three complications
needed to make it like industrial
strength and legit. Uh the second thing
we do is something called the residual
connection. So what we do is that
whatever comes out here right W1 through
W6 goes in and comes out as W1 hat W2
and so on and so forth right
actually sorry what comes out here is
the hats but what comes out here is some
intermediate W's right that is what the
selfident is going to give you some
intermediate W's what we do is and
because what's coming out here these
vectors are the same length as what goes
in we can just add them element by
element
So we take the input and we actually add
it to what comes out.
So why would we want to do that? Why
would we want to you know go to a lot of
trouble to process this thing and then
when it comes out we like literally add
up the original input? What's like what
do you think the intuition is?
So turns out, think of it this way. You
have a bunch of inputs. You send it to a
neural network. It transforms it and
gives you something else. Right? At that
point, you might be thinking, well,
everything that go everything that
happens in the network from that point
onward can no longer see your original
input. It can only work with the
transformed input. Right? But what if
your transformations are not great?
So as an insurance policy what you can
do is you can take the the transform
stuff and you can take the original
stuff and send both in.
Right? And this whole thing is and you
can Google it. It's called like a wide
and deep network and things like that.
But the whole point is that let's not
lose the original input anywhere. Let's
also send it along. But if you keep
adding the original input to every
intermediate layer, it's going to get
longer and longer and longer and bigger,
which you don't want because you want it
all to be the same size. So the simplest
alternative is to just add them up. You
take the transform stuff and you add the
original input. You get the same thing
again. The the what came out what came
in W1 was a 100 long vector and the
transformed version is also 100 long. So
just literally 100 100 add them up.
That's it. You get another 100 long
vector. So that is what's called a
residual connection. Okay. And as it
turns out, residual connections make it
m improve the gradient flow during back
propagation dramatically and that's why
they are very heavily used. And in fact,
RestNet, which we looked at for computer
vision, it stands for residual net
because it was the first network to
actually figure this out. It's not this
this is not just a transformer thing by
the way. It's widely used in you know
lots of new architectures. The notion of
a residual connection that's what it
means. Okay, so we do a residual
connection and then we come to the final
tweak which is called layer
normalization.
So once we add the residual connection,
we are going to do something else here
to these vectors before they continue
flowing. And what layer normalation does
is it basically says that
I you will recall from the very
beginning of the semester I've been
saying that whatever comes into a neural
network the inputs let's just really
make sure that they are all in some sort
of a narrow well- definfined range they
can't be in a big range right so for
pictures for images we divided every
number by 255 so that every little pixel
value is between zero and one okay for
continuous things like the heart disease
example we standardized by calculating
the mean and the standard deviation and
doing subtracting the mean and dividing
by the standard deviation. So when you
do that all the numbers are going to
roughly be in the minus1 to +1 range. So
in neural networks it's for backrop to
work really well you have to make sure
that no numbers get too big that all the
numbers are always in some sort of a
narrow range. So what layer
normalization does is to say you know
what whatever is coming out here I want
to make sure none of these numbers are
too big. I want to make sure they're all
well behaved in a small range because if
I don't do that back prop is not going
to work very well and so
is this what we do to ensure we don't
problem of vanishing right
>> so um so the there technically there are
there could be two problems there's an
exploding gradient and vanishing
gradient both are bad this is a way to
address it so you will find a whole
bunch of dash normalization techniques
layer normalization batch normalization
and so on and so forth all these are
methods to make that these numbers stay
in a small range so it doesn't cause
gradient issues later.
All right. So in particular
what we do is or what happens inside
this layer layer normalization is we
just calculate the mean and standard
deviation of every one of these
embeddings. Okay? Right? If you have
let's say six embeddings here, we'll
have six means and six standard
deviations, right? For each one across
the rows and then we standardize it.
Meaning subtract the mean divide by the
standard deviation. And when you do
that, all these things are going to be
nice and small. And then we do this a
little other thing where we we have
introduced two new parameters to rescale
it and move it around a little bit just
because adding more weights always helps
make these things better. So we add them
and this gets slightly complicated
because of the way the dimensions work.
So I'm not going to spend much time on
it. Uh and then what comes out the other
end is a very well- behaved set of
numbers in a nice and small and narrow
range.
Okay, so this is called layer
normalization. Um, you can see this link
to understand it a bit better. Um, and
we do that as well. So to put it all
together,
so this is a transformer encoder where
we have this multi head attention layer
where each attention head in the inside
of it is tunable with those a matrices
and then we have a residual connection.
We do that and then we do layer norm and
then we do the same thing in the next
feed forward layer as well. And then
boom out pops the output
>> by that definition in the multi head
attention layer when I'm doing tone and
everything theoretically I can add even
the biases or the hate speech aspects
which come in to take care of it right
so the model can account for the fact
that something is biased or something is
not
>> um the thing is it's not so much the
model is accounting for it is capturing
whatever patterns happen to be inherent
in the data it's capturing Right now
what you do with that capture is up to
you. It depends on the actual problem
you're trying to solve. In particular,
it is going to capture all the bad stuff
too because if your training header has
a lot of biased stuff in it, toxic
things in it, dangerous things in it, it
doesn't it doesn't have a sense of
values as to what it's good or bad. It's
just going to pick it up.
>> Yes.
>> On that then how do you actually make it
angle on those or how do you mitigate
the effect of those? That's a whole
course unto itself, but I'm happy to
give you pointers offline.
All right, so this is what we have and
remember what I said that this is just a
single transformer block and since what
comes in and what goes out are the same
dimensions, we can just stack them one
after the other, right? It's very
stackable. You can do it, you can
multiply, you can you can stack it
vertically as much as you want. And as I
mentioned, I think GPD3 has 96 of these
things stacked one on top of the other.
Um and so yeah that brings us to that is
it that is the transformer encoder and
this exactly maps to that. So basically
the input embeddings come in you add
positional embeddings and then you send
it to say these many attention blocks
and they all get added up and then it
comes over the attention block you add
the add and nom here means add means
residual connection because you're
adding the input which is why you have
this arrow going from the input being
added there and then you normalize it
send it along and do it again and out
comes the output.
So all right now just to be very clear
on what is being optimized during back
propagation in this complex flow right
now clearly the the embeddings that you
started out with both the standalone
embeddings as well as the positional uh
the position embeddings those things are
going to get optimized right those are
just weights they're going to get
optimized clearly everything inside the
transformer encoder block is going to
get get nominized right and what are
they well they are the aqa v matrices
for Each attention head layer norm has
parameters as well. The next like the
little feed forward layer has weights as
well. All these things are going to get
optimized and then it goes through this
relu which again has a bunch of weights.
It's going to get optimized and then the
final softmax has a bunch of weights.
That's going to get optimized.
All these things are going to get
optimized by back prop.
Okay. So in that sense you just step
back for a second and look at the whole
thing. It is just a mathematical model
with a lot of parameters
and we're just going to use gradient
descent or stoastic gradient descent to
optimize it. That's it.
Yeah.
>> For those eight matrices we train the
model, are we calculating weights for
like each cell of every possible matrix
based on the number of inputs like every
possible dimension up to the max number
of inputs?
Um actually the the weights themselves
um don't depend on how long your input
sentence is because remember what we're
doing is for each sentence that comes in
let's say the sentence has say three
words there are three embeddings for
that sentence each of those embeddings
gets multiplied by say AK right so AK
only needs to work needs to know how
long is each embedding it doesn't need
to know how many words do I have
and that's a I'm glad you raised that
question Ben because that's what makes a
transformer's number of weights
independent of the number of words in
your sentence.
It only depends on the vocabulary that
you're going to work with because the
vocabulary determines how many
embeddings you need, how many embeddings
you need. It the length only matters in
terms of the positional embedding
because if you have a thousand long
sentence, you need a thousand long
positional embedding matrix. But beyond
that, it doesn't care.
And that's why for example Google uh
Gemini 1.5 Pro which is a million it can
accommodate basically a million long
million token context window right it
can it's still very compute heavy but it
does not change the number of parameters
uh yeah
>> conceptually which weights are optimized
first but in sequential order or are
they optimizing the weights at the very
same time all
>> simultaneously because if you think of
back propagation ultimately you have a
loss function right and you calculate
the gradient of that loss function so if
you have a say a billion parameters that
gradient is basically a billion long
vector right and we're going to take the
gradient and we're going to do w new
equals w old minus alpha times the
gradient so all the w's are going to
update instantaneously
now the way it actually works in
computation is you're going to do it the
because of the back and back propagation
it's going to start at the end and
slowly flow backwards but when it's done
everything will be updated.
Yeah.
>> We take uh two attention heads and we
have the matrices of AK, A2 and AV in
them. Uh why would the parameters of all
three of them all the weights of the
three matrices on this side and this
side would be different because finally
the things you're inputting from this
side and the output is same. So the
learning process should be ideally the
same unlike like a CNN where we had put
filters which were different. So what
different thing we have to
>> because the initialization is different.
>> What do we mean?
>> Like what I mean is if you have two
heads right each head has three
matrices. The starting values of those
six matrix is different.
>> Starting value of A aka B AQ and A is
different for both the heads
>> right? Much like for all the weights
typically the values are randomly
chosen. If they were all the same thing
you're right. It won't you don't make a
difference right? They will all change
the same way. Yeah.
U is the input of the transformer of the
sentence or the the array of embedding
of each word.
>> Uh the in the transformer itself is
expecting embeddings in and so what
basically happens is that we get some
sentence we run it through a tokenizer
which connects it to a bunch of tokens
which are just integers and then it goes
through the embedding layer which maps
the integers to these embeddings and
then you feed it to the transformer. But
when you do back propagation, it comes
all the way back to the starting
embedding layer and updates those
weights.
>> Okay. So they can be trainable. So the
twist at the beginning must be input
here, but they can train.
>> They're trainable. Exactly. Exactly.
>> Uh yeah.
>> Are the attention heads solely parallel
or can you have like a stack of
attention heads?
>> Typically they are parallelized. Um and
because you can always stack the block
itself to get more and more power.
All right. So um so now to apply the
transformer right there are common use
cases are that you have a whole sentence
that comes in and then you just want to
classify it right the the canonical
thing being hey movie sentiment
classification boom positive or negative
right classification another common one
is labeling where every word gets
labeled as a multiclass label and that's
basically what we saw with our slot
filling problem and then there is
another thing called sequence generation
where you give it a sequence you wanted
to continue the sequence right generate
more stuff i.e. large language models
and all that good stuff. So, so this we
know already know how to do because we
actually literally built a collab with
this with the transformer stack. Now the
question is how can we do that right?
How can you do basic classification with
these things? So now if you again when
you send a sentence in after all that
stuff is done and when I say encoder
here I'm assuming that you may have one
one block you may have 106 blocks I
don't care at the end of the day you
send something in you get a bunch of
contextual embeddings out
right so at this point we need to take
these contextual embeddings and somehow
make it work for classification for just
classifying something into yes or no
positive or negative so it'll be nice if
we can actually take all these
embeddings and like essentially
summarize them into a single embedding,
a single vector
because if you have a single vector then
we can run it through maybe a relu and
then we do a sigmoid and boom we can do
a you know a binary classification
problem super easy right so this begs
the question okay how are we going to go
from the all the many blue things to one
green thing
okay now of course um what we can do is
we can simply average them we can take
each of the embeddings just simply
average them element by element, you'll
get a nice green thing. Okay. Um any
shortcomings from doing that?
>> You would lose the ordering of the
words.
>> You do uh well in some sense the
positional embedding, the positional
encoding you have in the input does have
this notion of position, right? So
you're not necessarily losing the order
necessarily, but you're sort of
averaging all this information into
something and averaging is going to lose
some richness.
Okay.
>> I think it's going to be skewed to the
one that has like the biggest number,
right? So something is influencing your
>> Yeah, the biggest ones are going to
dominate. But hopefully we won't have
too much of that because all the layer
nom business at the beginning has
hopefully made sure the numbers are all
in a reasonably small and well behaved
range. But the the point really is that
you're going to lose richness in the
information because you're just like
mushing it down. So there's a much
better and more elegant way to do this
which is that what you do is for every
sentence when you train it you add an
artificial token called the class token.
Okay, literally it's an artificial token
and it's designated as you know CLS in
the literature and then this token is
getting trained with everything else.
Okay. And so once you once you finish
training
that token has its own embedding too.
And because it has been trained with
everything else and this token is
remember it's a contextual embedding
which means that it's very much aware of
all the other words in the sentence.
So in some sense this context this CLS
tokens contextual embedding sort of
captures everything that's going on
about that sentence
right and so what we do is once we are
done training we just grab this thing
alone and then send that through a relu
and a sigmoid and boom you're done.
So this is a very clever trick to
somehow you know instead of averaging
everything at the end let's just have
something just for the whole thing the
sentence and just learn it anyway along
with everything else. So in like a meta
principle in deep learning is that
whenever you think you're making an ad
hoc decision about something like
averaging a bunch of stuff you should
always stop and say is there a better
way to do it where it doesn't have to be
ad hoc where the right way is learnable
from the data directly using back
propagation. Um there was a hand. Yeah.
>> Is there a reason that you
added the CLS at the start? Why not add
it at the
>> You can do it at the end. Is there any
difference?
>> Um the only thing to remember is that um
it's a good question. So different
centers are going to be of different
length, right? So there might be short
sentences, there might be long
sentences. In particular, the lot the
short sentences are going to get padded,
right? I remember I talked about padding
to make it to fit to one length. So what
internally the transformer will do is
ignore all the padded tokens because it
doesn't do it's just padding doesn't
really matter for anything. So if you
have the serless at the very end we have
to have much more administrative
bookkeeping to take everything but the
last one
ignore it and only do the last one just
much easier just to get in the beginning
that's the reason. Yeah.
>> What would be just a practical
application of this would be something
like sentiment analysis like a positive
or negative.
>> Yeah. So basically any kind of text
comes in and you want to figure out some
labeling problem like a classification
problem. The easiest example I could
think of was sentiment.
But you can imagine for example an email
comes into a like a call center
operation and you want to take the email
and automatically figure out which
department should I send it to.
Okay. So now now if the input data for a
task is natural language text, right? We
don't have to restrict ourselves to only
the input training data we have. Right?
Would it be great to learn from all the
text that's out there? So, for example,
to go back to that call center thing I
just mentioned, you know, why clearly,
let's say it's coming in English, the
ability to take that English email and
route it to one of 10 things. You know,
you should have to learn English just
for your call center application. You
should learn English generally and use
it for other things, right? So, why
can't we just learn from all the text
that's out there? And so, that brings us
to something called self-supervised
learning. And the idea of sens
supervised learning is this. So if you
recall the transfer learning example
from lecture four right where we had
restnet right and we took restn net we
chopped off the final thing we make made
it sort of headless and then we attached
that output of the headless restn net to
a little hidden layer and output and we
did the handbags and shoes and you will
recall that we were able to build a very
good classifier for handbags and shoes
with just like a 100 examples. Right? So
the question is why was this so
effective? Why was this so effective?
And turns out the reason why any of this
stuff actually works is because neural
networks or they learn representations
automatically when you train them. So
what I mean by that is when you imagine
a network, you feed in a bunch of stuff,
it goes through all the layers, it comes
out. Uh you can think of each layer as
transforming the raw input in some
different alternate representation of
the input. Okay? And so and these are
called representations. That's actually
a technical term. Um, and so you can
from this perspective when you train a a
neural network, a deep network with lots
of layers, what you're really learning
is you're learning a way to you're
learning how to represent the input in
many different ways. Each of these
arrows is a different way of
representing things. Plus, you're
learning a final regression model,
either a linear regression model or a
logistic regression model.
Fundamentally, that's what's going on.
Because the final layers tend to be
sigmoid, soft max, or just linear,
right? So the final layer if you just
look at the this part alone whatever is
coming in it's just going through
essentially a linear regression model or
a logistic regression model that's it.
So fundamentally you're learning
representations and a final little
model. Okay. But the reason why all
these things work so much better than
logistic regression is because those
representations have learned all kinds
of useful things about the input data.
They have sort of automatically feature
engineered for you.
So, so from this perspective you can
imagine that each layer here is like an
encoder. It encodes the input, right?
The first layer encodes it. The first
two layers encode something. The first
three layers encode something and so on
and so forth. So a deep network contains
many encoders. And so the question is
what do these representations actually
embody right? What do they capture? Is
it like specific knowledge about the
particular problem that you train the
thing train the network on or is it like
general knowledge about the input data
because if it is general knowledge about
the input we can use it to solve other
problems unrelated problems. So is it
specific knowledge or general knowledge
and it turns out they actually capture a
lot of general knowledge about the input
and that's why you can get reuse out of
them you can reuse them for other
unrelated things because they have
captured general stuff. So if you look
at this, I think I've shown you before,
right? If you if you look at a network
that classifies everyday objects into a
bunch of categories, it can learn all
these little patterns in the beginning
and later on and so on and so forth. And
this is a face detection network. It has
learned how to look at, you know,
identify little circles and edges and
nose like shapes and finally faces. So
all these things are examples of
representations, learning interesting
things about the input. Okay. So since
these representations are capturing
intrinsic aspects of the data, you can
use it for other things, right? You can
take a face detection neural network and
use it, reuse it for emotion detection
for instance.
U so the question is if you can somehow
get like an encoder that generates good
representations for your input data, we
can simply build a regression model with
those as input and labels as output and
be done. And this is exactly what we did
with RestNet for handbags and shows. We
found a thing that had already been
trained on similar everyday objects,
everyday images. And the key insight
here is that since we don't have to
spend precious data on learning these
good representations,
we won't need as much label data in the
first place because the pre-training
used a lot of data and you're sort of
piggybacking on that data. So in some
sense, your training data is everything
that the pre-trained model was trained
on plus your little 200 examples.
Um, okay. So this is what we did. We
used headless resonate as an encoder
that can take raw input and transform it
into useful representations. Uh this is
what we did. All right. So the general
approach is that you find a deep neural
network built on similar inputs but
different outputs. Uh and then you
basically grab maybe the penultimate uh
representation or the one before that.
Then you chop off the head. You attach
your own output head. Train the whole
thing just the final layer or train the
whole thing if you want. Right? This is
like the playbook we followed for
restnet. The same thing works for all
kinds of other data types as well. So
now to build such a model we need
labeled data, right? We were lucky
because restnet was actually trained on
imageet data which is like a million
images each of which labeled into
thousand categories which is very
convenient for us, right? But what if
you want to build a generally useful
model for text data?
Clearly we need to collect a lot of text
data. But that's no problem because
internet is full of text data, right? we
can easily escape the internet. We can
just download Wikipedia. So that's not a
problem. The problem is something else
which is that how do we define an input
label for a piece of text? So for an
input sentence, what should the output
label be? That's the key question.
Because if you can answer this question,
you can just spray train all these
things on all kinds of text data, right?
So the like a beautiful idea for doing
this is called self-supervised learning.
And the key idea is that you take your
input, whatever the input is you take a
small part of the input and just remove
it and then ask your network to fill in
the blanks from everything else.
Okay, so this is called masking and it's
just one of many techniques in
self-supervised learning, but this is
very commonly used. So this is original
input, right? And then you take it and
then you just like take this thing in
the middle here randomly and and and
zero it out or mask it. And so this
incomplete input is your now new input
and the thing that you took out becomes
your your fake label.
So you can almost imagine right if you
take if you if you're baking donuts you
you make a donut and then you punch a
hole in the middle of the donut the the
donut with the hole is your no input the
munchkin is the label.
Am I making everybody hungry at this
point? So,
so and once you do that, no problem. You
have an input, you have an you have
labels, you just train a neural network
to essentially predict those to
basically fill in the blanks.
And so if for example, if you take a
sentence like the Sloan School's
mission, you can just go in there and
just just knock out randomly a bunch of
words like this second. And the ones I'm
knocking out, I'm just putting the word
mask in it just to show what I'm doing.
And then what it's actually given this
sentence, it will try to fill in the
blanks with actual words.
Okay,
so now for the amazing part. In the
process of learning to fill in the
blanks, uh the network learns a really
good representation of the kind of input
data it's seeing. And it kind of makes
sense, right? Because if I give you a
sentence with a few missing blanks and
you're able to very successfully fill in
the blanks, you have learned a whole
bunch of stuff about the world to be
able to do that, right? If I say the
capital of France is Dash and you're
like Paris, okay, how did you know that?
It's sort of like that. By learning to
fill in the blanks, you really have to
learn how how all these things work, all
the the connections between various
words and so on and so forth. So, and so
what you can do is once we build such a
model, we can just extract an encoder
from it, right? And then we'll fine-tune
it like we do with library transfer
learning. But this how you build a
generic a generic pre-trained model on
unlabelled data.
And so we can use a transformer encoder
to build this whole thing in the middle
because remember the transformer can
take any sentence and give you the same
size sentence back along with
predictions for everything. So we can
just have it take this thing in and ask
it to just predict all the missing words
here.
And
so uh to put it in other words, masked
self-supervised learning is just a
sequence labeling problem.
So basically this is the sequence that
comes in and then you you tell the
transform and you get all these
embeddings. It goes through all that
stuff. You really don't care about these
outputs. But wherever the word mask went
in in the input, you you basically try
to get it to the right answer is for
example the word mission and you're
trying to and that is the right answer.
This is the right answer here. And then
you take these right answers, create a
loss function, and do back prop and
boom, you're done.
Inputs, right answers, and and you're in
business. That's it. Now, if we
pre-train a transformer model like this
on massive amounts of English text,
let's say we did that. We get something
called BERT. BERT is a very famous
transformer model. And BERT was the
first model actually that Google used to
upgrade its search in 2019.
like the br the Brazil visa example you
may recall from earlier lectures that
uses BERT under the hood. Okay. Um and
so now I just want to show you because
you can actually read the BERT paper and
it'll actually make sense to you now
based on what you have learned in this
class. Look at this BERT's model
architecture is a multi-layer
birectional transformer encoder. Okay,
transformer encoder. We denote the
number of layers transformer blocks as
L. The hidden size is H and the number
of attention heads as A. And how much is
that? Uh okay we want uh h is 768 okay
so which means that the embedding sizes
or 768
and the hidden feed forward layer is
four times as much so it's 4096 and so
sorry the the the 4096 the feed forward
layer the embeddings are 768 and you can
see there are two BERT models here this
one has 12 transformer blocks this one
has 24 transformer blocks
Okay, so you can actually read this
paper. You can you can actually relate
it to exactly what we discussed in
class. It'll all make sense.
Birectionally means that the words can
pay attention to every other word in the
sentence. And as we will see on Monday,
you can have you have a diff another
transformer thing called a causal
transformer in which you only pay
attention to the words that came before
you, not the ones after you. So
birectional means all words are seen.
[snorts] Okay. So um so what we do is
remember we said to do solve sequence
classification you can add a little
token at the beginning uh and then boom
use it for classification as it turns
out but very conveniently for us the
people who built bird they actually auto
they when they train bird they just use
the CLS business
during training so it's actually
available for us out of the box so when
you use bird for sequence classification
you don't even have to do any surgery on
it it just gives you the class token
automatically which is very convenient
uh and you can also use it for sequence
labeling as well. So for sequence
classifications and sequence labeling uh
BERT is actually usually a really good
starting point and in particular there
have been lots of improvements and
variations of BERT over the years and if
you're curious about this there's a
thing called the sentence transformers
library which has got a whole bunch of
BERT related code and resources that you
can use to do things out of the box.
Okay. So okay there's a bit of a word
wall.
So to solve any of these problems
classification or labeling where the
input is natural language we can
obviously use a model like BERT label a
few hundred examples attach the right
final layers and fine tune it like we
did for the restn net but if your
problem is like a standard NLP problem
okay you don't even have to do that
because people for these standard tasks
they've already pre-trained it on those
standard tasks right and so you can do
all these things without any fine tuning
at all like literally out of the box u
and so there are many hubs which have
these pre-trained models, but perhaps
the biggest one is the hugging face hub.
And I checked last night, it has 525,000
models
available. I think if I recall last year
when I taught Hodel, I think the number
was a lot smaller, maybe 50,000. So it's
like growing really, really fast. Um,
and so all right, let's just switch to a
hugging face collab.
So, hugging face, how many of you are
familiar with hugging face?
Okay, it's good. All right, so um for
the others, basically you have a whole
bunch of pre-trained models on hugging
phase. You actually have a lot of data
sets you can work with for your own
tasks. Uh there are lots of people
demoing what they have built in this
thing called spaces and of course a lot
of documentation and so on. So the thing
you can do is what they have done is
they have organized all these models by
the kind of task you can use them for.
So you can see here there are a whole
bunch of computer vision tasks that you
can use them for. There's a whole bunch
of natural language tasks like text
classification
uh feature extraction this and that lots
of interesting examples here. And so
what you do is you just literally can go
in there and say okay I want to do a
text classification. You hit it and then
it tells you all the models that are
available. Turns into 50,000 models just
for text classification. And you can
look at okay which is you know most
downloaded or which is the most liked
and then you can just use them as a
starting point for whatever you want to
do. Okay. So so that is hugging phase
and so the way you do hugging face is
I'm just connecting it. Um
if you have a problem which the input is
natural language text the first question
you have to ask yourself is it standard
or not? Is it a standard task or not? If
it's a standard task you just go go that
do not reinvent the wheel. This thing
will usually work pretty well. Okay. So
here we will use this thing called um
the transformers library from hugging
face in particular the pipeline function
to demonstrate quickly how to do this
thing. Fortunately this library as of
this year is pre-installed in collab so
we can we don't have to install it. We
can just start using it right away. So
we'll take this example where you have a
bunch of text which says um
dear Amazon last week I got an Optimus
Prime action figure from your store in
Germany. Unfortunately when I opened the
vicage I discovered to my horror that I
had been sent an action figure of
Megatron instead. Can you imagine that
person's like sheer distress at this?
Um, so as a lifelong enemy of the
Decepticons, I hope you can understand
my dilemma. So to resolve the issue, I
demand an exchange. Encloser copies
expect to hear from you soon. Sincerely,
Bumblebee.
Okay, that Okay, they should have come
up with a better name for this example.
Uh, all right, cool. So that's the text
we have. So we import the this pipeline
function is the one that basically gives
you the ability to out of the box start
using it without any pre-training,
nothing like that. Okay, so we download
this thing. Um, oh wow, I got an A00
today. That happens very rarely. All
right, sorry.
So here, let's say you want to classify
that text. Okay, you want just want to
classify it for sentiment. You literally
go in there and say pipeline
text classification. That's the task you
want the pipeline to do for you, right?
And you create a classifier. Okay, it's
going to download a bunch of stuff. Uh,
and then so on and so forth.
The first time it just takes time to
download and then you literally take the
text you have here and then run it
through the classifier as it was just a
little function right you get some
outputs and then actually just do this
this way
negative sentiment is negative with 90%
probability pretty good right sequence
classification solved I mean sent
sentiment classification solved so we'll
try a few different examples uh I hated
the movie I if I said I loved the movie
I would be lying okay that's a little
tricky The movie left me speechless.
Incredible. And then I had to add this
last thing here last night. Almost but
not quite entirely unlike anything good
I've seen. Okay. And that's not
original. By the way, people who have
read Douglas Adams will know this famous
sentence about somebody drinking some
beverage and saying it's almost but not
quite entirely unlike tea. So I was
inspired by that. So anyway, we'll see
what happens. Um.
All right. Put it in there. Okay. So
negative. I hated the movie. Okay, fine.
If I said love me, I'd be lying.
Negative. Movie left me speechless. Uh,
it says it's negative, but it could go
either way, right? A good classifier
would have probably given you a
probability around the 50% mark because
it's sort of right on the fence. Um,
incredible, it's positive, and then it
got fooled by my crazy long sentence and
it says it's positive. Okay, now that's
classification. Here's one other quick
example. So, you can actually give it a
piece of text, right? For example, you
can take like a a Reuter's news story.
You can feed it and say extract all the
company names from it. Extract company
names, people names and things like
that. It's called named entity
extraction. And there are in the back in
back in the day people would bring they
would hand build painstakingly all these
very complex systems to be to do named
entity extraction. Now it's just a
pipeline away. So you can take this
thing and you can say create a pipeline
for any name extraction and for any
particular task that you're using there
might be a few additional parameters you
can set right as a part of the
configuration. So we download this
pipeline.
Okay, perfect. And then we run the
output. So it says okay good. Amazon is
an organization
uh
and Germany is a location lock which is
nice. So these things have a standard
vocabulary as to or lock things like
that which you can read up in the
documentation. Uh and then Bumblebee is
a person and then boy all the like the
Optimus Prime transformer stuff is all
it got full right. It thinks Optimus
Prime is miscellaneous. Uh decept is
miscellaneous and so on and so forth.
But you get the idea. You can take
standard things like Reuters use stories
and so or you can just boop. You can get
a very good entity extraction right out
of the bat. And once you get these
entities extracted, then you can put
them into a nice structured data table
like a database and then you can run
traditional machine learning on it.
Okay. Um and then I had I think a few
more examples of question answering and
uh actually let's just try that. um you
can actually give it a thing and ask a
question about it and you can actually
give you the answer which gets into the
causal transformer thing that we're
going to see on Monday which builds up
into large language models because you
obviously can give something you can
give a passage to chat GPT and ask a
question ask it to give you an answer so
it's really in that thing but um just
for fun let's just do that to see if
it's any good um okay so what does the
customer want and the output is an
exchange of megatron and it's telling
you which where it starts in the text
and where it ends the relevant passage.
It's pretty good, right? So because
remember if you have stuff like this
then when you ask like a large language
model a question it gives you an answer.
You can actually ask it to give you
exactly where in the input it found the
answer and because you know these things
are going to elicitate you can actually
look at the input that it's claiming to
use and look at what it says and see if
they actually match. It's a way to sort
of essentially do QA on LLM output.
Um okay so that's what we have here and
I have other budget much of which which
I'll ignore for the moment because I
want to go back to the PowerPoint.
So yeah so if you have a standard task
uh you know you can just use pipelines
and hugging face to actually solve many
of them out of the box without any heavy
lifting. So I mentioned earlier on that
transformers have proven to be effective
for a whole bunch of domains outside of
natural language processing um like you
know speech recognition, computer vision
and so on and so forth. Um and so I want
to give you a couple of quick examples
of how to think about transform using
transformers for non-ext applications.
Okay. So uh the the key insight here is
that the architecture of the transformer
block that we have looked at amazingly
enough can be used as is with no changes
no surgery needed. No clever thinking
required for any particular application.
What is needed where the clever thinking
may be required is you need to take the
inputs that you're working with and you
need to figure out a way to tokenize and
encode them into embeddings
which can then be sent into the
transformer. So all the action is in
taking that input that non-ext input and
figuring out a way to cast them in the
language of embeddings. That's where the
that's the game. Okay. So um here is
something called the vision transformer
which is very famous actually. I think
it may be the first perhaps the first uh
transformer architecture that was
applied to vision problems. So um so
let's say you have a picture um yeah so
let's say you have this picture okay
it is just a picture okay so you have to
find a way to create embeddings from
this picture or to tokenize this picture
in some way with sentences you know I
love hard well obviously I love and hard
are three tokens it's pretty trivial to
figure out how to tokenize them but with
a picture what do you do right it's kind
of weird to think of tokenizing a
picture so what these people did is that
they say you know what I'm going to take
this picture and chop it up into small
squares.
Right? So in this example, they have
taken this big picture and chopped it up
into nine little pictures. Okay? Then
you can take each of those nine
pictures.
Each of those nine pictures, right? If
you look at the how it's represented,
it's just three tables of numbers,
right? The RGB values, right? So you can
take all those numbers and you just
create a giant long vector from it.
Okay? you have a huge long vector and
then you run it through a dense layer to
come up with a smaller vector
and that smaller vector is your
embedding.
That's it. But the way you transform the
long vector into small vector is just a
dense layer whose weights can be
learned.
So what these people did is they said
well I'm going to first chop it up into
these patches and then I take each patch
and do a linear projection. Right? A
flattened patch is nothing more than a
three tables of numbers flattened into a
long vector. That's what the word
flatten here means. And once you flatten
it, I'm just going to run it through a
dense layer. So, by the way, you will
see the words linear projection. It's a
synonym for run it through a dense
layer.
So, you run it through a dense layer,
right? You get these nice vectors, these
vectors.
And now you say, well, you know what? I
have to take the order of these things
into account because clearly this little
patch is in the top left while this
patch is somewhere in the middle. Right?
The order matters in the picture
otherwise every jumbled version is going
to be the same thing. So you use
positional embeddings
you basically say there are nine
positions in any picture right 0 1 2 3 4
5 6 7 8 there are nine positions. So I'm
going to create nine position embeddings
and then I'm just going to add them up.
Then I'm just going to add them up to
this embedding. Just like we did with
words. With words, we each word had an
embedding. Each position had an
embedding. We added them up. Here each
image has an embedding. The position of
the little patch in the picture has an
embedding. We add them up. Okay? And
then because we want to use it for
classification, no problem. We'll have a
little CLS token
and then we just run it through the
transformer. That's it.
and then you get the CLS token and then
you can attach a softmax to it and say,
"Okay, it's a bird, it's a ball, it's a
car.
That's it. This simple approach actually
works
amazingly enough."
Okay, so that is the vision transformer
and I'm going through it fast just to
give you a sense for how these things
work. Uh any questions? Yeah. Uh my
question is like uh in case of uh text
we had fixed number of tokens that is
amount of words which could be there in
your vocabul in the English vocabulary
but here if you look at images they will
probably go into trillions that I know
like we are not talking about one image
but we take a total set of plot of
images and we try to subset each one of
them each one would have its own uh uh
own weights like own parameters. There
is no notion of vocabulary here. All
we're saying is that given any image, we
create nine patches, sub images from it.
Each of those patches gets passed
through a dense layer and out comes an
embedding. So at that point, any image
you give me, I'm going to give get you
nine embeddings out of it. And once I
get the nine embeddings, I just throw it
into the meat grinder, the transformer
meat grinder.
All right. So uh another example I think
some of you have asked me outside of
class um how good are transformers for
structured data tabular data right for
tabular data in general um things like
xg boost gradient boosting works really
really well so it's good to try them
certainly I don't think transformers and
deep learning networks have any great
edge over xg boost for structured data
problems so it's worth trying both of
them however you can use transformers
for this stuff too so that's called the
tab transformer one of the first ones
wants to come out a transform of a
tabular data and again it's pretty
simple. All you do is
in any kind of input that you have, you
will have some categorical variables,
right? Like blood pressure, things like
that, right? Not blood pressure, bad
example, gender, right? Um, and so on
and so forth. And so what you do is you
take all the categorical features and
for each categorical feature, you create
embeddings
because a categorical feature is just
text.
A categorical feature is just text. So
you can create text embeddings for it.
No problem. Um,
and you take all the continuous
features, right? Cholesterol and blood
pressure and whatnot, right? To go to
the heart disease example, and then you
take just create all the correct them
all and just create a vector out of
them.
You're just a vector. Okay? Then you run
these the embeddings for all the
categorical variables through a nice
transformer block. And you can see here
it's exactly the block we have seen
before. no difference. And then at the
very end when it comes out of the
transformer, you take all the contextual
stuff coming out of the transformer and
then you concatenate it with the
continuous features.
Okay. And then you run it through maybe
one or more dense layers and boom
output.
So this is a tab tabular data
transformer. And there are many you know
refinements improvements over the years
that have come since then. But the key
thing I want you to rec remember from
here is that categorical variables can
be very easily represented as
embeddings. That's the key. Okay. Uh all
right. So that's that. Now once the
input has been transformed into sort of
this common language of embeddings, we
can process them without changing the
architecture of the block itself because
all it wants is embeddings. It's like
you give me embeddings, I give you a
great contextual embeddings out and
nobody gets hurt, right? That is the
deal with the transformer stack. So um
now this this ability this sort of since
the transformer is agnostic to the kind
of input as long as it comes into comes
in as a form of an embedding you can use
it for multimodal data very easily. So
for example let's say that you have a
problem in which you have a picture that
you have to be sent in some text that
goes in a bunch of tabular data coming
in well you take the text and do
language embeddings like we know how to
do you take the image and do image
embeddings like we just saw with the
vision transformer. You take tablet data
and do tab data embeddings like we saw
with the tab transformer. Once we do it,
it's all a bunch of embeddings
and then you attach a little class token
on top, send it through a bunch of
transformers blocks and then out comes a
contextual class token the contextual
version run it through maybe a sigmoid
or a softmax predict the label done.
So this is extremely powerful its
ability to handle multimodel data. Okay.
And that's why for example if you look
at Gemini Google Gemini 1.5 Pro GPT4
vision and so on you can send it images
and a question and you'll get an answer
back because every modality that goes in
is cast into embeddings and once it's
embedded one once it's embeddingized
then the transformer doesn't care. It'll
just do its thing.
It it will decide for example that this
word in your question actually is highly
related to that patch in the picture.
Right? you'll just figure it out.
Uh, okay. That's all I had because
there's a time pering 9:55. Perfect. All
right, folks. Thanks. Have a great rest
of your week.