So, all right. So, transformers, even
though they were originally invented for
machine translation, right, going from
English to German and German to French
and so on and so forth,
they have turned out to be an incredibly
effective deep neural network
architecture for just really a vast
array of domains. It has reached a point
where if you're actually working with on
a particular problem, you almost
reflexively to will try a transformer
first because it's probably going to be
pretty darn good.
Okay? So, they have just taken over
everything.
Um and obviously they have they've
transformed translation, which is the
original sort of target, uh Google
search, really information retrieval,
completely transformed speech
recognition, text-to-speech, even
computer vision. Even the stuff that we
learned with convolutional neural
networks, now there are transformers for
computer vision problems that are
actually quite good.
Right?
Um which is kind of shocking because
they were not even designed for that.
Um and then, you know, reinforcement
learning. And of course, all the crazy
stuff that's going on with generative
AI, large language models, multimodal
models, everything everything runs on a
transformer.
Okay? Uh and then there are numerous
special purpose systems
and I find these to be even more
interesting.
Um you know, like AlphaFold, the protein
folding AI, is run runs on a transformer
stack.
Okay? And I could just list examples one
after the other.
So, it's just amazing. It's incredibly
uh flexible architecture.
Um and I think we are lucky to be alive
during a time when such a thing was
invented.
And I'm not getting paid to tell you any
of this stuff.
All right, it's just amazing. Okay. So,
let's get going. We will use search um
or more broadly information retrieval as
a motivating use case. So, these are all
examples where people are typing in
natural language queries or uttering
natural language queries into a phone
and we need to sort of make sense of
what they want. And it's not like, you
know, write me a limerick about deep
learning where there could be many
possible right answers. It's more like,
okay, tell me all the flights that are
leaving from Boston to going to
LaGuardia tomorrow morning between 8:00
and 9:00. Well, you better get it right.
Okay? Accuracy is a high bar.
So,
um or, you know, how many customers
abandoned their shopping cart? Find all
contracts that are up for renewal next
month. Uh you know, tell me the all the
customers who ended the phone call to
the call center yesterday not entirely
pleased with the transaction. Right? The
list goes on and on. And so, in
particular, we'll focus on this
travel-related example today. Okay? Uh
find me all flights from Boston to
LaGuardia tomorrow morning, right? That
kind of query.
Um and so, in these sorts of use cases,
a very common approach historically has
been, well, we will take this, you know,
natural language query
and then we will convert it into a
structured query. By that I mean we will
parse the query and we'll extract out
key things in that query. Once we
extract out those key things, we will
reassemble it into a structured query,
like a SQL query, right? Uh SQL is just
one example of a possible structured
query. There are many many ways to
structure queries.
But SQL is sort of familiar to lots of
people, so I'm using that. So, you take
the SQL. Once you have the SQL query,
you're in a very comfortable structured
land, in which case you just run the
query through a some database that you
have, get the results back, format it
nicely, and show show it to the user.
Right? That's the flow.
So, the question becomes
um
how do we automatically extract all the
travel-related entities from this query?
Right? We want to be able to extract
BOS, LGA, tomorrow, morning, flights, so
on and so forth. These are all the
travel-related entities we want to
extract out, right? That's the problem.
And so,
we will use a really cool data set
called the airline travel information
system data set and I'll explain the
data set in just in just a bit. We'll
use this as the basis for this example.
And so, the way we think about it is
that
we we have a whole bunch of queries in
this data set.
And fortunately for us, the researchers
who compiled this data set,
they went through every one of these
queries, right? And we have, you know,
several thousands of them. They went
through every one of those queries and
they manually tagged each word in the
query
with what kind of travel entity it is
or none of them, right? So, for
instance, so they class they call them
slots. So, they will take each word in
the query and assign it to a slot, a
particular kind of slot, and I'll
explain what slot means in just a
second. Okay? That's the basic idea. So,
so, for example, if you have something
like I want to fly from
Okay? And this is a flight database, so
you can assume that everything is
related to a flight flying. So, if you
have all these words, I want to fly
from,
each of these words these five words
gets mapped to something called the O,
which means other.
It's the other slot, right? We don't
really care about it. It's the other
slot.
And then we come to Boston.
Oh, Boston is very special, right?
Because, you know, it's clearly a
departure city. So, we actually tag it,
we assign it this label. Think of it as
just like a classification problem,
right? A multi-class classification
problem. So, we assign it to B from
loc.city_name.
Okay? That is the label you assign it.
Okay?
And then you go to at. You don't care
about at. It's O, other. You come to
7:00 a.m.
And then, okay, that is depart time. So,
depart time and then another depart
time. And here you see there is a B and
then there is an I.
Right? So, what's what we are saying
here is that there could be entities who
are described using more than one word.
Like 7:00 a.m., right? Two tokens.
And for that, we need to be able to
figure out, okay, the second token is
really
is part of the first token. Together,
they define the notion of a departure
time. So, what the B means that is that
this is the word this is the token in
which we are beginning the idea of a
departure time. And then I means we are
in the middle of this description.
B is for beginning.
So,
you can see here. So, there is a B here
and there is an I. B for beginning, I
for intermediate or in the middle.
Um and then at, we don't care. 11:00 B
arrive time.
Boop boop boop. Morning arrive time
period.
So, this is an example of how you can
take a sentence and then manually label
every word in the sentence with
something that's relevant to your
particular problem.
And
turns out these people
every word is classified into one of 123
possibilities.
Okay? Um so, aircraft code, airline
code, airline name, airport code,
airport name, arrival date, relative
name. Now, you get the idea.
They want a round trip versus a one-way.
The relative to today because if
somebody say tomorrow morning, it's
relative to today, so you need to notion
you need absolute time and you need
notion of relative time.
So, they basically thought of every
possibility with these researchers. And
so, the every word in every one of these
queries is assigned one of these 123
labels.
Any questions on the setup?
Um
Did they have to contextualize what
comes before than let's say Boston? So,
if someone says from
Boston, so that there should be
contextualization with the from to
Boston. So, because they did it
manually, they could just read it and
figure it out, that's what they mean,
right? You Boston is the the departure
city and not the arrival city. So, do
they have two tags to Boston, which is
some like, you know, departure city as
well as arrival city
word Boston? In that particular phrase,
it's it's clear from that particular
case in the context of it as a human
reading it that Boston is a departure
city. So, it just only gets that tag. In
that sentence. In some other sentence
where people are coming into Boston,
it'll have a different tag.
I was wondering if my query like the
others, basically there is like, for
example, if my query was
giving flights from Boston at 7:00 a.m.
and
uh the
flights from Denver at 11:00 a.m.
You mean like a compound query? Yeah.
So, this one only takes single queries
into account.
Because most people are like, you know,
give me a flight from here to there. Or
what is the cheapest thing from here to
there? And we'll see examples of queries
later on.
Okay.
Uh all right. So, that's that's the
deal.
So, basically what we have this you
know,
uh this problem that we have here is
really a word-to-slot,
word-to-slot multi-class classification
problem.
Okay?
Um because if you look at that that
input, we want to be able to take that
input and a really good model will then
give you this as the output.
Right? Because this is what a human
would have done.
So, that is our problem. Okay?
So, the question is
um the the key thing here is that each
of the 18 words in this particular
example must be assigned to one of 123
slot types, right? Each word. It's not
like we take the entire query and
classify the entire query into one of
123 possibilities. Every word in the
query has to be classified.
That is the wrinkle.
Okay?
So, now, if we could run the query
through a deep neural network and
generate 18 output nodes,
it goes through some unspecified deep
neural network. And when it comes out
the other end, the output layer has 18
nodes.
Okay?
Because that is that is the that is the
that is the the the dimension of the
output that we care about. 18 in, 18
out. 18 in, 18 out, right?
And then for each one of those 18 nodes,
maybe we could attach a 123-way softmax
to each of those 18 outputs.
By the way, isn't it cool that we can
just casually talk about sticking a
123-way softmax onto each one of the 18
nodes?
Folks, wake up.
You're not easily impressed. I'm
impressed by that.
So, okay.
So, so the So, here's the key thing,
right? We want to generate an output
that has the same length as the input.
But the problem is the inputs could be
of different lengths as they come in.
They could be short sentences, long
sentences, we don't know, right?
Yet we need to accommodate this range
this variable size of input that's
coming in.
But the key thing is the output has to
be the same thing as the input, the same
cardinality as the input.
Okay, that's a one big requirement.
In addition, we want to take the
surrounding context of each word into
account, right? To go to Ronak's
question, when you see the word Boston,
you can't conclude whether it's a
departure city or arrival city.
You have to look at what else is going
on around it. Is there a from? Is there
a to? Things like that to figure out
what how to tag it. So, clearly the
context matters.
And then we clearly have to take the
order of the words into account.
Going from Boston to LaGuardia is very
different than going from LaGuardia to
Boston.
So, clearly the order matters.
Right? So, the context matters and the
order matters. And the output has to be
the same length as the input.
Okay?
So, context matters, right? Just a few
fun examples.
Remember from the last week that the
meaning of a word can change
dramatically depending on the context.
And we also saw that the standalone or
uncontextual embeddings that we saw for
last week, like Glove, um
you know, they don't take context into
account because they give a single
unique embedding vector to every word.
And if a word ends up having lots of
different meanings, that vector is kind
of some mushy average of all those
meanings.
Okay. So,
the word see. I will see you soon. I
will see this project to its end. I see
what you mean. Very different meanings
of the word see. This is my favorite,
bank.
Uh I went to the bank to apply for a
loan. I'm banking on the job. I'm
standing on the left bank. And so on. Uh
it. The animal Oh, this is actually very
It's a good one. The animal didn't cross
the street because it was too tired. The
animal didn't cross the street because
it was too wide.
Can you imagine
a deep neural network looking at this
word it and trying to figure out what
the heck does it word it mean?
What is it referring to?
Tricky, right?
Um and then, you know, if you take the
word station, and I have the station
example here because we're going to use
it a bit more the rest of the lecture.
The train You know, the station could be
a radio station, a train station, being
stationed somewhere, the International
Space Station. The list goes on.
So, clearly order matters. I mean,
context matters.
And
clearly order matters. You can come up
with your own examples. Let's keep
moving.
Okay?
So, the Transformer architecture
is a very elegant
architecture
which checks these three boxes
beautifully.
Okay?
Um it takes the context into account,
order into account, and then, you know,
whatever is produced out there
is the same length as whatever is coming
in.
And the reason it's called the
Transformer
is because if 10 things come in,
10 things go out, but the 10 things that
go out are a transformed version of the
10 things that came in.
That's why it's called the Transformer.
Okay?
If 10 things came in and like one thing
go goes out, well, sure, it's been
transformed, but what is it? It's some
weird thing. But when 10 comes in and 10
goes out, the 10 10 is preserved. Each
one is getting transformed in
interesting way.
That's why it's called the Transformer.
So, developed 2017, just dramatic
impact.
So, by the way, the effect of
Transformer, um
Google had spent a lot of research on
machine translation and obviously
search. Uh and then when the Transformer
is invented, uh they took a model called
BERT, which we will uh see on Wednesday
in detail, and then they introduced BERT
into their search, and the results were
dramatic.
And from what I've read, apparently the
impact of doing that was a
Typically, when you make an improvement
to search, the improvement is very, very
marginal because it's already a very
heavily optimized system.
And then when the Transformer thing came
along, there was actually a significant
jump in search quality. So, for example,
and you can actually read this blog post
uh which came out when they introduced
BERT into search. It gives you a bit
more detail. But here, so if you had if
you were querying something like uh you
know,
"Brazil traveler to USA needs a visa."
Right? You would think that it is it
should give you information about how to
get a visa if you're a Brazilian want to
come to the US, right? Uh but it turns
out the first result was how US citizens
going to Brazil can get you know,
get a visa.
So, clearly it's not taking the order
into account.
Uh but once they introduced it, boom,
the first thing was the US Embassy in
Brazil.
And a page on how to get a visa.
So, the effect was dramatic.
And so, this is a seminal paper,
right? And it's actually worth reading
the paper. And uh and it's worth and you
know, this is the picture this this is
like an iconic picture at this point
in the deep learning community. And we
will actually understand this picture
by the end of Wednesday.
Um and so, but the funny thing is that
when the researchers came up with it,
they didn't realize, in some sense, like
what they had stumbled on uh because
they were really focused on machine
translation.
It's only the rest of the research
community that took it and started
applying to everything else and found it
to be really, really effective.
Okay. So, we're going to take each one
of these things and figure out how to
address them and thereby build up the
architecture.
Any questions before I continue?
Yeah.
Is there any uh
benefits to discarding some of those
unclassified nodes before it goes out
rather than going like you have 18 words
input, discarding all the ones that
don't actually matter and just doing
like eight for your output?
Yeah, yeah. I think that's a totally
fine way to think about it. Basically,
what you're saying is that can we have a
two-stage model? The first-stage model
is like a O non-O classifier. And the
second-stage model only goes after the
non-Os. That's a totally fine way to do
it.
Yeah.
But as you can see, if you even if you
go with the just a simple one-stage
model, if you use a Transformer, you get
fantastic accuracy.
And we'll do the collab in a bit.
Uh all right. So, let's take the first
thing. How do you how do you take the
context of everything around the word
into account?
So,
so let's say that this is this is the
sentence we have. The train slowly left
the station.
Okay? For each of these words,
we can calculate a standalone embedding,
say something like Glove.
Okay? So, I'm just rep- depicting these
standalone embeddings using these uh
you know, thingies here.
Please appreciate them because it took
me a while to get them to do in
PowerPoint.
Okay? So, these are W1 through W6. These
are the vectors standing up. Okay?
Um now, let's say that So, we can easily
do that.
Now, what we want to figure out is we
want to focus on the word station.
And since station could mean very
different things in different contexts,
we want to figure out how do we actually
take
station's embedding and contextualize it
using all the other words that are going
on in that sentence.
Okay? Clearly, it's a train station.
So, we need to take the fact that there
is a train involved to to alter the
embedding of the word station. Right?
That's what taking context into account
actually means.
So,
how can we modify station's embedding so
that it incorporates all the other
words? That's the question.
Okay?
So, when you look at it this way,
imagine just for a moment,
just for a moment,
that
we
Now, some of the other words in the
sentence don't matter. The word the
probably doesn't matter.
But some of the other words like train,
slowly, left probably does matter.
And suppose, just magically, we have
been told
all the other words in the sentence,
this is how much weight you have to give
to them. These don't give it any weight.
Those give it a lot of weight. Okay?
Suppose we are told that.
Or to put it another way, and this this
is the word that's heavily used in the
literature,
someone tells you how much attention to
pay to the other words.
Whether you got to pay it a lot of
attention or very little attention.
Okay?
And this
how much attention to pay is given in
the form of a weight that you can use.
Okay? So,
um
if you look at it that way, from this
notion of which word should I give a lot
of weight to and very little weight to,
in this example, intuitively, which
words do you think should get the most
weight and which words do you think
should get the least weight?
Yeah. Train.
Train. Right.
Time matters.
Uh
you can do one at a time.
Train. Okay, thank you.
Uh
okay. Others?
Slowly.
Slowly. Right. So, that also seems to
have some bearing to it. What about
words that don't really I don't
we don't think is going to are going to
help at all?
The. The. Exactly. It probably doesn't
do much here. Some context it actually
might make a difference, but in this
sentence, maybe not.
Right? Intuitively.
So,
we should probably give a lot of weight
to train, maybe a little to slowly and
left, and hardly anything to the.
Okay?
And so, this intuition that we have
can be written numerically as maybe we
have a bunch of weights that add up to
one.
Okay?
Okay, maybe something like this. So, we
are saying the train 30% weightage,
maybe 8% weightage to left, maybe 12%
weightage to slowly, uh and then as you
will see here,
the station's own embedding also plays a
role. Because we want to take its own
standalone embedding and just move it
slightly, change it slightly, which
means that has to be the starting point.
So, it will get a lot of weight. We
can't ignore itself, in other words.
Right? So, we give it maybe 40% weight.
By the way, these numbers I just made
them up.
Okay? Uh yeah.
I'm sorry, it's a quick question. So,
the weights
are they
Are they Are they standalone for the
context of the entire sentence or are
they related to station that we started
off with? The The These six numbers are
only pertinent to station.
And for each word, we're going to do
something similar.
Yeah.
And at this point, does the model
understand order? Because like I'm just
thinking of like left because like I
gave it a very low
a
a very low weight. But let's say left
comes slowly, leave left station. The
station only have the two be higher.
Yeah, correct. So, at this point, we are
not worrying about order. We are only We
are worrying about context.
Later, we'll take order into account.
But how does the model know that left
here is of lesser importance because
it's a verb rather than a
It's It has to figure it out.
We don't It doesn't We We are just
giving it a whole bunch of capabilities.
How it manifests those capabilities is
all going to emerge from training.
Okay. So, all right. So, let's say we
have something like this. So, what we
can do,
right? And we'll get to the
all-important question of where do we
get these numbers from in just a moment.
But suppose you had the numbers,
how can we use these numbers to
contextualize W6? What can we do?
What is the simplest thing you can do?
You have W6, you want to make it a new
W6, which is now contextual, is aware of
what else is going on. Okay?
It's working now, I think.
We can take a weighted average. Exactly.
Exactly. So, when you have a bunch of
things and you have a bunch of weights
and I, you know, and we have when we
have to somehow modify one of those
things with those weights, the simplest
thing you can do is to take a weighted
average.
Right? So, that's exactly what we're
going to do.
So, we're going to take all these
weights
and just like move them up.
Okay?
Move them up.
Don't even get me started on how long it
took me to get this arrow to run.
I don't know about you, folks. Is it
It's extremely painful to get the U-turn
arrows to work in PowerPoint.
Okay?
Anyway, uh back to work. So,
so we just move these up here, okay? So,
now we can do 0.05 * this vector + 0.3 *
that vector and so on and so forth.
And the result is just another vector.
Right?
And that vector, folks,
is the contextual embedding vector of
station.
Okay? That was the standalone embedding.
And now we did the We multiplied this by
that that by whoop whoop whoop, add them
all up, and then you get a new vector.
And contextual embeddings have this
bluish kind of color.
Okay?
And I'll maintain that color scheme as
we go along.
So, that's it.
That's it. That's the idea.
Any questions?
Yeah.
How did you come up with the original
weights again? You just kind of guessed?
No, these weights I just I just
hand typed them in manually just to make
the point. And And now I'm going to talk
about how we are actually going to
calculate them.
Okay.
Uh all right, cool. So, now I'm going to
uh okay, enough pictures. Let's switch
to some math. So,
so basically what I'm So, let's write it
a bit more formally.
So, we have these W1 through W6, which
are the standalone embeddings.
And then for station, we want to
calculate, you know, W6 with a little
hat on it, which is the contextual
embedding. And the way we do it is to
say we calculate some weights for each
of these words. So, this weight S16
means that the weight
of the first word on the sixth word,
which happens to be station.
The The weight of the second word on the
sixth word, and so on and so forth. And
so, what we are saying is that W6 is
just, you know, this weight times W1,
this time W whoop whoop whoop,
that's it.
Okay?
I have to inflict all these, you know,
subscripts and all that because
you know, we need it.
All right. So, that's it.
That's what we have.
Now, let's talk about Okay, any
questions on the mechanics of it
before I get to Okay, where do these
weights come from?
Yeah.
Utilizing something like Google, for
example, like how does it understand
like the context of
new words
and context like
process immediately through the training
data the users played or
like basically
>> like a totally new word that didn't
exist before? A new word or a new
context to a word that already exists.
No, I think that the context is supplied
because the query coming into something
like Google is a full sentence.
And we only take that sentence and take
only the sentence into account as the
context for us.
So, the context is always present to us
when we get the input.
But the other question you had uh of
Okay, what if there's a brand new word
you've never seen before, for which
there is not even a standalone
embedding? What do you do then?
So, let's punt on that till Wednesday
because I have to talk about something
called byte pair encoding and stuff like
that before I can answer that.
And And really quickly, does that
immediately translate to their
predictive search queries?
Utilizing like verb
Yeah, a new word, for example.
Does that automatically get applied to
the predictive search queries like when
we're saying how to and then just home?
Oh, you mean like the auto complete?
You know, auto complete uses a slightly
different mechanism.
Um I They had a very complicated
non-transformer thing for a long time.
I'm sure they have a transformer version
now, but I don't I'm not privy to how
exactly they've done it. So, I don't
quite know how they do it. But what
you're proposing is a reasonable way to
think about it.
Yeah.
Um my question is like we have six
words, station and but number parameters
as in weights, let's say 10 of them.
And then we have calculated the
contextual version of W6. Yeah. So, this
has a different parameter or it remains
the same? It replaces. Okay.
Yeah, W becomes W6 becomes W6 hat.
Okay. And how we are expecting
Right.
This contextual word will be really
good. That's what we want.
Do we lose that
or retain No, we lose it. And as you
will see here, as it flows through the
transformer, it's getting more and more
and more contextualized.
So, it's a left-to-right flow.
All right. Uh all right, great. So, the
By the way, this thing that we did for
station, we will do it for each word in
the in the in the sentence.
The same exact logic. Obviously, the
weights are going to change.
Okay? But what will happen is that W1
through W6 will become W1 hat through W6
hat.
The same exact logic is going to hold.
Okay? That's what I just don't have the
slides for it because it's a waste of
time.
The same exact logic is going to hold.
All right. Now, switch gears
and and answer the all-important
question of where are the weights going
to come from.
Okay? So, the intuition here is really
really interesting and elegant.
So, clearly the weight of a word
should be proportional to how related it
is to the word station.
Right?
The word train clearly is very related
to the word station.
The word the is not clear how it's
related it is. Probably not all that
related. So, the relatedness matters to
the weight. More related, higher the
weight, right? Just intuitive.
So, one way to quantify how related two
words are is to take their standalone
embeddings and calculate the dot
product.
Okay? So, um
in case folks have
sort of forgotten about the dot product,
Oops, that's not what I want.
So, um So, if you have a Let's say you
have a vector.
Okay, let's Let's Let's say this is the
vector for
train.
This is the vector for station.
Okay? So, the dot product of these two
vectors,
I'll write it as train
station
equals
basically the length
of
the vector for train
times the length
of the vector for station
times the cosine
of the angle between them.
Okay?
So, how long is each vector?
Product of the two and then the angle
between them. Okay? Now, let's assume
for simplicity that these lengths are
roughly the same.
They're just one unit length. Okay? Just
roughly.
So, if you assume that,
okay? This thing, let's say, becomes
becomes one, let's say.
Okay?
This thing becomes one.
So, all the action
is here.
Okay?
So, all the action is here.
So, basically, the dot product of these
two vectors is really the cosine of
angle between them.
So, now, the question is, if you have
something like this,
right? Which are very close to each
other, the cosine of a very small angle,
actually, the cosine of zero is what?
One.
So, if the angle is really, really
small, the cosine is going to be very
close to one.
Right? Because zero is one. The cosine
of zero is one. So, this thing is going
to be, you know, pretty close to one.
If you have a cosine of two vectors that
are like this, 90° apart, what is the
cosine?
Zero. They're orthogonal, right? Which
maps to the English orthogonal.
So, the cosine of that is zero.
And then, if you have something like
this,
where they're literally pointing in
opposite direction,
what is the cosine of that 180?
Minus one.
So, that's it. So, the if these things
if these these these two vectors are
very close to each other,
the cosine of the angle between them is
going to be very close to one. If they
are really kind of unrelated, it's going
to be zero. If they're anti-related,
it's going to be minus one.
Right? So, that's how dot products
capture this notion of closeness or
relatedness.
Okay?
So, all right. Um iPad.
So, we can use the dot product of these
embeddings to capture relatedness.
And so, okay, iPad done.
So, now, what we do is we know now that
we know that dot products can be used,
we can't use them as is because we need
to do one more thing to make them proper
weights. And what I mean by proper
weights is that the we want the weights
to be, first of all, non-negative, and
we want to add up we want them to add up
to one, right? That's that's what a
weighted average actually is going to
mean.
But these cosines could be negative.
Right? And so, we need to now adjust
them to make them proper so that every
one of them is guaranteed to be
non-negative and they will add up to
one.
When was the last time you had to take a
bunch of numbers, which could be
anything, and then somehow make sure
that they are going to be positive,
non-negative, and they add up to one?
When was the last time?
Yeah, softmax. Exactly. So, we'll do the
same trick.
So, what we'll simply do is we'll just,
you know, exponentiate them, right? So,
like this W1 W6, this angle bracket
thing is the dot product. That's the
notation I'm using. EXP of that is just
you exponentiate them, e raised to that.
And once you exponentiate them, they all
become non-negative, and then we just
divide each by the sum of everything.
So, it the whole thing will become like
a probability, right? It'll just add up
to one.
Make sense? So, that's how we take
arbitrary numbers and make them proper
weights.
All right.
So,
to summarize,
from embeddings to contextual
embeddings, that's what we do.
We take all the stand-alone embeddings,
we calculate these weights using this
formula, and then we just do the
weighted average, and we arrive at the
contextual embedding, and boom, done.
Okay?
And so, by way choosing weights in this
manner, the embedding of a word gets
dragged closer to the embeddings of the
other words in proportion to how related
they are. So, just imagine for a second,
right? In this case, station obviously
has many contexts, but let's assume for
a second that only has the train context
and the radio station context.
Okay?
In the current context, train is closely
related to station, and therefore exerts
a strong pull on it.
Right?
Now, radio is also related to station,
but it doesn't appear in the word in the
sentence.
So, effectively, it has a weight of
zero.
Okay? And that's the beauty of it. And
And please do not ask me things like,
you know, I I was listening to a great
song on the radio station and the train
pulled out of the station.
Okay? Transformers can deal with stuff
like that. Okay? But yeah, but you get
the idea, the main idea.
So, by paying moving station closer to
train,
by paying more attention to train, we
are contextualizing the station the word
the embedding to the context of trains,
platforms, departures, tickets, and so
on. It's like this portal into the whole
train world.
Right? It's beautiful. This simple idea
will get you there.
Okay?
So, this, folks, is called
self-attention.
What we just described is called
self-attention.
And it's the key building block of
transformers.
Okay? Um and so, the the So, to
summarize, stand-alone embeddings come
in, contextual embeddings go out.
Any questions?
Uh yeah.
Uh I'm still struggling a little bit
with the intuition of the word
contextual embedding. So, like the
weight of station in the station
embedding, how how should I think about
that? It seems intuitive that it would
be high for all contextual embeddings,
but I assume that's not the case.
It'll be high. It'll be typically be a
high number because the cosine of the
the vector to itself is going to be very
cosine is going to be one, right? So,
it's going to be pretty high, but it
there's no guarantee it's going to be
the highest.
Right? Because they're not actually the
the length doesn't have to be one. They
could be We try to keep them kind of
smallish, but they don't have to be.
Uh so, the way I would think about it is
imagine that you take an average of
everything else first, and then you
average it with the new the old
embedding.
Effectively, it's the same as just
calculating the different weights and
averaging the whole thing together.
Sure.
So, why should you say that the
embedding of a word would be the same
number but same place? But is this the
reason why you need a contextual
embedding?
But even if it's like a
other word
and it's not related, that's what
I'm saying. Correct. Correct. Exactly.
Exactly. And the other thing to remember
is that by getting
by keeping the origin the input sort of
the size of the input cardinality intact
as you move through the transformer
stack,
when you finally come out the other end,
there is sort of no loss of information.
And in the very end, you can choose to
aggregate, simplify, summarize, and so
on and so forth. It preserves your
optionality as long as possible.
Do you know
how how long the embedding contextual
embedding is?
Is that a factor between the
two?
You know
Yeah, so, what we do is the the sentence
comes in. There's a whole notion of
something called a context window, or
what is the sort of the maximum length
that these sentences will handle, and
that's a parameter you can set. And
we'll come to that when you actually
look at the collab.
Um
Was that a question in the middle? No.
Okay.
All right. So, that is self-attention.
Um and now,
because that's felt too easy,
we're going to do a little tweak called
multi-head attention.
So,
this is this is the self-attention we
just saw.
What we can do is we can be like, you
know what?
Why can't we have more than this? Why
can't we have more than one of these?
So, this is called an attention head,
self-attention head. We'll have multiple
self-attention heads. Okay?
Now, and I'll come back to the top thing
in a second, okay? But So, the question
is, why should we have multiple
self-attention heads?
Because a particular attention head is
going to pick up some patterns. The
reason is because
it'll help us attend to the multiple
patterns that may be present in a single
sentence.
So far, when I've been explaining, uh
I've sort of basically been looking at
what the meaning of these words are.
Just the meaning of these words. But in
any complicated sentence, you have to
worry about grammar, you have to worry
about tense, you have to worry about
tone. You have to worry about facts
versus, you know, opinions. There could
be any number of complicated patterns
that are sitting in a simple sentence.
Which means, well, there is just not one
way to pay attention. There could be
many ways of paying attention, many sort
of There could be many needs to pay
attention. Right?
Which means that we'll let's have many
of these attention heads.
And each one could be learning something
else. It's exactly like having lots of
filters in a convolutional network.
Right? Uh one filter might learn a line,
another filter might learn a curve, and
so on and so forth. And we don't want to
decide a priori, oh, you're going to
learn a line, right? Similarly here,
we're not telling any of these things
what you have to learn. They just have
to learn based on the training process.
So, what we do is
So, actually, this is an example where
this is from the original transformer
paper, where this sentence is the lawyer
will Sorry, the law will never be
perfect, but its application should be
just. This is what we are missing, in my
opinion.
The complicated sentence, right? So, the
first one attention head, actually, this
is the pattern of things it's it's it's
So, for example, the word perfect here,
the contextual embedding of the word
perfect
draws upon heavily from the word law
in this example.
Okay?
If you look at another attention head,
the contextual embedding for the word
perfect is actually drawing heavily from
just perfect and nothing else. Right?
And if you look at other words, the
patterns are subtly different of what
it's paying attention to.
So, these are two different attention
heads, and they're learning different
kinds of attentions.
Okay? In reality, trying to make sense
of why they
pay attention to the way they do, it's
usually quite sort of difficult to
figure that out. You can't actually
interpret it. But when you have lots of
attention heads, the performance on the
task that you care about gets really
much better.
Right? And then you're saying, okay, I
can use that. Uh yeah.
That's the
I think that's the idea behind this. Is
that the idea behind this?
Right.
Exactly. Same logic. Same logic.
Yeah.
Actually in the convolutional case, the
ones and zeros I had were just example
numbers to show that that particular
filter could detect a vertical line or
horizontal line. You will recall that
when we actually train a convolutional
network, we actually don't specify the
numbers. We start with random
initialized weights and then we let back
back propagation figure it out.
Similarly here, we don't decide any of
these things. We just let back prop
figure it out.
Okay? And now the question of what are
the weights that are actually going to
be learned? We'll come come to that in a
bit.
Okay? Uh yeah.
Uh I was wondering how come we have
different attention head even though
uh it seems like they're only function
of a dot product and we have the same
dot product for same embeddings.
Great question. Great question. And I
literally have a a note in my slide
saying, "If a student asks this good
question, tell them to wait till
Wednesday."
So, great question. And we'll come back
to that uh on Wednesday and spend a fair
amount of time on it. So, uh
the the the point that's being made here
is that oops.
When we look at self-attention,
the embeddings came in and we did all
these dot products and the contextual
things popped out the other end. Note
that inside the self-attention box,
there are no parameters.
There are no parameters.
So, the question that is being raised
here is that so what are we learning
really? If there is nothing inside to be
learned, if there are no parameters, no
coefficients, what are we learning?
That's the question. And by extension,
if we have two of these and neither of
them is learning anything, what's the
point?
Sadly, you have to wait till Wednesday.
Okay? But we have a great answer to the
question. So,
it'll be worth it. And if you can't
stand the suspense, read the book.
All right. So, that is uh that's why we
need multiple heads. Okay? And now to
come back to this, so what we do is it
goes through this head and you get these
W's, right? And it goes through here and
we get the another set of W's.
Then what we do at the very end is we
concatenate them.
Okay? We concatenate them and we do a
projection. And this is what I mean by
that.
So, we have
uh this this is one self-attention head,
self-attention one.
This is self-attention two.
And let's say that
W1 hat comes out.
And I'm just going to call it Z Z1 for
the same thing so that there's no name
clash.
Okay? And uh the W2, W6, all of them are
coming, right? Let's focus on W1 and Z1.
W1 and Z1 are both contextual embeddings
for the same word.
Okay? For the first word, word one. And
so what we do is let's say this is W1 uh
let's call let's say this vector is like
this. Okay?
And let's say that this vector is like
this.
What I mean when I say concatenated here
is we literally take
um this word here,
this embedding here, then we take this
thing here.
Okay? And we just make it a long vector.
We concatenate it. But now this vector
has become twice as long, right?
So, what but remember, we always want to
preserve this the the number of inputs
we have and the lengths of these vectors
everywhere as we go along. So, what we
do is at this point, we run it through
a single dense layer
which will take this thing and make it
back into the same small shape as
before.
So, this is a dense layer.
That's it. So, this vector comes in
and it becomes it gets compressed back
to the original shape that came out of
here.
So, you could have like 20 of these uh
attention heads
and the concatenated will be 20 times
long and then just project boom, one
dense layer comes back to the original
shape.
So, that's that is the projection step.
And that's what I mean here when I say
concatenate and project.
So, at this point, what we have is
things come in, we contextualize them
using these different attention heads,
and when they come out of the attention
heads, we take them all, we just like
concatenate them, and then compress them
back to the same original starting
shape. Right? If these vectors are 100
units long or 100 dimension long,
whatever comes out is 100 still.
And to pre- preserving this
size as we go along is very important
for reasons that'll become apparent a
bit later.
Okay. So, that is the multi-attention
thing.
Now, a final tweak for today
is that we will inject some
non-linearity
with some dense layer dense ReLU layers
at the very end. So, we'd went through a
bunch of attention heads. We we came up
with a bunch of contextual embeddings
now.
So, at this point so far,
there are no since there are no
parameters inside these boxes,
uh
right? And there are some parameters
here.
We need to do some non-linearity. So
far, there's been nothing that's
non-linear so far. So, here we actually
send it through one or more ReLUs.
Typically, they just use one ReLU. So,
and what I mean by that
Sorry.
So, this is what we had here and then
we take it in
and then run it through
actually
we typically run it through
a ReLU.
This is a nice ReLU.
Okay? And all and and the rule of thumb,
as you will see, if let's say this
vector is say 100 dimensions long, they
typically will choose a ReLU which is
about 400
wide. And then it just gets projected
out again back to 100.
So,
this is just a simple, you know, the
input comes in, goes through a single
hidden layer with four four times as
many as here, and then it
project another dense layer
to 100 again.
And this since there are ReLUs here,
we in- we have injected some
non-linearity into the processing.
Okay? Now,
a lot of this stuff when it came out
felt very ad hoc.
Right? It didn't come from some deep,
you know, theoretical motivations.
But and people had strong intuitions as
to why these things were helpful. And as
it turns out, since the transformer came
out, people have tried to optimize every
aspect of this thing.
It's actually pretty difficult to beat
the starting architecture.
Right? Improvements have been made, but
it's actually very robust architecture.
So,
so that's what's going on here. And then
when we come out of this thing,
this is what we have, the story so far.
We start with random standalone
embeddings. This could be
GloVe embeddings, it could be random
weights, doesn't matter. It goes through
a bunch of self-attention heads. We
concatenate it when it comes out the
other end.
Concatenate it when it comes out the
other end. And then we project it back
to the same size as before. Then we run
it through, you know, a ReLU followed by
a linear layer and we get these things
again. So, in this whole process, if six
things came in, six things will come
out. And if six and if those six things
that came in
were embedding standalone embedding
vectors of 100 dimensions, what comes
out is also 100 dimensions.
So, in that sense, you could think of
this whole thing as a black box in which
whatever you send in, the same number of
things will come out of the same length.
The numbers will be different because
they will have been heavily
contextualized.
The numbers are much smarter, in other
words.
So, so far what we have seen is that we
have satisfied two of the three
requirements. We have taken the context
of each word into account
by using these dot products in the
self-attention layer, and we can
generate an output that is the same
length as the input, but we have ignored
the fact that we have ignored word order
completely.
Okay? Because whether I had said the
train slowly left the station or I had
said the the station slowly left the
train,
this thing won't know the difference.
Because dot products
function on sets, not on sequences. They
function on sets.
Okay? Regard- You can you should
convince yourself of this. Regardless of
the order, the dot product calculation
doesn't change anything.
Because we are doing every pair.
Okay? So, the question is how do we take
the order of the words into account? Um
right. As I was saying, we can scramble
the order of the words in a sentence and
we'll get the exact same contextual
embeddings at the end.
So, by the way, if you're working on a
problem in which the order doesn't
matter,
then you can stop right now and use the
transformer.
And there are many problems that are
actually in that category where the
order doesn't matter. So, if you take
traditional structured data, right? Uh
tabular data,
uh you know, blood pressure, cholesterol
level, boom boom boom. Does it predict
heart disease? Well, there is no order
in that thing. You can use the
transformer as is without doing anything
more.
So, transformers work for both sets and
sequences where order matters.
Okay. So, the fix for this is something
called the positional encoding.
Um
so what we do is very simple. There are
By By there are many things that been
invented um to to to tell transformers
to give an transformer some information
about the order of each of the things
that are coming in.
I'm going to go with something called
the, you know,
the simplest possible way which actually
works pretty well in practice. So, what
we do is
for each position
each possible position in the input
starting from the first position all the
way through the last position
we imagine that that position itself is
a categorical variable.
Right? If a sentence can only be 30 30
words long, let's say, we say that hey,
the position of each word is a number
between 0 and 29.
And so, we can just think of it as a
categorical variable.
And because the categorical variable, we
can just imagine an embedding for that
for each potential value. So, it'll
become clear in just a moment because I
have a numerical example.
And so, what we do is we will just take
that standalone embedding and then we'll
take this position embedding
which represents the position of the
word in the sentence, we just add them
up.
Okay? Uh yeah.
So, if
in the initial sentence itself, I have a
mistake, so I just write it as the train
slowly the station.
So, which means my output is actually
going to be wrong. Yes.
Now, the transformers are since they're
trained on lots of data,
they will be quite robust to these
things.
But strictly arithmetically speaking
correct, yes.
Um okay. So, here's let's look at an
example.
Let's assume that
um
your standalone embeddings, right? This
is your vocabulary, okay?
Unknown, cat, mat, I, sit, love, the,
you, on. That's it. That's our
vocabulary.
And for this vocabulary, we have these
standalone embeddings.
And just for argument, let's assume
these embeddings are only two long.
Okay? The dimension of these embeddings
is two.
If you recall the glove embeddings we
used last week, I think they were what?
100 long?
And the ones we're using in the homework
are even longer than that.
Um but here we are assuming they're only
two long, okay? So, the embedding for
cat is 0.5, {comma} 7.1.
All right. Now, let's assume that the we
can have at most 10 words in any
sentence that's coming in.
And obviously, a particular word could
be in position 0 all the way through
position 9.
And we will learn embeddings for each of
these positions, and these embeddings
are also two long.
Two units long. Dimension two.
Okay?
Now, where will these embeddings come
from?
What's the answer to that question? What
is the answer to the general question of
where will these weights come from?
We will learn it with backprop.
Okay?
We will start initially with random
numbers and then we'll get them make
them better and better
as over the course of training.
So, what we do is we have these two
tables
of embeddings.
Um the standalone embedding for the word
and the position embedding.
And then, we literally add them up.
So, for example, let's say the word the
sentence that came in is cat sat mat.
That's the sentence. It's got three
words, cat sat mat. So, what we do is we
say, well, the embedding for cat is this
thing here, 0.571.
So, I write it here, 0.571.
Cat happens to be the zeroth position of
the word.
So, I grab the embedding for zero, which
is 1.3, 3.9. I stick it there, and then
I literally add them up. 0.5 + 1.3, 1.8.
11.0. That's it.
So, now the positional encoded embedding
for the word cat is 1.8, 11.0, not 0.5,
7.1.
So, if cat happens to show up in another
part of the sentence, let's say instead
of cat sat mat, we had
mat sat cat.
Now, cat is now the third position,
right? Which is 0, 1, and 2. Which means
its embedding doesn't change. It's just
the embedding for cat, but now instead
of picking zero, we'll pick this one,
0.6, 8.1, and put that here and add them
up instead.
So, this is the idea of the positional
encoding.
This is how we inject position knowledge
into the transformer.
Yes.
Um
the positional embedding would be
different for each sentence, right? How
do you No, this is just one table which
tells you what the position is.
So, the it says for a word that appears
in the seventh position in any input
sentence that you're feeding in,
this is the embedding that you need to
use
for that position.
If the word appears twice in the same
sentence, how do
Great question. So, if if let's say just
for argument, let's say the word the the
sentence was cat cat cat.
So, the
for each one of those cat for cat cat
cat,
the this embedding will be the same,
0.571, because that is happens to be
just the embedding for cat regardless of
position.
But then, the first cat
for the first cat, we will use 1.3, 3.9
as the addition. For the second cat,
we'll use 6.3, 3.7. The third cat will
use 0.6, 8.1.
So, only the things that are adding the
position encoding will change, the
positional embedding. So, the resulting
sum is going to be different for each of
these three words, even though they're
exactly the same word.
Is that position embedding table
specific to the standalone embedding
table? Like if you were to add or remove
some words from the standalone It's
independent.
Independent. It only depends on your
assumption about how long the sentences
can be.
That's it.
It doesn't really care about what's what
words are coming in. That's a whole
different thing.
So, these are two independent tables
that just learned as part of this
process.
So, yeah, I have the same thing for sat
and mat.
Sat and mat, that's what we have.
So, just make sure you understand these
two slides to really like make sure the
mechanics are clear. Yeah.
How do you control for filler words? For
example, if you're taking
NLP output for transcription and you're
trying to run a transformer and you have
a lot of
um's and likes that are
disproportionately large and have these
random assignments or
really deep embeddings, is there other
ways to look at through the noise?
Typically, what they do is um
as we will we'll talk about this thing
called byte pair encoding in which we
take individual characters,
fragments of words, and words into
account as tokens. So, when you hear
stuff like uh and so on, it gets mapped
to these small tokens.
Right? And then we treat them as just
any other token.
Um yeah, is aggregation like a simple
sum where here and is the actual
semantic meaning of the word standalone
not be more important than its
relative position in the sentence?
It could be. We just don't know a priori
whether it's going to be important or
not for any particular sentence.
We when we train the transformer with a
lot of textual data,
right? It'll just figure out the right
values for these things so that on
average, the accuracy is as high as
possible.
So, in many of these things, there's
always a tension between our human
intuition as to how it should work and
whether you should just throw it into
the meat grinder of backprop and see
what happens.
And so, here it does it turns out you
can just throw it into backprop, it'll
actually do a pretty good job.
Uh yeah.
For the positional encoding, we would
just be as using the sum vector, we
would be using like this 2 by 3 matrix
that you have for our right?
Uh oh yeah, this is just for
demonstration. Basically, this is the
thing that will actually go into the
transformer. Correct.
Yeah.
That was just me being overly verbose in
the slides.
Uh yeah.
I can see sentences in the input. At
this point, are we still parsing out
punctuation or if we have like a
multi-sentence input, is there a
positional embedding vector for each of
the sentences? Yeah, so here um
basically, the starting point is tokens.
Right? And in our example, because we're
working with the idea of simple
standardization and stripping and things
like that, I'm just showing actual
words.
If you go to something like GPT-4, since
it uses a different tokenization scheme,
uh each token might be part of a word.
It might be it might be an individual
character, it might be a punctuation
mark, it could be in fact um the GPT
family doesn't strip out punctuation.
Which is why when you ask a question, it
comes back with intact punctuation in
its response.
Uh and so, we'll get we'll revisit this
when you look at BPE, byte pair encoding
later on.
But the key thing to remember is that
all the stuff we're talking about starts
from the notion of a token.
As to how you define a token given a
bunch of text, that's the tokenizer's
job. And we just assumed a simple
tokenizer for the time being.
Okay? So, at this point, folks, we have
satisfied all the requirements.
Uh we have taken the surrounding context
of each word, we have taken the order,
and so on and so forth, because what's
coming in here is the positional
embeddings. Okay? And it runs through
the whole transformer stack.
So,
this is called a transformer encoder.
Okay?
This is the transformer encoder.
And you can see here, this is the
original picture from the paper.
It's an iconic picture at this point.
So, it says here this is these are the
input This is like the cat sat on the
mat.
It comes in here, gets transferred to
transformed into embeddings, standalone
embeddings.
And then, based on the position of each
word, we add that's why you see a plus
sign here, we add the positional
embedding to that.
And the resulting thing goes into this
transformer block. And here,
we go through multi-head attention.
And things come out the other end.
Then there is this thing called add and
norm, which we'll visit we'll revisit on
Wednesday.
And then it goes through a feed forward
network, another add and norm, which
we'll revisit on Wednesday.
And then it comes out the other end.
That's it. That's a transformer encoder.
Okay?
Um
and so if you look at this
just to point out a couple of things,
the input embeddings can be random
weights or it could be pre-trained
embeddings.
Um
we add in a position-dependent embedding
to represent the position of each word
in the sentence. That's the plus.
Then we pass it through multi-headed
attention to get a contextual uh
representation.
Then we finally we pass all this through
a simple
typically it's a two-layer network. A
one hidden layer with relus and then a
linear layer after that and boom. Uh and
then we do it. This is the encoder. And
here is the perhaps the most important
point to keep in mind.
Because we have taken inordinate care to
make sure that the things that are
coming in and the things that are going
out have the same size
both in terms of the number of tokens as
well as the length of each vector.
We can then stack them up like pancakes.
We can have lots of transformers stacked
one on top of each other.
Right? Because it's the perfect API.
It's the simplest possible API. The same
thing comes in, same thing goes out.
In terms of size. So you can have a
transformer encoder, another one top,
boom, boom, boom, boom, boom, one after
the other. GPT-3 has 96 transformer
stacks.
And like in all things deep learning
related, the more layers you have, the
more complicated things we can do with
it.
As long as you have enough data to keep
the model happy so it doesn't overfit.
Okay?
All right. So, what we haven't covered,
which we'll cover on Wednesday
uh is is the question that
he had posed about how
uh you know, since there are no
parameters inside the self-attention
block, what are we actually learning?
And then there is these things called
residual connections and layer
normalization. We'll talk about all
those things on Wednesday. Those are all
like, you know, refinements to the idea.
So, all right, 9:39. Um let's apply the
transformer encoder to an actual
problem.
Any questions?
Uh yeah.
My question is regarding like you said
you could have multiple transformers.
What is the difference with having
multiple self-attention heads uh and
rather than that having multiple When I
say a transformer block within the block
there could be multiple heads. So, if
you're if the accuracy is the same, why
would you use this rather
Yeah, you can have a lot of attention
heads. And that's totally fine. And
typically I forget how many GPT-3 and 4
have. They have a whole bunch of them.
But you can So you can go wide and you
can go deep.
Both are done in practice.
But the thing is if
The one thing you have to remember is
that if you if you go wide, you have a
lot of attention heads then given the
particular input that's coming into that
block, it'll learn different patterns
from it.
While if you stack them all up, it's
going to learn different ways to
contextualize the things that are coming
in. It operates at higher levels of
abstraction. So the analogy would be
that like the seventh layer of a
convolutional net may take the sixth
layer's output and say, "Oh, I'm seeing
a lot of edges here. I'm going to take
an edge like this, two circles like that
and call it a face."
So it'll operate at a higher level of
abstraction.
Okay.
Um
All right, let's go to the collab.
So what we're going to do is we're going
to take the transformer that we just
learned about and we're going to apply
it to solve the the travel uh slot
problem. Okay?
Uh all right. So
Okay, so we'll start with the usual
preliminaries.
And then we have taken the ATIS data set
I talked about and we have stuck them in
raw box for easy consumption.
It's here.
Okay.
So if you look at to the top view
you can see here, for example, I want to
fly from Boston 8:30 a.m. And then this
is the output. The slot filling is the
output. Um and so as it turns out here
there is
this these people also gave it a another
They took the whole query and gave it an
intent as to is it it's a flight query,
it's a something else query and so on,
which we're not going to use. Are you
kidding me?
I want to fly from Boston at 8:30 a.m.
and arrive in Denver at 11:00 in the
morning. What kind of ground
transportations are available in Denver?
What's the airport at Orlando?
Um how much does the limo service cost
within Pittsburgh? Okay.
And so on and so forth. So you get So
you get the idea. It's a very wide range
of queries that are in this data set.
Um okay. So let's just ignore that for a
sec. Um okay. So what we're now going to
do is we are going to take only
um this column, right? The query column.
That's going to be our input text. Okay?
And then the slot filling column is
going to be our dependent variable, the
output.
So we'll just gather them all up
uh here.
Let it run. We'll do it for the training
data and the test data.
And so what we have done is that we have
taken um the transformer related code in
Keras and we have packaged it into a
little hardel library for easy
consumption.
Um and so that thing is here. You can
download it.
Calling it a library is like overstating
it. We literally just collected a bunch
of code and stuck it in a file. Okay?
So
and so what we'll do is from hardel
we'll we'll import the transformer
encoder.
And we'll import this positional
embedding layer.
Because what we're going to do is we are
going to take the input do the
positional encoding business and then
send it into the transformer.
Okay?
Um so but first let's vectorize the
input uh queries that are coming in.
So we'll define a thing here.
The use this uh
max query length is not defined. That's
what happens when you
don't run everything.
All right.
Okay. So now we have this thing here. So
turns out that there are 8,888 tokens,
right? 8,888 words in the input queries
that are we have in the data. Uh so I
take a look at the first few.
And you can see here, you know, there is
unk. Uh and because the output mode here
is you just want integers to come out
not multi-hot encoding or anything
because we're going to take these
integers and then do embeddings from
them. So it'll it'll create it'll
reserve this empty string as the pad
token. This should be familiar from last
week.
And then the unk for unknown tokens and
then two from flights these are all some
of the most frequent. Um turns out
Boston is actually the most frequent. I
don't know what's up with that.
It is what it is. Then we'll do the same
vectorization to the train and test data
sets.
Now uh we need to do STIE for the output
side of the problem because the slots
the the dependent variable here,
remember, are all sentences as well with
the B, O, things like that, right? So we
need to vectorize those.
So we do or we need to do STIE on them.
So let's take a look at some of these
slots.
And you can see here all this stuff is
going on.
Note So now here is an example where you
have to be very careful when you do the
standardization.
Typically standardization you will
remove punctuation and you know, do
things like that and lowercase, right?
But here
these things have a specific meaning.
We can't just go in there and remove the
period and the underscore and then take
make the B into lowercase B and stuff
like that. That'll just harm it.
Right? We need to be able to preserve
the nomenclature of the output in terms
of all those tags. So
um so we don't want the standardization
to do all those out. So what we do is we
say standardization none.
Look at that.
We tell Keras do not standardize this.
Do not do your usual thing.
Okay?
Um so
we do that
for the output side. And then let's look
at the vocabulary.
Yeah, so this sounds pretty good.
These are all the things that we would
expect to see.
These are the distinct tokens in the
output strings.
Um all right.
Okay, we get it.
So we have 125 of them. In the in the
lecture I said there are 123 slots,
possible slots. Why is it 125 here?
Yes, unk and pad. Correct.
Um okay. Now we'll set up a transformer
encoder, right? Uh this Oh, wait, wait,
wait. I forgot about um doing this. My
bad. Um
All right.
I just thought when I saw the slide that
we should go to the collab
without giving you a bit more
background. No problem. So
So
the way we're going to model this
problem is that we're going to have
something like this, right? Fly from
Boston to Denver.
That's the input that's coming in and
that is the correct answer.
0 0 some B something or others I mean O
and then something else, right? That's
the correct answer. That's the that's
the input and that is the right answer.
So what we'll do is we will
create these positional input embeddings
like we have discussed before.
We will run it through a transformer.
It gives us contextual embeddings.
So if we send five in, it's going to
send us five out except the color is now
blue.
Right? And then what we do is
we will run it through a relu.
Okay, we'll run it through a relu.
We will still have
you know, five vectors here, five
vectors will come in.
And then for each of the things that
comes in, we will stick a 123-way
softmax.
Okay, for each thing that comes out
we'll have a 123-way softmax and that's
the classification problem we're going
to solve.
Okay?
So
the weights in all these layers will get
optimized by backprop.
All these weights are going to get
optimized.
Uh yeah.
Sorry?
Oh no, the that's a layer. The weights
in the layer will still need to be
learned.
It's sort of like the text vectorization
layer is a bunch of code and then you
actually run it on a particular corpus
to adapt it and fill our vocabulary out
of it.
So, it's like an empty shell that needs
to get populated.
Okay, so with the weights and all these
things are going to get updated when we
when we train the model
by backprop.
Uh and that's it. That's the setup.
Does this make sense before I switch
back to the collab?
In particular, does this make sense?
This part of it.
Bunch of things come out and then for
each one of those things we need to
figure out a classification of a 123-way
classification. And that's where we
stick a softmax on every one of those
output nodes.
Yeah.
Oh oh, I see.
Yeah, so
It could be whatever or to put it
another way, it is your choice as the
user as the modeler. Correct? The thing
is at this point with the blue stuff the
transformer is basically saying, my job
is done.
It has given you these valuable
contextual embeddings at some high-level
abstraction. What you do with it depends
on your particular problem. And so that
the best practice would be to take it
and then maybe, you know, if these
embeddings are embeddings are really
long, maybe you make them a little
smaller, right? Using a ReLU. And using
a ReLU is always a good idea because
when in doubt, throw in a bit of
non-linearity.
Right? Uh and then once you're done with
that, well, at this point you need to
actually classify it. So, you stick an
output softmax on it.
Okay. So, that's what we have.
Um
All right, back to this picture.
So, what we're going to do is we
we also get to decide how long are these
embedding vectors. How long because here
we're not going to use Glove embeddings.
We're just going to learn everything
from scratch.
Right? We're going to learn everything
from scratch. So, and we can decide how
long these embedding vectors are. So, um
these embedding vectors I'm going to
decide
uh I have decided that I want them to be
512 long, right? I want these actually
to be 512 long. So, that's what I have
here, 512.
And then inside the transformer,
remember
when we
concatenate everything and then we have
something, we run it through a final
ReLU layer, how big should that layer
be?
That's what it here what I mean by dense
dim. I want it to be 64.
And then I, you know, for fun I'm going
to use five attention heads.
Because why not?
Okay. And then in the final thing here
to go to Ali's question here these
things are all 512 long as I mentioned
earlier, right? These are all 512.
But this thing here I'm going to make it
just 128.
Okay, that's what I mean by units here.
And so if you look at the actual model
okay, whatever comes in has a max query
length of I think 30 if I recall.
Um actually let's just make sure of
that. What did I assume?
30, correct? Max query length 30. So,
each sentence is 30. So, if a sentence
has 35 words in it, what's going to
happen?
The last five will get chopped,
truncated. If it comes in at 22, we're
going to pad it with eight more tokens
with a pad token. Okay? That's how we
make sure everything uh gets to 30.
All right. So, we come back here.
So, the input is still sentences which
are 30 long, tokens which are 30 long.
And then we run it through a positional
embedding layer.
Okay? This positional embedding layer
has the the actual embedding for each
word, that table and it has the
positional table, positional embedding
table. So, just to be clear, this
positional embedding layer is basically
it's basically this.
So, this table
and this table together are packaged up
into the positional encoding layer.
But they are two distinct tables. They
just happen to be packaged up.
So,
so this is what we have here.
And then we get a nice positional
embedding out and then boom, we run it
through the transformer. And you know,
this transformer encoder object we have
to tell it obviously, hey, this is the
embedding dimension that's going to come
out. This is the dense dimension you're
going to use in that final feedforward
layer inside each attention block and
this is the number of attention heads I
want you to use. That's it.
Very, right? Only three things have to
be specified.
And then whatever comes out of the
transformer encoder are these blue
vectors.
And then we are back into good old sort
of, you know, traditional DNN stuff
where we take this thing, run it through
a ReLU with 128 units, we add a little
dropout uh and then we run it through a
dense layer which the the vocab size
here is 125, which is the 125-way
softmax.
Okay? Activation softmax.
Connect up everything into model input
and output and boom, that's the whole
model.
So, that's what we have here.
Okay?
Now,
this for the you know, after Wednesday's
class
for extra credit and for your personal
edification
try to work through this thing to come
up with this number.
53 million
um sorry, 5.3 million.
Right? Uh and see if it matches this
number here.
It should match.
Hand calculate the number of parameters
inside the transformer. Okay? For fame
and fortune. That's an optional thing.
So,
uh do it after Wednesday's class, not
right now.
And I have actually listed the exact
math that goes into it here. Okay? All
right. So, by the way, you can peek into
any layers' weights using its weight
attribute. This is the embedding
uh the positional embedding thing we
had. So,
we can click it and you can see here it
has two tables. There's the first table
which is just the embedding table which
says
there are eight eight eight tokens in my
vocabulary and each of those tokens is a
an embedding vector which is 512 long.
That is the first table here. And then
it has the second object which is the
positional embedding and it says here,
well, my sentences can be 30 long and
for each position of the 30 long
sentence, I will have a 512 embedding.
Both these tables as I mentioned earlier
are packaged up inside and you can
actually see what the weights are before
you do any training.
Okay?
So, all right. So, I'm going to stop
here uh because the model it's going to
take a few minutes to run and we're
already at 5 9:45.
Um so, we will continue the journey on
Wednesday. If some of it is not super
clear, don't worry about it. It will
become much clearer on Wednesday. All
right? All right, folks, have a good
couple of days. I'll see you on
Wednesday.