We'll continue our journey with
natural language processing.
We looked at the bag of words model,
one-hot embeddings, and so on and so
forth. And today we will talk about
embeddings, or to be more precise,
stand-alone embeddings, and then that
will tee us up for something called
contextual embeddings, which is where
the transformer really sort of comes
into play.
All right, so let's get going. So so far
we have encoded input text
one-hot vector. So to just to refresh
your memories from Monday,
if you know, if this is the phrase
that's coming into the system, we run it
through the STIE process. And when we do
that, what happens is that first of all,
we you know, we standardize, then we
split on white space to get individual
words, then we assign words to integers,
and then we take you know, each integer
and essentially create a one-hot version
of that integer. And when we do that,
basically we have a vocabulary.
Right? And in this example, we just have
100 words, and you will note that this
vocabulary, which are which you arrive
at once you standardize and tokenize,
you know, has words like the because we
decided not to remove stop words like A,
and the,
and so on. So just to be clear,
standardization
here, standardization, while it has
historically been all about stripping
punctuation, lowercasing everything,
removing stop words, and stemming,
while that has been true historically,
if you look at modern practice, people
essentially strip punctuation maybe, and
then lowercase and and they often don't
even bother to do stemming and things
like that, or to remove stop words.
Okay?
And that's why in Keras, the default
standardization is only lowercasing and
punctuation stripping.
This detail may actually be handy for
homework two, perhaps. That's why I'm
pointing it out.
Okay. So that's what we have. And so for
each word that's coming in, we have a
one-hot vector.
Right? But the one-hot vector is just
like on to the vocabulary. And then, you
know, and we can either
quote unquote add them up and get a
multi-hot encoding, or
sorry, get a count encoding, or we can
just do or, right? Look for just any
ones in a column and get multi-hot
encoding.
So that's what we saw last class. But
this scheme, while it's quite effective
for simple kind of problems,
is it has some very serious
shortcomings. And so we will sort of
delve into those shortcomings, and then
sort of step back and say, all right, is
there a solution to fix these things?
Problem with one-hot vectors.
There are lots of problems. Any
volunteers?
Similar words are understood
differently.
Absolutely. So that what he's pointing
out is that if you have two words which
are synonyms, let's say, great and
awesome,
hope that the way we represent them
using these vectors would have some
connection to what the words actually
mean. In particular, we would hope that
if they mean similar things, that they
are sort of close by. If they mean very
different things, we would hope that
they are very far away. Right? Things
like that. Sort of common sensical
expectations of what you want the
vectors to have. So it clearly it won't
have that, and we'll look into it in a
detail in a bit. But before we do that,
there is also a computational issue,
which we covered last class, which is
that if the vocabulary is really long,
then each token, each word that's coming
in here, will have a one-hot vector
that's as long as the size of
vocabulary. Right? If you have 500,000
words in your vocabulary, every little
word that comes in has a vector which is
500,000 long. Which feels like a gross
sort of waste of it stuff.
Now you can mitigate it somewhat by
choosing only the most frequent words,
but it does increase the number of ways
the model has to learn, and increase the
need for compute and data, and so on and
so forth. Okay?
Now
let's say that we have created a
vocabulary from a training corpus. Okay?
We have a bunch of
strings, text that's coming in. We have
done it We have done the ST the
standardization and organization. We
have created a vocabulary from it. And
let's say we get the words movie and
film.
So the question is, and and I always
observation gets to this immediately, if
you look at the words movie and film,
are these two vectors close to each
other or not? Okay? So if you have two
vectors, how would we measure closeness?
What's the simplest way to think about
closeness?
It's not a trick question.
Distance. Yeah, exactly. So if they are
really close distance-wise, we would
hope, right? The words similar words
should do should should be close by. So
here, if you let's just imagine that the
vector for movie,
let's say your vocabulary is, I don't
know,
um
100,000 long.
So your vector is 100,000 long,
and the word for movie
is the position, so this this has a one,
everything else is zero. Right?
Sorry, this is the vector for film, and
maybe this is the position for film.
So that has a one, everything else here
zero. Okay? What's the distance between
these two vectors?
You just use the Euclidean distance. So
the Euclidean distance, you will recall,
you literally just take the difference
of
these values,
square them, add them up, take square
root.
So which means that all the zeros will
obviously give you zero. This one is
going to give you a one.
This comparison is going to give you
another one. 1 + 1 = 2. Root 2. That's
the answer.
So the distance between these two
vectors is root 2.
Now,
so the distance between them is root 2.
What about the one-hot encoded vectors
for good and bad? Clearly good and bad
mean opposite things.
What is the distance between the good
and bad 01 vectors?
Still root 2.
Because the zeros don't mean anything,
the ones are not in the same place.
So when you subtract the one and the
zero, you'll get ones and ones, add them
up, two, root 2.
In fact, you take any two words in your
vocabulary, what's the distance between
the two one-hot vectors for those words?
It's root 2.
So if any two words have the same
distance, does this even have a notion
of distance?
It doesn't.
There's no notion of distance from
one-hot vectors.
It has no connection to the actual
meanings of these words.
It's just a way of representing them.
Okay?
So that is the big problem with one-hot
vectors.
So
the distance between them is the same
regardless of the words. It's got
nothing to do with the meaning of the
words.
And this is a huge problem, which we'll
have to solve.
So to summarize where we are, if the
vocabulary is very long, each token will
have a one-hot vector that's long as
vocabulary. That's that's sort of a
computational and sort of training
problem. And then this is a deeper
problem, where there's no connection
between the meaning of a word and its
vector.
So wouldn't it be nice if
vectors that represent synonyms,
movie and film, apple, banana,
hopefully they're close to each other.
It would be nice if the vectors for
things that mean very different things
are far from each other.
So let's take a look at a particular
example. Okay? Let's assume that we have
been magically given
these vectors, so that they actually
have some notion of meaning.
And for convenience, let's say that we
take the just the first uh
two dimensions of these vectors, the
first two dimensions, so that we can do
a scatter plot on them.
So we plot the first dimension of the of
these vectors, the second dimension, and
what we have in this little cartoon is
we have plotted the the word for
factory, uh for home, for building, and
they all happen to be clustered here.
Clearly this representation is capturing
some notion of what the thing is.
Right? Some sort of building.
Uh and here we have, you know, bicycle,
truck, and car. Clearly some This is
like the automobile cluster, right?
Transportation cluster. And here we have
like a fruit cluster, and here we have
some, you know, sports balls cluster.
Okay?
We Because it's a cartoon, things are
all nice and cleanly separated. Okay? So
now if you take the word apple, where do
you think it's going to go?
It's going to go in into A, C, D, or B?
C, right? It makes eminent sense it's
going to go to C.
Good. Now,
wouldn't it be nice if
in more generally, if the geometric
relationship between word vectors
represent the semantic relationship
between the underlying objects that the
words represent?
Okay?
And it's And I say relationship and not
distance, because it's not just
distance. It's actually more than that.
Okay?
So let's take another one.
Here we have
uh this is the the vector plotted for
puppy and dog,
and this is calf.
Uh right? We have plotted the word for
calf. And let's say that we need to
figure out where would the embedding,
the word vector for cow appear?
It is the most logical. Should it be A?
Should it be C? Should it be B? Where
should it be?
This is
C? Okay, what's the logic?
Any volunteers? Just put your hand up.
Uh, yes.
Uh
A calf is a baby bull, whereas the cow
is an adult.
So, it should be closer to the dog,
which is the adult version of a dog.
Got it. So, you're basically saying go
from the puppy version to the grown-up
version. Right? That's sort of what
you're getting at, right? And that's a
totally valid way to think about it.
But there are a couple of ways to think
about this, which is this is one of the
those two ways. So, what you can do is
you can actually look at it and say,
well,
Okay, if this is big bringing you, you
know, bad memories of GMAT and GRE and
stuff like that, I apologize.
But
So, a puppy is to a dog like a calf is
to a cow, right? Which means that that's
exactly what Jay is pointing out. You
can go from like the baby version to the
full-grown version if you go in the
horizontal direction. Okay? But maybe if
you go in the vertical direction, you're
essentially going up and down the young
entities of animals.
Okay?
So, here you are growing with, you know,
you're still across the same dimension
of animals. You're just going from, you
know, the the same age level, right?
That is the band here.
So, this is the grown-up version of a
whole bunch of animals, the puppy
version of a whole bunch of animals. So,
the vertical dimension measures some
sort of variation across animal species
of the same roughly sort of maturity
stage.
Okay? So, these directions also matter.
It's not just the distance.
Okay. That's what I mean when I say
semantic relationship and geometric
relationship.
Relationship is distance and direction,
right? Both have to be involved.
So, so
Uh, now word embeddings, as we will dis-
learn soon, are word vectors designed to
achieve exactly these requirements.
Okay? They will achieve these
requirements.
Uh, and they will fix both these
problems very elegantly.
Okay?
So, let's say that we have word
embeddings that achieve both these
problems. Are we basically done?
Can we declare victory?
Or are there any- is there anything that
even words which actually capture the
meaning of the underlying thing
don't fully address? Is there any
remaining problem we have to worry
about? Yes?
Context. Context? Yes.
Context, right? What about The fact is a
word's meaning Sure, every word has a
meaning, but we know that some words
have multiple meanings.
And that meaning is really sort of
inferencable, or you can make sense of
it only if you know the surrounding
context, right? If I give you if if you
see the word bank, b a n k, bank,
sure, it could be a financial
institution. It could be the side of a
river. It could be the act of a plane
turning in one direction.
It could be someone hoping for
something, banking on something. The
list of possible meanings of the word
bank is basically enormous.
And you cannot figure out what it means
unless you know what else is going on
around that word. So, context is super
super important. And these embeddings,
word embeddings, just tell you what the
meaning of the word is. And basically
what's going to happen when you have a
word which could mean many different
things, it's going to give you some
average version of that meaning.
And that average version is not going to
be very good.
Now, there are some words which only
mean one thing, and you'll be okay
there.
But for the rest of it, right? It's
going to be tough.
So, what we need is some way
We need to find a way to make word
embeddings contextual.
Meaning we need to somehow consider the
other words in the sentence.
Okay? So, if we can do that, then we
will be in great shape.
Solve all sorts of NLP problems.
Now, as it turns out, contextual word
embeddings, or word vectors, or word
embeddings that achieve both these
requirements.
They capture the semantic geometric
relationship thing I talked about, and
they are contextual.
Okay?
They're really fantastic. Uh, and the
key to calculating contextual word
embeddings is the transformer.
That is why transformers are justifiably
famous.
So, what's sort of the the lay of the
land here? So, today we are going to
look at how to calculate
stand-alone or uncontextual word
embeddings.
And then starting Monday, we will take
these, you know, un- stand-alone
embeddings and make them contextual
using transformers. Okay? That is the
plan.
Any questions so far?
So, now let's think about how we can
learn these stand-alone embeddings from
data, right? Now, the naive way to think
about it would be, hey, let's Why don't
we manually collect a whole bunch of
synonyms, antonyms, related words, etc.,
and try to assign embedding vectors to
them that satisfy
our requirements. Okay? Now, as you can
imagine, this is going to be a long,
painful, and never quite complete
exercise.
Okay?
Uh,
so and uh you mean and given that we are
machine learning people,
the question is, can we do in a better
way? Can we just learn it from the data
without doing any of this manual stuff?
Okay? And
the key insight that makes it all happen
is this humble-looking line on the
screen by John Firth, who was a
linguist.
You shall know a word
by the company it keeps. I wish I could
deliver this in a British accent.
Know a word by the company it keeps.
Okay? It's a very profound statement.
Okay? And here is the sort of the key
intuition behind this.
It says,
let's say that you have a sentence like
the acting in the dash was superb.
Okay?
What are some words that you folks think
are likely to appear in the sentence?
Shout it out. Play. Play.
Movie.
Show.
Musical. Right? Those are all some great
candidates, right? The acting in the
movie, the film, musical, and so on and
so forth. Okay? Now, let's say that I
ask you, what are some words that are
unlikely to appear in the sentence? And
I think we could all be here for like
days, you know, listing them out. Uh, I
just listed these out. Um, I love the
word tensor, so I have to find a way to
use it somewhere.
So, all right. So, the acting in the
banana was superb. Clearly nonsensical,
right? So, what this actually What What
we are seeing here is that if certain
words are sort of interchangeable in a
sentence,
meaning you you change them, they still
the sentence still makes sense, right?
If they appear in the same context very
often, i.e., if they're interchangeable,
they are probably related.
Sort of like we don't even have to know
what the word is.
All we have to know is that this word
and this word, you can drop them into a
particular sentence, you can fill in the
blank of that sentence with that word,
and it actually makes sense, then we're
like, oh, wow, okay, these words are
related then.
Right? You're sort of inferring their
relatedness not by looking at them
directly, but by seeing where they live.
Right? It's a very very clever idea. And
it'll slowly sink into you. Okay? Um, so
that's the first observation. If they
appear in the same context very often,
they are likely to be related.
More generally, related words appear in
related contexts.
So, all we have to do
is to figure out a way to calculate
context.
And then use that to understand, you
know, what the words are that happen to
be living in this context.
And there are some beautiful ways to do
these things, and we'll you and we'll
really dive deep into one such way to do
it.
So, so the So, what we're going to do in
this approach
is that
since
words that appear in
related contexts mean related same
similar things,
first of all, you have to define what do
you mean by context?
And there are many ways to define
context. We're going to go with a very
simple explanation, simple definition,
which is that if words happen to appear
in the same sentence a lot,
then we think that, okay,
they are in the same context. So,
context here means sentence.
Okay?
So, what we can do is we can actually
take a whole bunch of text, maybe all of
Wikipedia,
and then break it up into sentences.
We'll have billions of sentences, right?
And then for all these billion
sentences, we can literally go and count
for every pair of words, how many times
are both these words showing up in the
same sentence?
Okay? And we call this co-occurrence,
right? The words are co-occurring in the
sentence.
And it doesn't have to be next to each
other,
right? We know that in complicated
words, a word at the very end of the
sentence could actually alter the mean-
could be its meaning could be altered by
a word that happened in the very
beginning of the sentence, and it could
be a really long sentence.
So, we take the whole sentence and say,
are are two words co-occurring in the
sentence, yes or no? And we just count
them up.
And when we do that,
right? When we do that, we will get
something like this.
So, I'm just
This just captures what I've been
talking about. Identify all the words
that occur, let's say, in Wikipedia. And
then for every sentence, you look at
every word pair and count the number of
times they appear in the same sentence
across all those sentences. Okay?
This is a word-word co-occurrence
matrix. So, for example,
let's assume that you took all of
Wikipedia, looked at all the words,
distinct words, and you found there are
500,000 words.
Okay? So, there are 500,000 words
here in the columns
500,000 words on the rows.
The columns and rows. And then you go
and each cell of this table is basically
has a number that you calculate which is
the number of times the word in the row
and the word in the column happen to
show up in the same sentence. That's it.
So, for instance
if you look at deep and learning, right?
The word deep and the word learning
maybe that
the those two words occurred in the same
sentence maybe 3,025 times.
3,025 sentences across all of Wikipedia.
You put 3,025 right in that cell.
Okay?
Many words are unlikely to appear in the
same sentence.
So, much of this matrix is going to be
zero.
But, we
fundamentally form this co-occurrence
matrix.
This matrix essentially embodies all the
context information that we can work
with in a very compact, beautiful you
know, sort of
elegant
And using this, we're going to try to
figure out
what the word embeddings actually are
going to be.
Okay?
And so
So, by the way, the approach I'm
describing here to calculate standalone
embeddings is called Glove.
Uh it's called Glove and
standalone embeddings first sort of came
onto the NLP deep learning scene. Uh
there were two sort of ways of doing it.
One was called word to vec, word to vec.
Uh the other one is Glove.
And they're both comparable, right? They
use slightly different mechanisms of
doing this.
We went with word for for this lecture
because I think it's actually a little
easier to understand and equally
effective.
Okay?
So, this is what we have. And so, what
we want to do is
we want to learn these embedding vectors
that can be used to essentially
approximate this matrix.
Right? If you can find vectors that can
actually approximate this matrix, then
hopefully those vectors do in fact
capture some notion of what the words
actually mean. Okay? So, let me put it
differently.
You come to me with this matrix. Okay?
And you say uh okay, Rama, do you have
embeddings for me?
And I'm like, yeah, I reach into my bag
and I'm like, okay, every one of those
500,000 words, I have an embedding.
Right?
Let's ignore for a moment how I actually
calculated embeddings. I have the
embeddings.
How will you know if my embeddings are
any good?
How will you know?
How can you actually assess if those
embeddings are any good?
Well, you can certainly say, okay, give
me the embeddings for movie and film and
you can see if they're really close by.
If you can look at the you look at the
embedding for movie and tensor and
hopefully they're far away.
But, you'll never get done.
Right?
How can you systematically evaluate
this?
Well, what if you could actually what
what if I come to you and say, not only
am I going to give you an embedding,
here is a procedure
which you can use with these embeddings
to validate how good they are and here
is the procedure. What you can do is you
can use the embedding to recreate the
co-occurrence matrix.
And if the recreated co-occurrence
matrix actually matches the real matrix
well, these embeddings probably are
pretty good.
Remember, the whole point of the
co-occurrence is to handle this context
information. So, if my embeddings can
actually recreate them, reconstruct them
pretty close, right? It'll never be
perfect. But, it comes pretty close,
then we're like, wow, okay, these
embeddings do mean something.
So, if it turns out for instance that
the matrix has, you know, 3,000 possible
va- value of 3,000 for deep and learning
and values of uh
say
50 for extreme learning
and our embedding comes in and says
3,002 for the first one and 48 for the
second one, we'll be like we'll be
pretty impressed.
Whoa, it didn't need to be that close.
Unless it was actually capturing
something.
Okay? So, that's what we're going to do.
And so, we're going to take this logic
of saying
find embeddings that can approximate the
what we actually see in Wikipedia.
Right? And we're going to use that idea
to actually build the model and learn
the
using nothing more than basically linear
regression.
And here you are thinking that linear
regression is useless now that you've
graduated machine learning, right?
So
So, we can think of the embedding
vectors that we want to figure out as
just the weights in a model.
In a linear regression.
We can think of the co-occurrence matrix
as just the data we're going to use in
this model to estimate these weights.
And the model we're going to use
is something like this.
So, first I have to inflict some
notation on you.
We would denote the co-occurrence matrix
of say words I and J as Xij.
Xij is just data.
It's just data. Okay? It's not a
variable, it's data.
Uh
and then we will denote an embedding
vector for each word. Remember, we need
to have a vector for each word. So, we
call it Wi, right? Wi is the embedding
vector for each word.
And we will also assume that
some words are just inherently very
popular. They're going to show up all
the time like the word the.
Okay? So, we'll assume that every word
has some natural frequency of occurring
like movie versus flick.
The versus tensor. So, we want the
vectors to capture the co-occurrence
patterns independent of how naturally
frequent the words are.
Okay?
And so, to capture this natural
frequency, we will assign a bias or Bi
to each word that we're going to
calculate. And all this will become
clear in just a moment. Okay? So
with this setup, basically what we're
saying is something very simple. We're
saying, look, this co-occurrence matrix
that we have
that we're able to compute, it came
about because in in truth, in reality,
in nature, there are these embedding
vectors for every word.
There are these biases Bi for every word
and every co-occurrence number that you
see just came about because, you know,
under the hood, mother nature grabbed
the bias number for the word I, the bias
number for the word J took the two
embedding vectors, which only mother
nature knows at this point did the dot
product of them, add them, and that's
how we get this number.
So, it basically says the number you see
is the sum of the inherent popularity of
the first word plus the inherent
popularity of the second word plus the
way in which these two words connect to
each other.
That's it.
And
you will agree with me
that literally can't get simpler than
this.
If I tell you, hey, here are two things.
I want you to tell me how connected they
are, you'll be like, well, let's take
the first one, figure out how inherently
popular it is, inherent popularity, and
then of course you got to worry about
the connection. So, we do a dot dot
product.
That's it. Those three things.
Right?
So, this is what we have. Now, you may
have seen
uh
from your, you know, good old linear
regression that whenever uh your
dependent variable happens to be
positive, guaranteed to be positive
and it ends up having a big range
we always advise you folks
to take the logarithmic transformation
to squash it into a narrow range because
that will make these models much more
well-behaved.
Regression if the Y value is like a huge
range. Like the canonical example is
that, you know, if you are trying to
model, you know, the net worth of
people, right? It's going to have a long
right tail with people like Elon and
Jeff and so on on the right side, right?
And the rest of us on the left. So and
so, to model this big long tail
distribution, you just take the
logarithm, just squash everything to a
very narrow range. And that will make
regression much better behaved. Okay?
Here
most of the counts are going to be zero.
But, some of the counts could be very
high.
Right?
And therefore we wanted to If you take
the logarithm, it makes it much better
behaved, so we take the logarithm here.
So, this is actually our model. That's
it.
And I know that many of the numbers are
zero and log of zero is not defined. So,
we can just add the one a number one to
all the numbers
to avoid that kind of, you know,
technical arithmetic problems.
But, this conceptually is what's going
on. This is the model we want to
calculate.
So, given that we have essentially
postulated this model
and we have this data, this
co-occurrence matrix, how can we
actually find the weights? How can we
actually find the Bs and the Ws? What
would we What should we do?
Go back to the fundamentals of
regression. Think about it conceptually.
You have some model which has some
weights.
There's some data you can use to train
the model.
Right? And you need to find the best set
of weights. What does the best mean
here?
The lowest
The lowest error. Exactly. There are
many ways to measure error, right? What
would be What is the simplest thing we
could use? So, what you do is you would
actually do mean squared error. Right?
Which is what you're getting at.
You could take the actual thing, you
could take the predicted thing, take the
difference, square it, and minimize the
sum of it.
Okay? If your model exactly nails every
number in the co-occurrence matrix, the
error is going to be zero.
Okay? So
what we do is we literally just do that.
This is the data.
This is the actual predicted value.
Predicted value, actual value,
difference squared, add them all up,
minimize.
Okay?
Uh yes.
And in the loss function, how is this
capturing the context? Because unless my
input data is having that context
how will this actually differentiate
based on where the particular word is
used?
The word The way the word is
the
So, let's take two words like deep and
learning. Now, let's take this word and
change it according to the context.
Okay.
Sorry, go ahead. Yeah, so basically,
let's say I'm talking about the word
banana. So it's a fruit in some context
and I could be saying he's going
bananas. That's a
whatever, right? So now these are two
different contexts in my understanding
and my same model needs to be able to
tell me that banana is the right word in
this context but wrong word in this
context or
correct in both contexts. Yeah, very
good question. So let's actually spend a
minute on that. Good question. I'm going
to swap to my iPad.
So let's let's assume that this is our
co-occurrence matrix.
Right? And then we have words going from
A all the way to let's say zebra, right?
This is the all the words in our
vocabulary
and we have A through zebra here.
And now what we have is
we have uh
apple
and banana.
Right?
So basically what's going on at this
point is that
here every number here measures
for every word here, how many times that
word and apple show up in the same
sentence, okay?
It is not measuring, to your point,
how many times apple and banana are
showing up. It's measuring how much how
many times apple is showing up in each
sentence, right? Now, if apple and
banana are sort of interchangeable,
what do we expect these numbers these
two rows of numbers to look like? Let's
assume that apple and banana are perfect
synonyms.
Just for argument, okay? Let's say it's
a perfect synonyms.
What do we expect these two
numbers
to look like?
Very similar.
So if two words are related, their
entries their entry row vectors in the
co-occurrence matrix are going to be
very very similar.
So that is how the context comes into
the co-occurrence matrix.
So what we want to do is we want to find
if if embeddings can recreate the same
pattern of numbers in these two
uh in these two rows, it's actually
capturing the underlying context.
So words which are similar will sort of
zig and zag together the same way
through the co-occurrence matrix.
And that's where it comes in.
Yeah.
What's up with the diagonal of the
co-occurrence matrix where you have
apple showing up twice? Oh oh, I see. So
yeah, here the you can just ignore the
diagonal typically
uh because all the action is off the the
off-diagonal entries.
So so that's basically the idea and uh
if words which are very similar will
have a very similar pattern of numbers
and then any
embeddings that can actually recreate
the same pattern of numbers is capturing
the underlying reality of what's going
on.
If words are kind of unrelated, those
two those two vectors, let's say that
the word you have is uh
Let's assume the word is uh of course
you know what I'm going to say, tensor.
Right? These two vectors
will sort of won't have any connection
to each other.
Which means if you look at something
like the correlation of those two
vectors, it's it's going to be around
zero.
Right?
Words which are
you know, interchangeable will have a
very high correlation.
Words which are antonyms and never show
up in the same place together may have a
highly negative correlation, close to
minus one for instance. So that's sort
of the intuition behind what's going on
in these two row vectors on these row
vectors.
And so the point is given this
co-occurrence matrix is capturing all
these word word correlational structure,
any embedding that can recreate it must
have captured the structure as well.
Because you can't recreate something
like this with great fidelity unless you
have some notion of what's going on
under the hood.
That's the basic idea.
Yeah.
So just connecting to Sophie's question.
So in that example then
banana is a fruit and apple is a fruit
as well. Banana and apple are synonyms
and you're going mad, you're going
bananas. How that comes together is that
Oh, I see. You're going mad, you're
going bananas, yeah. So uh so those will
also have some correlational structure
to it which the embeddings will
hopefully catch, but words like banana
which are very they they
the thing is it's called polysemy where
the word looks one way, it looks the
same way. It's like the word bank,
right? It can mean very different things
in very different context. So the
embedding is going to be some average
representation of it, right? But we are
not happy with that average and we'll
get around that average
next week when we do contextual stuff.
All right.
Um
So that's what we have here. So to go
back to this thing,
so what we can do is yeah.
I didn't understand how do we get the
mean squared error in this because we
didn't
do any reading from the data set we got.
We haven't calculated the embeddings.
We are trying to calculate them. Those
are just it's sort of like, you know, in
regression you have, you know, beta beta
one times X1 plus beta two times X2 kind
of thing. The betas are what the
regression produces for us, right? The
the embeddings are exactly that. They're
just coefficients that we're trying to
figure out.
The data is only the X's, the Xij.
And so this is what we're trying to
calculate,
right? And so what you can do is you can
actually start with some random values
for these things
and then
keep on trying to improve to minimize
the error
starting from these random values.
Do you folks are you aware of any
algorithm that which allows us to take
random value starting point and then
minimize some notion of error?
Well, how do you know it's actually
random? Oh.
So that's actually a very deep question.
Um
and
so
it's actually a tough question, right?
Because ultimately the random number is
coming from a computer
and we know how the computer runs. It's
deterministic at the end of the day.
So we actually use something called
pseudo random numbers,
right? Um and there's like a whole
specialized field of math
which essentially says, "Look, how can I
get random numbers that are sufficiently
random even though they come from a
non-random computer deterministic
process?" So we can talk offline about
it,
um but fundamentally all these systems
have some random number generators built
in. We just cross our fingers and hope
for the best and just use them.
So come back to this,
right? We can start with random values
for these weights
um and then we can try to minimize the
squared error. Are are you folks aware
of any algorithm that can help us do
that?
Yes.
Gradient descent. Yes, gradient descent.
Again, comes to the rescue. Uh and since
we are cool, we'll do stochastic
gradient descent.
Okay? So that's it. So gradient descent
actually doesn't care what the function
is as long as it you can calculate a
derivative from it. As long as you
calculate a gradient, you're good.
Right? So we can just run gradient
descent on this thing, right?
Uh one key point here is that gradient
descent, stochastic gradient descent
work for any
any models as long as you can calculate
good gradients from them.
It doesn't have to be a neural network.
Any mathematical function as long as
it's differentiable and gives you a good
gradient.
Okay? So here this is not a neural
network per se, but we can still use
gradient descent for it.
So we do that.
Um and when we are done, we would have
calculated some nice embeddings. We
would have all calculated or we can also
calculate all these biases, but we don't
need the biases anymore. We can just
throw out the biases because we only
care about the embeddings and how they
connect to each other.
Okay? Yeah.
So when when you're doing that
regression, are you predicting the
co-occurrence matrix? Mhm. Okay.
Exactly.
So
um actually let me just show a very
quick example
numerical example here.
So let's say for example that um
you know what?
So this is say W1 and this is W2.
Okay? And this is the vector and let's
assume for a moment that we it has two
dimensions, okay?
Two dimensions.
And we also need to calculate B1 and B2
which is just a number, okay?
So and let's say the number for deep
learning in the co-occurrence matrix it
happens let's say it has occurred 104
times.
So all we are doing is to say log of
104.
That is the actual value
minus
B1 which we don't know plus B2 which we
don't know
and then this thing here, let's just
call it,
you know, W11,
W12,
W21,
W22.
Okay? And then we're just doing the dot
product which is
times W12
plus W21
W22.
Okay? So this is our prediction.
Where is that cool laser pointer? Yeah.
So this is our prediction.
This is the actual.
So all we do is to say, "Okay,
this thing, the difference, we're going
to square it."
And then we're going to do the same
exact thing for every other word pair.
Okay? And when we are done with all of
that thing, we just take this whole
thing
and say gradient descent minimize.
So then it has to find the B's and the
W's and everything for every every pair
every word.
So that's actually what's going on.
Make sense?
All right. So by the way uh here
I said
I said, you know, let's assume that the
embeddings are just vectors which are
two dimension dimension two.
Well,
that's an arbitrary decision that I made
just to show you how it works because I
was doing it by hand. But more
generally, we get to choose how long
these vectors are.
Right?
And the longer the vector, the more
interesting ways it can actually
reproduce the co-occurrence matrix. It
has more flexibility. But the longer the
vector, what is the risk that you run?
Overfitting.
Because these are all parameters at the
end of the day. More parameters you
have, the more risk of overfitting.
Okay? So, you get to choose how big
these things can be. Uh yes.
Um don't you find it surprising that
we're able to fit the model where we
have a lot more parameters than we have
data because usually with most machine
learning with our experts, you would
like to not have a lot of parameters,
but here we're going to have
as you said, the number of dimensions
times more parameters than we have
data points. Well, here in this
particular case, as it turns out, um
let's assume that you only have 10
words, right?
And for each word, let's assume that you
have let's just just keep the math
simple. You have a two-dimensional
vector.
So, 10 words * 2, that's 20.
Plus you have 10 biases for the words,
right? So, that's another 10, that's 30.
But 10 * 10, the matrix has 100 entries.
So, because of the matrix being a order
n squared matrix, you'll have a lot more
numbers than parameters.
In this particular case, you have more
data than parameters.
So, that particular problem doesn't
apply in this case.
But that does show up in other cases and
there is some
very interesting research in neural
networks which suggests that often times
the traditional assumptions of data and
overfitting and all
can all be called into question under
some situations.
Um happy to tell you more offline, but
if you're curious, just Google something
called double descent.
You know what I mean.
But in this case, it's not a problem.
Okay.
So, so what that means is that we can
choose how big these things are. So, if
you look at one-hot word vector, one-hot
vectors, right? Where
there's a one and everything else is
zero depending on the position of the
word, these are long vectors as long as
a vocabulary, right? As we saw earlier.
Word embeddings on the other hand,
right? They can be very dense, right?
The numbers
that make up these embeddings, we're
actually going to figure out from the
data what they are. So, it can be
anything. It can So, the first dimension
may stand for some combination of, you
know, um
brightness plus speed plus animalness or
something. We have no idea what it
means.
All we know is that it's able to
reproduce the co-occurrence matrix
really well, so it's probably has
figured something out.
Okay? And so, we can keep it really
short. So, the word embeddings tend to
be very
dense,
meaning not zeros and ones, but some
arbitrary numbers. It's very lower
dimensional and it's of course learned
from data.
Right? So,
so once you do this, once you actually
run Glove on this data and do gradient
descent and so on and so forth, uh you
will actually come up with embeddings
and then you can actually plot the
embeddings. You can take like this they
say the you know, you can take these
embeddings and just plot them. Here um
they're not literally plotting the first
two dimensions. They're using a
particular technique called t-SNE, which
is a way to take long vectors and
project them to 2D space for
visualization purposes.
And you can see here
some very interesting things are showing
up. So, they basically they plotted the
embedding for brother,
nephew, uncle, sister, niece,
aunt, and so on and so forth. It's all
showing up here.
This the embedding for man, embedding
for woman,
sir, madam,
empress, heir,
duke, emperor, king. You get the idea.
Right? So, clearly there are patterns
here where
things which are sort of similar in
their nature are all hanging out
together in the same part of the space.
Which is comforting, which is good to
know.
Right?
Now, but as I mentioned earlier, it's
not just about the fact that similar
things happen to be near each other.
The direction also actually matters. And
beautiful things happen when you look at
directions. So, for instance,
you know, let's say that
man and you want to go from man to
brother.
Okay? So, to go from man to brother, you
have to start with man and then travel
along this arrow, right? To get to
brother.
So, this arrow has some notion of a
person becoming a sibling.
Right?
So, you would hope that if you take that
same arrow
and then
start here with that arrow, hopefully
the woman will become a sister.
Sure enough, this.
So, this is called word vector algebra.
Right? Embedding algebra. And these
relationships are actually showing up in
the data. We didn't tell it any of these
things.
We just literally gave it the
co-occurrence matrix
and said and and asked it to reproduce
it.
So, I find it pretty shocking that these
things are actually true.
And it gives us evidence and comfort
that whatever has been learned does have
some deep connection to describing the
underlying nature of what's going on.
It's not some statistically fluky
artifact.
Um yeah.
So,
I said
by context or by adjacency to other
words and not by
the place in the same word, right?
Cuz you can't click they won't appear in
the same sentence.
They have
keywords. Right.
They won't appear in the same sentence,
but the pattern of co-occurrence will be
the same for them.
Which is what we've been able to
reproduce with these embeddings. So,
that's the key idea.
Um
so, my question is along like how are we
able to capture all these directions in
2D
matrix versus a multi-dimensional matrix
because I feel like okay, so this
relationship is kind of
uh
confirmed that you're moving to
kind of like
family or like blood relationship or
something of the sort, but like how does
it not mess up the other sides of that
matrix? Like
No, this is just a visualization thing.
So, we're basically taking this uh you
know, as you will see, Glove embeddings
come in lots of different sizes. And
this I think uses the 100 dimension
embedding and just projects it to 2D
space using a particular technique and
then looks to see what's going on.
Um yeah.
Uh if the input data being co-occurrence
matrix is biased, aren't we amplifying
that bias? Yes, we are. Yes. No, it's a
great observation. Uh any sort of data
you scrape from the internet and use for
this sort of modeling exercise will be
subject to all the biases that produced
the data in the place first place. And
the model will faithfully learn those
biases. And if you're not careful, it'll
perpetuate them.
So, and that's a whole very important
topic that unfortunately won't cover in
this course because of time constraints,
but it's something you always have to
worry about when you're building these
models.
How do you think about the
dimensionality of the embeddings not the
2D representation of the actual data?
The one that we choose, that's that's in
our hands. So, you should think of them
as a hyperparameter.
So, much like the number of hidden units
to use in a particular hidden layer,
um it's a hyperparameter. Uh so, you
know, I would again start small and if
it solves the problem that you're trying
to solve with these embeddings, great.
If not, keep increasing them. And at
some point there might be like a a
flattening out and a overfitting sort of
dynamic and then you stop. So, just
think of it as a hyperparameter.
Yeah.
Do you see any benefit practicing using
like penalized regression to do this
instead of having the embeddings more
sparse or just like
lowering the magnitude of them? Yeah.
Yes. So, there are lots of techniques to
uh
to apply regularization in the
estimation itself of all these numbers.
Um happy to give you pointers. It's I'm
just going with like the simplest
version possible.
Yeah.
Am I understanding why overfitting is a
problem in this case cuz we're not doing
any like out of sample
prediction. So, like wouldn't you want
like the embeddings to be
like high dimensional so you can capture
like
your relationships? Uh interesting
question. So, the question is given that
there's no notion of a test set, out of
sample test set that we got we're going
to evaluate these things on, why do we
really care about overfitting? Don't
should we do the best we can to capture
everything in the data, right?
Well,
the thing is
even when you're not trying to use it
for out of sample prediction, you do
want to make sure that your model only
captures the true patterns and not the
noise.
In every data set, there's always noise.
Right? And you want it to capture a
signal but not the noise.
And regardless of what you use it for.
Because if it captures the noise, then
the insights you draw from the word
embeddings may be flawed.
That's the reason.
Okay.
Um all right, so let's keep going. So,
here the algebra is brother minus man
plus woman is sister.
That's it. Human biology reduced to a
single sentence.
All right. So, now the pros and cons of
these things are you should use
something like a Glove embedding if you
don't have enough data to do to to sort
of
to learn a task-specific embedding for
your own vocabulary. As we As I'll show
you in the Colab, you can actually learn
these things just for your own data set
if you want. You don't have to use these
Glove embeddings. But the reason to use
these pretrained embeddings is that if
you're working with natural language,
you know, the word is the word, right?
It means something.
And so, there's no reason for you to
have for your model, for your little use
case, for you to actually somehow learn
all the fundamentals of English.
The fundamentals of English are the
fundamentals of English. May as well
learn it once and then piggyback on it.
So, that's the whole idea of using
pre-trained embeddings.
Because it These things are all common
aspects of language. May as well learn
them using all the data you can throw at
it and then you can sort of fine-tune
and tweak and adapt to your particular
use case.
Right? So, if you and this particular
useful when you don't have a lot of data
in your particular use case.
Uh right? That's one big advantage. Now,
it does have the drawback that this
embedding will not be customized to your
data.
Right? For example, if you're trying to
build an application for a medical or
legal use, it's going to have a lot of
jargon.
Right? And this pre-trained embedding
trained on all of Wikipedia may not
capture enough of the jargon and know
its meaning really accurately. So, what
you want to do is you want to take this
thing. You may still want to take this
thing and then you can adapt and
fine-tune it using your jargon-packed,
heavy, domain-specific data set.
Okay, those are some of the things to
keep in mind.
And of course, we can also learn it from
scratch if you want and the collab I
demonstrate all these options.
So, when you're working with embeddings
in Keras uh Keras, so what we do is
remember STI
where we after we standardize and
tokenize and index, right? At this
point, we go from integers to vectors
and so far we have been using integers
to one-hot vectors. Here, we're going to
use embedding vectors that we're going
to learn or that we're going to pre-use
from glove. And so, what we do is we
tell Kera we tell Keras's text
vectorization layer to do only STI.
And then we will use a new layer called
the embedding layer to do the encoding.
Yeah, that's how we're going to do it
divide divide it up.
So, we'll take a look at this first uh
before we switch to the collab. So,
before
we told Keras in this layer output mode
should be multi-hot or whatever, right?
Here, we don't want it to actually
encode anything in multi-hot. We just
wanted to give it integers back. So, we
tell it give me int.
Okay? That's the first change. We only
We tell it give me give us int. If you
say give us int, it'll stop with STI.
I'll just give you the integers.
Uh and then what you do is that
all the incoming sentences are going to
have different lengths. So, what we want
to do is we want to actually take all
these sentences and sort of normalize
them so they are of the same length.
Okay?
And the way we do that
And the way we do that very quickly is
that we either trunk we choose a maximum
length for every sen- for for the
sentences and then if something is
uh exactly fits that length, perfect.
Let's say in this case we want a max
length of five. Cats sat on the mat is
exactly five. Boom, fits perfectly. But
if something is smaller, I love you is
only three of these things, we actually
pad it with something called the pad
token.
Much like the unk token, pad token is a
special token which we use for padding.
And then it'll you know, and so and
Keras you will see will use zeros for
these paddings. So so that it fills it
up and gets all the way to the end. And
if you have something which is much
longer than five, you just truncate
everything else and just use the first
five.
So, this is what we do to get all the
sentences to be of the same length.
Okay?
And once we do that we then go to the
embedding layer.
And the embedding layer is actually very
simple.
What is What is an embedding? It's just
a vector and we need a vector for every
token.
Of course, we're going to learn these
vectors. We need one for every token.
So, in this case for example, uh let's
say that these are all the tokens we
have
in our vocabulary after the STI process.
Maybe in this case we have 5,000 tokens.
Each token we have this embedding
vector, right? And we choose what the
dimension of that embedding vector is,
right? And so, we can set it up by
saying Keras layers.embedding and we
tell it max tokens which means what how
many rows do we have here.
You know, how many What is the
vocabulary size that we're working with?
And then we tell it, okay, this is how
long I want each embedding vector to be.
So, rows, the size of the columns, and
that's the embedding layer. And we'll
use it in a second. I just want to show
it to you here so that's because it's
slightly clearer.
So, when an input sentence arrives, the
text vectorization layer will learn STI
on it. It'll truncate and pad it to max
length as needed. So, let's say this
phrase comes in, STI will give you the
same tokens plus pad pad because let's
say the max length is five and then
these are the corresponding integers.
And then
the embedding layer will just look up
the corresponding vector. So, for
example here, uh the vectors are we need
to look up the vectors for 23, 9, 5, 0,
and 0. So, we just go here and look up
23, 5, 9, and 0. And then once we have
that, boom.
This is the resulting output. So,
whatever input sentence comes in, we
have now
five embedding vectors that have been
looked up from the embedding layer.
And once we do that
this is a table. So, I love you comes
in, it becomes this table. As we have
seen before
neural networks can only accommodate
vectors as inputs. We need to you know,
make this into a vector. And as we have
done before, you know, we can either
take all these things and concatenate
them, make a one long vector, or we can
find a way to average them or sum them
and things like that, right? As we have
seen before. And we will use the same uh
we'll the simplest thing is probably
just to average them. So,
uh these are some options and we but
we'll average them here. So, and this is
called the global average pooling layer
1D. And it's all it does is whatever you
give it a table you give it, it just
takes each dimension and averages it.
The first dimension average, second
dimension average, and so on and so
forth. And once that's done
that's the whole
So,
the phrase comes in, STI gives you these
things, padding as needed or truncating
as needed. We look up the embeddings
from the embedding layer and then we get
all this thing. We do global global
pooling on it and it's done.
The resulting thing is a vector that can
then be passed into hidden layers just
like we normally do.
I'm going over this a little fast, but
make sure you look at it afterwards and
understand every step and the collab
will mirror this
you know, perfectly.
All right, so let's switch to the
collab.
Okay. All right.
Can folks see this okay?
All right, so we'll do the usual.
Um
import all the stuff we need and then
because I want to plot some of these uh
loss and accuracy curves to
you know, just to see what's going on,
I'll just bring in the functions from
the previous collabs.
Here.
And then um and I think I already have
downloaded this. Let me just make sure I
have it.
Uh it's not there. Okay.
Do it again.
This is same songs data set that we
looked at on Monday.
Okay.
So, roughly 49,000 examples as we saw
before. We'll one-hot encode them.
All right, so there's a bunch of stuff
that we already covered in class. So,
this is the thing
uh this URL has all the glove vectors
available for download. I downloaded it
uh before class because it takes a few
minutes. Uh and I've also unz- Did I
unzip it?
Uh yes, I did. And so, let's just look
at the first few.
All right, so these are all the first
few. We'll create a sort of an easier to
view version of these glove vectors.
So, I'm going to use the vectors which
are 100 long, but it comes in many
different shapes.
So, we have 400,000 vectors, 400,000
word vectors. Each is 100 dimension.
Uh and these all have been calculated
from Wikipedia using
the model we described using gradient
descent. Okay?
Uh all right, so this is the
vector for the word for movie.
Yeah, I don't know what these dimensions
mean, but it is there's something going
on. It has figured stuff out.
Uh but the proof is in the pudding,
right? So, all right, now we'll first
set up the text vectorization and
embedding layers like we saw before.
Um and so, I'm going to use uh a max
length of 300 for the songs.
Um right? Because all the sentences have
to be the same length. And you might be
wondering, okay, why did you pick 300
and not say 400 or 200? So, typically
what you do is you actually look at the
the length distribution of the songs you
have and you will find you're looking
for like an 80/20 or a you know, one of
those things. And in this case it turns
out 90% of the songs have less than or
equal to 300 words in our data set. So,
I'm just going to go with 300. Okay?
It's pretty good. Uh the problem is if
you actually say if you look at the song
which has the maximum length
that might have be like 3,000 words and
there would be any hardly any songs of
3,000 long. You're just wasting a lot of
capacity by doing that. So, you're just
being a little pragmatic here.
So, okay. So, and then we as before for
the vocabulary itself, we tell Keras use
the most frequent 5,000 words, right?
When you're doing the STI
um STI. So, we do that and we tell it
the output mode is int like we saw
before.
We have there.
Okay, perfect.
Okay, this is a very dangerous thing
where somebody is remotely changing it
in another tab somewhere.
Fingers crossed. Okay.
Okay. So, we have this um and this is
what we did with all this stuff uh as
I've covered. So, now we will adapt this
layer as we have seen before using all
the lyrics we have.
And once we that, we'll take a look at
the first few.
And so, here's a very important thing.
Before, when we asked it to do multi-hot
encoding and so on in on Monday,
uh the zero, the first position was unk.
Right? Unk had zero. But here, unk
actually has one.
And the reason is that
the zeroth position is going to be uh
used for essentially the You can think
of this as the empty string. That's how
Keras will print out pad.
So, the zero position is the padding,
the pad token. The first position is the
unk token. Okay?
So, it's an important thing here.
So, let's say that we do
"HODL you're the best."
We take a vectorize it. Um
Do you think HODL
is going to be part of those 400,000
word vectors?
Wikipedia. Not yet. So,
Um all right. So, let's try that.
Okay, and as you can tell,
um
HODL is an unknown word, right? That's
why uh it's showing up here.
Right. So, one is unknown, right? The
index value one is unknown. Zero is pad.
But then,
this is unknown HODL, I
Sorry, you are the best, and then
everything else from that point on is a
zero because we are padding all the way
to 300.
Okay? So, that's why you see all these
zeros here.
All right. Uh now, let's just, you know,
run everything through
the vectorization layer, and then we'll
get to the embedding layer.
Okay. Now, we will we'll we'll first
There's just a bit of Python uh
housekeeping
um to create a nice, easy to look at
matrix. So, what we're going to do is
we're actually going to create a nice
matrix which shows us all the the word
the GloVe embeddings.
Um
And so, here, this is the embedding
matrix.
And this matrix has only 5,000 words,
and each is a 100 long.
Why is this embedding matrix only 5,000
even though we downloaded 400,000
vectors?
Right. So, clearly the 5,000 we used
there has some bearing to this, but what
is that 5,000?
We told Keras to take the most frequent
5,000 words in our corpus.
So, we'll only have 5,000 in vocabulary.
That's why there's 5,000. So, we grab
just the word the GloVe vectors for
those 500 5,000 that Keras has chosen to
be in the vocabulary. Okay? And that's
our embedding matrix.
And then, if you look at the first few
rows, the first two rows should be all
zeros because it's pad and unk,
which clearly GloVe doesn't know about.
It's all going to be all zeros. And um
so, you can see all these zeros here,
and then from third on, words, you start
getting some numbers. Okay?
All right. Next, we'll set up the
embedding layer.
Uh
so, basically, what's going on here is
when you we tell the embedding layer how
many rows, which is just the vocab size,
max tokens, what is the embedding
dimension? Well, that's going to be 100
because the GloVe vectors are 100. And
then, here's the thing. You can tell it
um in this embedding layer, just use
this matrix I'm giving you as the
embedding layer. Because we already know
what the embeddings are. We downloaded
from whatever GloVe, right? So, we will
tell it to use GloVe as as the as the
weights for here, as the embeddings
here. So, we initialize it using that
embedding matrix, right? And then, we
tell it
don't train. When we do back propagation
later on, don't change any of these
weights because somebody spent a lot of
money create these weights for us.
Stanford. So, we don't want to like
further change them. Just freeze them
and use them as they are. Okay?
And this mask zero business I'll come
back later. Don't worry about it for the
moment.
All right. So, once we do that, we all
we are ready to set up our model. So,
this model is pretty simple. Uh Keras
input, the length, of course, is the
length of the sentence, right? Which is
uh 300 long, and then it runs the input
runs through an embedding layer right
there, right? And out comes a 300 by 100
table, and then we global average pool
it,
right? And that becomes a 100 element
vector, and then we are back in familiar
ground, and we run it through a dense
layer with eight ReLU neurons, uh right?
Eight ReLU neurons, and then we run it
through the final output layer, which is
a three-way softmax as before, hip hop
rock pop. And then, we tell Keras that's
our model, and then we summarize it.
Okay. So, this what we have. And you can
see here,
the total parameters are 500,835,
but the trainable parameters are only
835.
It's because the total parameters are
all the GloVe embeddings plus the the
things we added to the GloVe embeddings
like the hidden layer and so on.
But the GloVe embeddings are us we have
told Keras, freeze it. Do not train it.
Right? Which means only the rest of it
is going to be trainable. That's That's
the 835. Yeah.
So, when we do the global average
pooling, don't we don't we lose any
sense of meaning that we gain from the
embedding as we average very different
embeddings together?
Sorry, say that again. I I missed the
first
>> if we average the the embedding of apple
and learning, for instance, they are
very different words that are used in
different meanings, so we have different
embeddings, but we average it, so can't
lose it.
We will lose a bunch of stuff. Yeah,
yeah, yeah. So, you're barely Anytime
you average anything, you're going to
lose some new nuance and so on. So, the
real question is, is it Despite that
averaging, is it good enough for you?
And sometimes it's good enough.
Very often it's good enough, as it turns
out. But as you will see when you go to
contextual embeddings, there's just a
better way to do it, right? When you
have contextual embeddings, uh but it
requires bigger models, more powerful
stuff, and so on and so forth. And
that's where you're going from the
foundations to the advanced stuff.
Yeah.
When we're doing optimization, like
let's say we are word problem, it's
often best to optimize everything
together than to optimize one part of
the system and then optimize the other
part of the system.
So, in that case, why wouldn't we want
to also change the embeddings?
We would like I understand why we would
like to stop with
with those weights that
some people have spent a lot of money
trying to find, but will
we be able to find more specific uh
embeddings related to our problem if we
optimize if we let everything be
trainable? Yeah. Absolutely. Absolutely.
And in fact, you will see in the collab
uh that we will do that next. I just
want to show people you don't have to do
it. You start with not training it
because it's going to be much faster.
And then, you train everything and see
if it gets better. And sometimes it'll
get better, in which case it's great.
Sometimes it won't get better. And I
will also show you, and I probably will
run out of time, which I'll So, I'll do
it on Monday. I will also show you, hey,
what if you want to do your own
embeddings from scratch without using
GloVe?
So, all possibilities will be covered.
Um yeah. So, to come back to this, this
is the model we have. Um and then, all
right.
So, we'll If we take a look at the first
few embedding vectors, by the way, this
model.layers
uh will give you every layer as a list,
a list of all the layers, and then you
can just grab any layer you want and
look at its weights. Okay? It's very
handy.
So, we're looking at the weights, and
you can see here
the first two vectors are all zeros
because that stands for unk and pad, and
then we have everything else. So,
everything looks fine so far. And now,
we just, you know, compile and fit it.
So, as usual, Adam, cross entropy,
accuracy.
Um and then, we'll just fit the model.
All right.
It's going to take
a few minutes.
And while it's running, so what what you
will see in this collab is that
uh in this particular case, the
embeddings actually don't help a whole
lot.
Why do you think that is?
What if it could be because we're
averaging a lot of stuff? Maybe that's
hurting us.
Yeah.
Um I mean, I think that the embeddings
were pre-trained on some corpus, right?
Like Wikipedia or something like that
that is different from the a little bit
different from the language we tend to
use in song lyrics. So, so maybe it's
not
its ability to sort of extract the
meaning of
um
candy from like a song lyric um
maybe is limited because Yeah. it's
thinking of all the other ways Right.
like that could be our presentation.
Yeah, so there could be a mismatch
between the corpus on which the
pre-trained stuff was trained on versus
the the corpus that you're working with
right now. That's one big reason. The
other reason is that we actually may
have We have 50,000 examples, basically.
It's a lot of data.
So, when you have a lot of data, you may
not need any of these things.
These things tend to do really well when
you don't have a lot of data, and which
means you you you get to piggyback on
what these embeddings have learned from
all of Wikipedia.
So, so when you have a smallish data
set, basically, the the rule of thumb
here is that when your data is really
small, try to use a pre-trained model.
Right? And that's what you saw with the
handbags and shoes classifier, right? We
had 100 examples of handbags and shoes,
and we used ResNet to got basically get
to 100% accuracy.
The same sort of logic applies here.
All right. So,
here, let's see what's happening. Uh
okay, it's done.
So, we'll plot.
Right.
Uh okay, this look at a very
well-behaved uh loss function curve.
Uh
Okay.
So,
uh there doesn't seem to be any massive
overfitting going on. They are moving
really nicely in lockstep. Let's see
what the thing is.
Okay, 63%, which is not great. Um right?
Uh it's not as good as what we saw
before when we used all 50,000 examples
and just trained something from scratch,
and that's just because in this case, we
have lots of examples, these pre-trained
embeddings aren't, you know, as helpful
as they could be.
But if you have a small data set, they
could be very helpful. And now, we go to
what um
he pointed out. Like, why can't we just,
you know, optimize these embeddings,
too? Why don't Why do we have to take
trade them as sacred? We'll just Let
Let's just use Let's
inflict Let's just apply unleash back
prop on it and see what happens.
So, we'll do that. Um
So, here, what we do is we retrain it,
but here, we set trainable equals true
for the embedding layer. Okay? This is
the key step. Trainable equals true.
Otherwise, it's unchanged.
Uh and then,
let's skip that.
We'll run it and see what happens. So
before it was whatever 63% accuracy or
something, we'll see if it gets better
if you train the whole thing.
And the thing is you can never be sure.
Right? Because it may start to overfit.
Uh which is why you just have to
empirically see what's going on. There
are no guarantees.
Um all right, any questions while it's
training?
Yeah.
In that first graph of when um you have
the training accuracy still increasing,
that might suggest that you could use
even more upstream. Correct. Exactly.
Exactly. So in the in the in that curve,
we saw that the training was continuing
to increase. Typically what's going to
happen is the training will continue to
get better the more you train it. The
key thing is is the validation also
improving. If the validation continues
to improve, there is a little bit more
gas left in the tank. You can keep
increasing more. If it starts to flatten
and even worse if it starts to go down,
then you want to pull back.
Yeah.
Um so you had used the maximum against
the limit like the vocabulary
of the most common 5,000. And then the
width of that was 100. What is the 100?
The 100 is just the length of the glove
vector.
Does that mean that it can only capture
how that word is related to 100 other
words? No, no. It it basically we are
saying that every word its intrinsic
meaning can be captured using a vector
of 100 dimensions.
Those dimensions mean something. We
don't know what it is. The first
dimension could mean color. Second could
mean some sort of location. The third
could mean some sort of see time of the
year. We just have no idea.
Okay, and then the pre-trained model is
we're not We're not going to learn the
pre-trained model like has those
already. We don't know what they are,
but it has some cat The people who
created it don't know what they are
either.
All they know is that for each word they
learned a 100 long vector.
And that 100 long vector was able to re-
kind of recreate the co-occurrence
matrix.
And then they probed it using that
visualization of man woman sister
brother all that stuff and it seems to
sort of fit with what you would expect.
Can you think of it as analogous to uh
when we did the convolutional ones, you
have the number of kernels, right? So in
in this case, so if you have 32 kernels,
it's sort of like 32 things it can
learn.
I think that's actually a great analogy.
I love it. That's that's a great way to
think about it. Yes. Uh much like we got
to choose decide how many filters to
have, here we get to decide how long the
embedding dimension needs to be and our
hope is that the more things we are able
to accommodate, the more complicated
things it will pick up. Right? Uh at the
same time, you don't want to have too
many of these things because it's going
to start picking up noise.
And that's not a good That's never a
good thing.
Okay.
Um
Another question on this side?
Yeah.
Go ahead. My
question is
why did we use Why do we use embeddings
and not the actual uh
correlation matrix called rows to
represent words, right? Like why do we
need to abstract Yeah, yeah, yeah.
That's actually a good That's a That's a
good That's a good question. Um one
immediate reason is that that row is
500,000 vectors long. 500,000 long.
Right? So you want a compact dense
representation of a word.
The second thing is that thing is
subject to all the counts of the
Wikipedia corpus. It's not normalized.
So you need to normalize it so that if
you take any two rows and do dot
product, you will get some number which
is sort of in a narrow range. Otherwise
things don't become comparable.
No, both these objections can be
handled. You can normalize, you can
reduce the size of the corpus and so on
and so forth. And in fact that used to
be a very common way people used to do
it before.
But what they have discovered is that
these the way we learn embeddings now
tends to be much more effective in
practice.
So So what what we thought is
what what what this process does is it
creates this like n-dimensional
incomprehensible matrix that captures
in essence a summarized version of these
relationships.
Correct. A compact representation of
relationships which is not subject to
the size of your vocabulary.
So you know, you have 500,000 words
today, tomorrow somebody comes up with
the word called selfie which didn't
exist 5 years ago.
And now your corpus has gotten a little
bit more, right? So here it's very
compact and it tends to have a much
longer shelf life.
Yeah.
Uh all right, so let's see where we are.
Uh okay. So evaluate.
68 69% almost. It was 63 went to 69. So
clearly here training the whole thing
including glove actually helps. Uh and
so that sort of begs the question, well,
if it um every if training glove helps,
maybe we should actually train the whole
thing from scratch.
Like why the hell not, right? Why the
heck not? I apologize.
So uh what we'll do is we'll actually
create our own embeddings and just train
them. And here we don't have to worry
about co-occurrence matrices and so on
and so forth because we have a very
specific objective. We want to be very
accurate in predicting genre for these
songs.
The people who had who had worked on
glove,
they didn't have any objective. They
just wanted to create embeddings that
were generally useful.
Okay? Here we want to be specifically
useful for genre prediction.
And so what we can do is we can actually
train the whole thing ourselves, right?
We can actually give it
uh we can actually put an embedding
layer here. I you know, we just
arbitrarily decided to choose 64 as the
uh the dimension as opposed to 100. It
will run faster. Uh and then it's the
same thing. Global average pooling,
activation, blah blah blah blah blah. Um
and then you run it.
We'll see if it finishes in the next
minute.
And we'll see if it actually does better
than the pre-trained embeddings or the
pre-trained embeddings that have been
further fine-tuned. And I don't remember
what I saw when I ran it yesterday.
Uh and while it's running, other
questions?
Yeah.
So my question is regarding embeddings.
When we call embedding for a particular
word, we indicate that we have certain
number of parameters. Let's say in this
case we have defined
We defined 100. So there will be 100
parameters and there will be
coefficients weights for each of them.
So when we take a pre-trained model,
right?
The one we took glove. So for each word
there would already be those number of
parameters in that different Yeah. So
but then how do we redefine them? Is
that we want only 100 or we want only 10
parameters
You know, the the glove thing actually
gives you packaged It's pre-packaged to
be 100 long. I think they have 200 and
300 as well if I recall. We just
happened to use the one the one with
100. The one is
The one is available in Google
Yeah, yeah. And there are many
available. We just get to pick and
choose and I happen to pick 100.
Uh
Oh, it's okay. So it's a bit slow, but
it's actually looking promising.
Um
9:55, yeah.
So during the CNN models training during
our assignments,
changing the filters gave us more depth
than improvement in performance.
So here would I be right in concluding
that it's actually training the
embeddings which is giving us more
assuming that epoch and batch changes
are not
changed as much. So if I really want a
genuine change in performance, we go
to the level of retraining the
embeddings.
What Yeah, so what we saw was that using
glove as is was okay. Using glove and
then training them helped a lot. And now
we are basically saying, well, what if
we just abandon glove and train our own
embeddings for our particular problem.
See, glove is a general purpose tool.
So a general purpose tool is really good
if you don't have a lot of data
as a good starting point. But when you
have a lot of data, you should always
try to do your own thing and see if it's
any better.
And in this case, I
well, whoa. Okay, I think it's
uh
Come on, it's 9:55.
The button is going to enter any moment
now.
Right, let's just look at the thing.
Okay, folks. So 74% 72%.
So you can actually return your own
thing because of 50,000 examples and you
can see an even better thing. Thanks a
lot. Have a good rest of the week.