Okay. So today we start the the natural
language processing sequence and so just
to give you a quick idea we're going to
start with uh what's called
vectorization
uh and then the bag of words model and
then we'll spend a fair amount of time
on a collab uh and then on Wednesday we
talk about these things called
embeddings which you'll come to
appreciate over the the next couple of
weeks form like the sort of the core
atomic unit of all modern natural
language processing and for that matter
vision processing as well. uh and then
we will uh following week we'll do
transformers two lectures on
transformers we'll get into the theory
and then we'll get into a bunch of
applications and then lectures nine and
10 will be all LLMs all about LLMs so
it's going to be a lot of fun u this is
one of my favorite segments of the class
of course truth be told every segment of
the class is my favorite so don't judge
me all right so let's get going uh so
why why natural language processing u
you know these are in some sense the the
things I have on the slide here are sort
of obvious but I think it's actually
worth reme reminding ourselves of how
important text is for everything we do.
Uh obviously human knowledge is mostly
encoded as text. The internet is mostly
text. At least this was true till the
advent of Tik Tok and YouTube. Uh and uh
human communication is mostly text and
cultural production you know movies,
books, uh arts and so on. So much of it
is so textheavy and so in some sense uh
text forms not just a big chunk of all
the media that's out there but it also
happens to be the way in which we think
and communicate and so on and so forth.
So it's sort of uh primacy is in my
opinion sort of unparalleled uh in how
we think about the world. And so the the
tantalizing possibility is that imagine
if we had an AI system which could just
read and quote unquote understand all
this text, right? Um and so you can
imagine such a system reading all of
PubMed, reading all the medical
literature and then coming back and
saying you know for this particular
disease you know this particular sort of
protein is actually the malfunctioning
protein and for that that small molecule
is going to dock into the protein and
cure the disease and you didn't know
this. It came back and told you that.
Wouldn't it be unbelievable? So my
feeling is that such things are going to
happen. It's just that it's not going to
happen soon enough for my lifetime, but
perhaps it'll happen in yours. All
right. Okay. So, let's continue. So, NLP
is an action all around us. Um, you
know, according to Google, apparently
Google autocomplete, uh, which uses a
fair bit of NLP, uh, saves 200 years of
typing time apparently, every day. Uh, I
actually thought it was, you know, this
I wasn't very impressed with this
number, frankly, because billions of
searches are being done every day and
I'm like only 200 years. So anyway u but
I think the more important point is that
it made mobile possible right if you if
you didn't have autocomplete people
would not be you know typing and pecking
on their keyboards it's going to be much
worse it would have had a hugely
dampening effect on e-commerce for
instance so this humble little
autocomplete has incredible incredible
impact on the world economy and the
other thing which I heard about I'm not
sure if it's 100% true but it's an
interesting example apparently the very
first iPhone keyboard that came out
right the soft keyboard not the hard
keyboard. Um they had some very basic,
you know, sort of word continuation
prediction going on. And so if if when
you start typing T and H, obviously it's
going to guess the E is going to come
next, right? So that part is old old
news, nothing new there. But apparently
the E letter in the keyboard will become
slightly bigger. So when your finger
goes towards it, it has a better shot of
actually connecting with it. Right? So
these kinds of things are used to change
the UI in real time in a whole bunch of
applications and you just don't even
realize it. All right. So uh and of
course we all know about uh LRM at this
point. So I asked it to write a
limmerick about the beauty and power of
deep learning yesterday and it says in a
world where data flows like a stream
deep learning is more than a dream.
Sifts through the noise with an elegant
poise unveiling insights that gleam.
Cool, right? All right. So let's get
back to work. Uh so NLP has
extraordinary potential for making
products, service and services much much
smarter. Uh and what I want to point out
here is that you know even if you focus
on this very very simple sort of
formalism right a bunch of text comes in
a bunch of text goes out that's it. If
you take that very simple text in text
out formalism this little humble little
thing has just an enormous enormous
range of applicability. Right? So
obviously you can send a bunch of text
in and ask it to classify it right for
mo you know sentiment route it for
customer support you can try to figure
out the intent of what the person is
asking in search you can filter it you
can content filter to make sure there's
no toxic abusive stuff going on I mean
the the possibilities for just text
classification are numerous okay but
that's a that's sort of a use case we
are all kind of familiar with right so
no surprise there now text extraction we
may be less familiar with here and the
idea is that you can actually look at a
lot lot of uh unstructured textual data
and extract all sorts of interesting
entities from it. Right? Hedge fun hedge
funds use it very heavily. They will
extract all sorts of company information
from news articles u and then obviously
doctor's notes. There are a whole bunch
of NLP startups that will take the
doctor's the doctor patient conversation
transcribe it and then extract disease
codes, diagnosis codes, medication codes
and things like that. Uh right. So the
possibilities for this are enormous. Of
course text summarization and we all
have been doing it thanks to chat GPT
right take text in and any kind of
summary that comes out of the text is
just text out okay and then text
generation of course we can take text
and do marketing copy sales emails
market summaries so on so forth and
including troublingly for educators
college application essays
code generation is a more subtle example
of text out because code is just text
right so text in text out also covers
was text in code out. Okay. And question
answering. So you can take a bunch of
text,
you can take a whole bunch of documents,
you can add a bit of text to it which is
your question and this whole thing at
the end of the day is just text it in
and then you can have and you can use it
to answer questions and therefore create
chat bots for all sorts of interesting
applications.
And you know if you look at this example
call centers that's that is where a lot
of money is being spent right now to
build these call center chatbots for
text and text out question answering and
so just if you drill into this right if
you imagine taking all the call center
transcripts and their internal product
documentation service documentation FAQs
etc stick it in you can start to answer
these kinds of questions okay yesterday
what are the top reasons why customers
were upset with us what interventions
made by the agent actually worked what
did not work, right? What characterizes
the best agents from the rest? How
should we grade this particular agent's
interaction with the particular
customer? How should she how should we
chain the call center script? How should
we coach the agent in real time? Every
one of these applications is aminable to
this very humble text and text route
model.
Okay. And so, and of course the
potential for is now everybody knows
this potential because of the advent of
large language models. Uh, by the way,
Google is uh released something called
Google Geminy 1.5 Pro u a couple of days
ago. Uh, and it's incredible.
It's incredible, right? And anyway,
we'll get back to that later. But the
point is that the kind of potential we
have is just amazing even for text and
text. Okay. And as you would imagine
>> this is all like though we are calling
it language this is all primarily
English right
>> now there are lots of multilingual uh
models as well uh there are multilingual
models by that I mean models which are
specialized to other languages
non-English languages and models which
are mult truly multilingual like
polyglot models as well and both of them
are available uh right now and many many
modern LLMs are actually trained from
the get-go to be multilingual in a bunch
of the what are called high resource
languages. Languages which are spoken by
lots of people. Uh but actually it's
funny you should ask that question
because this this Google Gemini model
that I just described they actually u so
there is a language called kalamang
which is spoken by 200 people in the
world and so a researcher had created a
one book which is sort of like a grammar
manual for kalamag right because there
are no other written works in that
language. And so what they did is they
took a whole bunch of English dialogue
and this book fed it into uh Google
Gemini 1.4 Pro 1.5 and it translated
into Calamong at human level
proficiency.
It had never seen it before. So that's
an example
of of this.
Yes. So the question is the question
text here is all the things you want to
translate from English to kalamong. The
documents here is just one document
singular the grammar book the manual and
then what comes out is a translation. So
these models even when they're not
explicitly trained on a different
language if you give them enough of sort
of grammar manuals and stuff like that
they may do a pretty decent job from the
get-go with no training.
It's kind of a shocker. Two years ago
people would be like that's impossible.
All right. So
back to this.
All right. And as you folks, you know,
may already know and maybe you're in
fact participating in this gold rush
already. Um, you know, lots of people
are creating lots of really cool
companies to take some of these ideas
and actually create really interesting
products and services out of them. Um,
so if you're not doing it and if you've
been thinking about entrepreneurial
stuff, here's a word of advice. Take the
plunge.
Dismissed. Just kidding. All right. So,
and as you can imagine, enterprise
vendors are rushing to add NLP to all
their products. Salesforce Einstein now
has Einstein GPT. Microsoft has
co-pilot. I mean, the list goes on.
Everybody, everybody's like scrambling
and really trying hard to infuse some
GPT magic into whatever they're doing.
Okay, some of it is real, a lot of it is
not. Uh, okay. So, let's go to like the
arc of NLP progress. How did we get to
this kind of crazy times that we live
in? Um so if you look at natural
language processing basically efforts to
take language and try to analyze
language and you do predictions with
language and so on and so forth. Um
the first phase of it was just
handcrafted rules based on linguistics.
So these are all linguists who would
really understand the grammar of a
language and then they would use a deep
knowledge of linguistics to figure out
all these rules by which you can process
and analyze natural language text. And
then this other thing came along which
was a statistical machine learning
approach which basically said never mind
all that complicated knowledge of
linguistics and grammar. Why don't we
simply count things? Let's count the
number of times these two will co-
occur. Now let's count that. Let's count
this basically just count a lot. Okay.
And let's see if it does right if it
does for predicting things for say for
classifying text and so on. And
shockingly those methods ended up being
really good. They ended up being really
good and in fact they actually were
better than the lovingly handcurated
linguistically driven rules. Okay, so
much so this is a famous quote which
says every time I fire a linguist the
performance of speech recognizer goes up
right obviously made in justest but
there is a kernel of truth to it.
So that was that's what that's that's
what that's where we were and then deep
learning happened okay um in 2012
roughly and then we had these things
called recurren neural networks which
are based on deep learning which
actually moved the ball forward and then
in 2017
something called the transformer was
invented
2017 and the transformer replaced
everything else across the board so we
just going to leaprog directly to
transformers in hodle we will not spend
any time on recurren neural networks and
that is not to say that they are sort of
dead. Um there's there's a very
interesting work which actually is
trying to now revive recurren neural
networks to make it work for these kinds
of modern LLM kinds of tasks but it's
still very early days. Okay. So for now
we'll just focus on transformers.
Okay. So the the very high level view of
the problem here is that like most
things in deep learning it's basically
fancy regression.
There is some variable X that comes in.
It goes through a bunch this goes to
this very complicated function along
with this W which is the weights and
then out pops an output. Right? That's
just the view that you've always had.
And so in this case X happens to be
text. Y can be text. It could be labels.
It could be numbers. It could be
anything else. The W is the weights. And
the function is a deep neural network.
Right? This by by at this point when you
look at this slide it should be like
blindingly obvious.
So now the key question here is how do
you actually represent X? That's the key
question for pictures for images. We saw
that we just took the pixel values which
were light intensity numbers between 0
and 255 and you could just use that
directly but when a when a sentence
comes in like I love deep learning like
what do you do right how do you actually
represent it because remember we have to
numericalize everything that's coming
in. So that's a key question and and
this actually is a very subtle question
very important question and we'll focus
on that today and then next week when we
look at transformers we will look at
what neural network architecture is best
suited to process this sort of text
inputs that are coming in right those
are the two big questions we're going to
look at all right so processing basics
we going to follow this very standard
process
this is the process by which we take any
any text that comes in and we do run it
through these four steps and this
process is called text vectorization and
as the name suggest that we are
essentially taking text and creating
vectors of numbers out of it right text
vectorization and we'll go through each
of these processes one after the other
so I just find it very useful to just
have this acronym stie in my head like
stie just keep that in mind it may be
helpful um all right so we what we do is
the setup here is that we have a whole
bunch of documents, right? We call it
the training corpus. We have a whole
bunch of text documents, text data. Uh,
and as far as we are concerned, you can
just imagine it as just lists of long
passages. Okay? What is a novel? It's
just a long passage, right, of text. So
whether it's a novel or a sentence
doesn't really matter. We just think of
them as a big list of strings, a big
list of text. Okay, that's a training
corpus. And what we do is we take this
training corpus and we run it through
and we apply standardization and
tokenization which I will describe to
this entire training corpus up front.
Okay. So we first do this and and
standardization is basically
the default for most applications tends
to be this which is we first strip
capitalization and make everything lower
case
and then we remove punctuation and
accents and so on and so forth. Okay,
that's the first thing we do. I'll talk
about why we do it in just a moment, but
the mechanics of it are we do this
first. Then we look at words like a,
the, it, and so on and so forth.
Basically filler words, right? Which
which we we need to actually make
complete sentences, but they may not
have any value predicting things. So we
remove them and they are called stop
words. And then finally we take words
which are very similar which have sort
of a same kind of stem or root and then
we just map it to like a common
representation like ate eaten eating
eaten all these things just becomes
let's say eats and we do that sometimes.
So this we almost always do this we
often do and this we do it sometimes.
Okay. Now, why do we do any of these
things?
>> I think we want to try to recognize the
essential thing with the word, right?
Whether it's eaten or eat, but the
essential thing is the eat, right? So,
we want to try to sort of abstract from
it the more essential thing,
>> right? So, why do we need to abstract? I
guess you're absolutely correct. We're
trying to abstract. Why is there a
benefit to doing this abstraction?
How about somebody from this side of the
room? Oh yes.
>> So I want to reduce the library.
>> Why is it a good idea to reduce the
library? The size of the library
>> because of the the amount of computation
needed. So that is part of the answer.
There's another part to the answer which
says all right let's swing to the right
um is it faculties comparison between
different sets
of standard
[clears throat]
>> okay so I will go with that but I think
the the key thing we want to uh the key
thing to realize here is that you want
the model much like when you go when we
talk about computer vision we said look
if it's vertical line. I want to be able
to detect it wherever it happens. I
don't want the model to think that the
vertical line on the left side is
different from the vertical line on the
right side and then later realize they
are the same thing because you would
have wasted valuable capacity learning
things which actually happen to be the
same because you didn't know it was the
same. So here if you for example take a
word and lowerase it clearly the case of
it whether it's uppercase or lower case
most of the time it's not going to
matter for anything you want to predict.
So you're essentially telling the model
you know the lowerase version uppercase
version they are not different they're
actually the same and the easiest way to
tell the model they are the same is just
make everything lower case so that is
the key idea okay and similarly if you
look at stop words the reason is that
these stop words may not help you
predict anything whether a word uh and
the showed up in a movie review probably
does not affect the sentiment of the
review and therefore let's remove it so
that's a slightly different reason
stemming is the same reason as the first
which is that all these words kind of
mean the same thing. We don't have to be
super precise about it and so let's just
like collapse them onto the same thing.
Now that these are all the standard
things we do there are totally notice
you know important exceptions to all
these things. Okay we'll come back to
the exceptions a bit later but that is
the standard thing we do make sense. All
right.
So if you look at something like this um
this sentence here right hola what do
you picture when you think of travel
Mexico boom and then you can see here
this is the standardized version like
everything has become lower case like
the h has become small h the punctuation
has disappeared that's part of
standardization and then uh travel and
you can see here that Mexico m has
become small sipping has become sips uh
things think has become things and so on
and so forth
So that's an example of strandization at
work.
Okay.
The next thing we do is something very
important and it's called tokenization.
So what we do typically is that okay now
we have standardized everything. We have
a bunch of words. Uh we need to now
split them into what are called tokens.
So the most common default is to just
think of a word as a token.
We just split on the white space, right?
You take each string and wherever there
is white space, meaning actual spaces,
uh, carriage returns and things like
that, boom, you just split on them and
you just create words out of it. So, so
for instance, if you have this
standardized sentence here, you just
split it after every word and you get
this thing. Okay? So, each of these is
now a token.
Now, this has some disadvantages.
What are some disadvantages of just
splitting on on the space between words?
Uh yeah,
>> I think we lose any context because we
look at each word separately. Uh so we
don't have any password or what happens
next,
>> right? So for example, the cat sat on
the mat and the mat sat on the cat will
have the same like set, right? Yeah. So
you lose the order. What are some other
issues with it?
for words that should have two together
like you lose the fact that that's one
name because you separated
>> right exactly so there are compound
words right like father-in-law for
instance that's one problem another
problem is that lots of non-English
languages they actually don't have this
notion of a space between words right
actually runs one after the other and it
is and the native speakers know from
context how to chunk it and break it so
well what do we do Right?
Because you basically will have one word
for the whole passage, one token. The
other problem is that there are
languages, German is perhaps the most
notable one in which you have very long
words.
Um I saw a word uh which I think I might
have it on the site somewhere this like
this long which means uh
you realize that something amazing is
happening but the rest of the world
hasn't woken up to it yet. It's that
feeling.
There's a word for that. Amazing, right?
Anyway, so yeah, some words or Japanese,
for example, there's a word called. Do
people know the meaning of the word
combi?
It means the transient beauty of
sunlight going through fall foliage.
There's a word for that. How cool is
that? Anyway, sorry. I love that word.
So, back to this. Um so we have this
thing here. So there are all reasons for
which splitting on the the space between
words not going to work. Okay. Um
so what we will so what happens is that
modern large language models. So the the
what we have described so far despite
its shortcomings is actually really good
for lots of NLP use cases. Okay. If you
want to classify text as good enough for
instance but if you want to generate
text like LLMs do it's not going to
work. It's not going to work because you
know when you ask the strategia question
it comes back with perfect punctuation.
Clearly punctuation was not stripped. It
comes back with particular upper and
lower case clearly that wasn't stripped.
You can actually make up new words and
ask it to use the new word. It'll make
it I'll use it. Therefore, it's not like
it can only recognize a finite set. So
there's a very clever scheme called bite
pair encoding right which is which is
invented to do all those things. And I
have slides at the end and if we have
time we'll talk about it.
All right, for now let's continue this
thing. So when this is done for every
sentence or every uh passage in our
training data set, we have now have a
list of distinct tokens, right? We have
a list of distinct tokens. In this
simple case, it happens to be all the
distinct words that we have seen, right?
That's called the vocabulary.
So now we move to the third and fourth
stages. In this in these stages, the
indexing and encoding stage, we only
work with the vocabulary. Okay. And so
what we do is the first thing the
indexing we assign a unique integer to
each distinct token in the vocabulary.
So for instance, let's say that you know
you took a whole bunch of English
literature as your training corus and
you ran it through you basically you'll
come up with English dictionary right?
So it'll have maybe starting with a all
the way to zebra a whole bunch of words.
Um, and so I'm just putting 50,000 here
because turns out the GPD family uses
something called 50,000 tokens. So I'm
just using 50,000. It's not the actual
number of words in the English language.
It's much more than that. So let's say
that we give a number one through
50,000. And then we actually also
introduce a special token called UN. It
stands for unknown. And we'll come back
to this later. And we give unknown the
integer zero.
Okay. So this what this is what we mean
by indexing. take the word the tokens
you have identified and just map it to
an integer.
Okay, that's the indexing step. Then
what we do is we assign a vector to
every one of these integers.
Okay, and that is the encoding step. We
assign a vector to each integer.
So you have a bunch of distinct words
and each word we put an integer on it
and then we take that integer and map it
to a vector. Yeah. Can you please
explain
to
>> Can you please explain what unknown
means?
>> Yeah. So, so I'll come back to that for
now. Just assume that we have a token
called unknown. And the way we are going
to use it will become apparent in a few
minutes.
>> Does it mean there's a base to it
though? There's like a letter or
something.
>> It's it's a it's a placeholder for
something else which I'll describe
shortly.
Okay. So, that's what we have. U so
let's say that we want to assign a
vector to each integer in our vocabulary
and let's assume that we have uh okay
let's say we have 50,000 possible
integers because we have 50,000 possible
words and we want to assign a vector so
that if you take the vector of two
different words they should look
different right clearly that's the whole
point of mapping from integer to vector
they better be different uh what is the
simplest way to come up with a vector
for each each of these tokens
the same as the index.
>> Sorry,
>> the same as the index. It's just a
vector one one by one with the index.
>> So, a vector of uh zeros and ones or
>> it's just a vector with one dimension.
>> Oh. Oh, I see. So, god. Well, it's it
it's creative, but it's a little
cheating, right? Because you're
essentially putting a square bracket
around the number and saying it's a
vector. Good try.
>> You can try one hot encoding,
>> right? You can try one hot encoding.
So remember the list of distinct tokens
you have, you can just think of them as
the distinct levels of a categorical
variable,
right? And you can just use one hard
encoding for it.
So what you can do is you can the
simplest thing is do one one hard
encoding and the way it's going to work
is that if you have let's say 50,000
uh 50,000 possible values the vector is
going to be 50,000 long it's going to
have zeros everywhere except in the
index value of whatever that token is.
So for instance since we said ank is
going to be the first uh first number
zero it has a one here and the zero the
zero index position has a one everything
is zero a happens to be the second one
so it happens to be one in the second
position zero you get the idea
okay
>> so this real zero hot encoding we can do
the zero hot one coding one hot encoding
and so so the dimension of this encoding
vector how long it is it's basically the
number of distinct tokens that you have
seen in in the training corpus plus one
for this unk thing that you'll get to.
Okay,
so that is a dimensional encoding vector
which is this is called the vocabulary
size.
It's called the vocabulary size.
All right. So at this point we have
created a vocabulary for the training
data training corpus. every distinct
token vocabulary has been assigned a one
hot vector and we are done with basic
pre-processing.
Okay, so all the text that has come in,
every token has been mapped to some one
hot one potentially very long one hot
vector.
Any questions on the mechanics of this
before we continue on?
>> Now let's see if when you get a new
input sentence in a new sentence freshly
arriving and we want to feed it into a
deep neural network, how will this
process actually apply to the new
sentence that's coming in? Okay, so
let's assume um that we have completed
our SDIE on the training corpus and it
turns out we found only you know 99
distinct tokens 99 distinct words and
then we add this ank thing to it so we
got a 100 okay so this is our vocabulary
it starts with ank a and then goes all
the way to zebra but there are only 100
of them in total right and just to be
very clear we didn't bother to do things
like stemming and stop word removal and
stuff like that which is why you have
words like 'the' showing up in this
list.
Okay. All right. So,
let's say this input string arrives, the
cats are on the mat, and then we run it
through STIE. So, the cats are on the
mat goes through this thingoop.
Then the output is going to be a table
with a bunch of rows and a bunch of
columns. Any guesses
how many rows and how many columns?
Just raise your hands. I'll call on you.
>> Yeah, you use a microphone. Go for it.
>> Yeah, I would guess uh 100 rows and uh
six columns.
All right, we'll take a look. Uh
100 and six as well as six and 100 are
both correct. So, so the way I've done
it is six and 100. And the and that's
exactly right. So, the idea is that this
is your vocabulary, right? So, the word
the cat sat on the mat once you change
the case of it, it becomes like this.
So, the the happens to be a one hot
vector with a one where there is a the
and zero everywhere else. I'm not
showing all the zeros because it'll get
too cluttered.
Similarly, cat has a one where the the
cat position is and zero everywhere else
and so on and so forth. Does that make
sense? So, the the phrase the cat sat on
the mat came in as just whatever six
words and then it became this you know
600 entry table.
Okay. Now, what is the best way to feed
this table to a deep neural network?
What can we do?
It's not a vector. It's a table.
If it's a vector, we know what to do. We
just feed it in. We'll just maybe send
it to some, you know, hidden layer and
declare victory at that point.
>> Yeah.
>> You would like to flatten it. And like
how how might you do it?
Flattening is a reasonable answer by the
way.
I think you mean you just have to like
take each like each column
take the first one each row and each row
each word kind of like
>> yeah so basically you can take all the
first columns and then take the second
column and attach it under the first
column and so on and so forth right so
we can certainly do that and that's very
akin to how we work with images right u
but there is one downside to that what
is that downside
uh Um,
>> it's pretty long. Like I wonder if
instead you could for the first word
it's one, for the second word it's two,
and then you maintain the order, but you
still keep it just as like one row.
>> One row. So one issue, so we'll come
back to what we do about this, but what
you're pointing out is it could be very
long, right? Because if each word is a
50,000 long one vector with just six
words, it becomes a 300,000 long vector.
Imagine take the 300,000 long vector and
sending it into a 100 hidden unit hidden
layer. 300,000 times 100 parameters. Too
much can't learn anything.
So that's one issue. The other issue is
that different length texts that are
coming in will have different sized
inputs.
So here the cat sat on the mat has six
times 50,000 but maybe the cat sat on
the mat and the rat rat ran over to the
cat becomes even longer. We can't handle
variable sized inputs.
the inputs all have to be mapped to the
same length.
That's another problem.
>> So maybe you can count how many you can
sum the columns basically and count how
many times each word appears since
you're using the like spatial
relationship.
>> Yes. So you Yeah. So both you and are on
the same sort of trajectory which is
that uh we need to somehow take this
table and make it into a vector. And
there are many ways like what you folks
are describing to make it into a vector
and turns out um this is all the things
that we've been discussing so far the
varying length ratio and so on. So, so
what we can do is we can aggregate all
these things. If you just add them up,
this is what you described. I believe
it's called sum encoding.
And if instead of adding you just or
them, meaning if you look at the column
and say, is there any one in this
column? If there's a any one, I'll put a
stick of one, otherwise it's a zero.
It's called multihot encoding. So, so if
you look at this thing, if you literally
just go column by column and count
everything. Okay, there's a one here,
one here. Oh, wait. There are two twos
here. So you put a two. That's count
count encoding. Multih hard encoding. It
just looks for any ones and puts on.
Make sense? So by the way there are many
ways to take these tables and make them
into vectors. These two happen to be
very commonly used and they kind of make
common sense.
Okay.
Right. So this aggregation approach that
we just described is called the bag of
words model.
Bag of words model. And the reason is
that first of all this bag that we have
has words either it counts whether a
word exists or not or it counts how many
words how many times the word has
appeared right count versus multihot
versus sum encoding count encoding but
more importantly and this goes back to
your observation is that we have lost
the order of the words now whether the
phrase came in was the cat sat on the
mat or the mat sat on the cat the count
encoding and the multih hard encoding
are exactly the same. There's no
difference because we're just looking
for the the presence or absence of
words. That's it. We don't care in what
which order they appear, right? That's a
huge limitation, but shockingly for many
applications, it doesn't matter. It's
good enough. So, it's called the bag of
words model.
All right, so this called the bag of
words model.
Um, now does it have any shortcomings? I
already talked about the first
shortcoming which is that it loses
sequentiality the order we lost this
order information right uh we we lose
the meaning inherent in the order of the
words what are some other issues with it
what do you mean by that
>> right so there are lots of zeros not
that many ones so you have it's a very
sparse amount of information but maybe
is carrying around a lot of information
to to make it all work. Now there are
some tricks CS computer science tricks
to handle sparsity in some clever ways
but it is certainly an issue. Now the
other issue is that let's say the
vocabulary is very long.
Each input sentence whether it's the
collected works of William Shakespeare
or the phrase I love you will have the
same length input.
Is that the same length input
because ultimately every incoming thing
gets mapped into one vector. Okay, that
feels a little sub suboptimal.
Clearly the collected works of ins have
a lot more stuff going on in them.
Right? So that's a problem. In
particular, very very small things that
come in, you'll be spending a lot of
compute on those long vectors and
processing them. Um, now you can
mitigate some of this by choosing only
the most frequent words. You don't have
to take, you know, I think the English
language I read somewhere has roughly
500,000 words or so. Uh, but turns out
the top 50,000 most frequent words are
responsible for just about everything
you're going to see ever. And the other
50,000 are what's called the long tail.
They almost never happen, right? You
never see them. So, you can be very
pragmatic and say, "I'm not going to
take every little word that I see in my
vocabulary. I'm going to only take the
most frequent words. I'm just going to
ignore the rest.
I'm just going to ignore the rest."
Okay?
But if you ignore the rest, let's say
the there is one word uh let's take some
Shakespeare word hamlet. Let's let's
assume that you ignore the word Hamlet
from your training corpus. You just
delete it because it's not one of the
top most frequent things you have seen.
And then somebody sends you a text
saying, you know, Hamlet was a bad
prince.
Analyze the sentiment of the sentence.
Well, when you see Hamlet, what is your
system going to do?
It's going to look at the Hamlet and
say, I can't see it in my vocabulary
anywhere.
And if it can't see in the vocabulary,
what is the only thing it can do?
Replace it with Unk. So that's where
comes into the picture.
So whenever it can't see something in
the vocabulary in a new input, it just
replaced with ank. Which means that
if you had ignored Romeo, Juliet, and
Hamlet in the in the training corpus,
all of them are going to be replaced by
the same ankh, which means that we can't
distinguish between them anymore.
>> So is this whereation
comes into play here where it doesn't
recogize
H interesting question. This is
whereation comes up. Actually, as it
turns out, no, as we will see when we
talk about LLMs later. Uh LLMs actually
will not have this UN problem because
they use a different tokenization scheme
which can handle anything you throw at
it, including new stuff you just made
up.
So, we'll come back to that.
All right. Um so, that's what we have.
And so what we're going to do is despite
its shortcomings, bag of words is
actually a really good default for many
NLP tasks. Uh and in the spirit of do
the simple stuff first and do
complicated things only the simple
doesn't work. We'll use a bag of words
model right now. Okay. So we'll switch
to a collab and see how it's done.
So here the the application we're going
to work with is kind of a fun
application. Uh we're going to try to
predict the genre of songs.
Okay, it's a nice classification use
case. Um, so we want to take some
arbitrary song and then classify it into
either hip-hop, rock or pop.
Okay. Um, and so for instance,
right, this is the kind of lyric you're
lyrics you're going to see. And as you
will see in this data set, the data set,
just a quick word of caution, uh, the
data set does have lyrics which may not
be sort of, you know, safe for work as
it were. So I'm not going to be like
exploring the lyrics in the collab, but
I just wanted to be aware of it. Okay.
Um, so but it's just some data set that
we downloaded from somewhere, right? Uh,
it's got all these lyrics. Okay. So
we're going to try to classify each
verse that we see into one of three
things. Hip hop, rock or pop. It's a
multi-class classification problem.
All right. Actually, what is the
simplest neural network based classifier
we can build
for this problem?
All right. So what is the simplest
neural network we can build for this
problem? So remember what is the input?
The input is going to be a bunch of song
lyrics. It's going to be a really long
song for all you know, right? And we're
going to use the bag of birds model. Uh
and let's assume for a moment that we
will use multihot encoding, right? We'll
create a vocabulary from this for the
song. We'll take all the songs. We'll
process them, run it through STI. will
do multihod encoding which means that
every song that comes in will have will
be a vector that's how long
it'll be as long as the
correct as a vocabulary size right so um
so maybe what comes in is this phrase um
since it's supposed to be songs I'll say
something which is probably common to
90% of songs I love you
okay that goes in
it goes into our ST STIE process
and then this SDIE process gives us a
vector which is X1 X2 all the way to XV
where V stands for the size of
vocabulary. Okay. So that that's our
input layer
all the way. So knowing what we know now
about deep learning what can we do next?
Couldn't you or maybe I'm getting ahead
but wouldn't the classifier just be like
the baseline would be classify it as the
most common genre?
>> That is the baseline. Correct. Correct.
I'm just saying and we'll come to the
baseline a bit later. But here I'm
saying suppose you need to you wanted to
build a neural network model for this.
How would you set it up?
>> You think about the layers that you
want,
>> right? And what is the simplest thing
you can do with a neural network? How
many layers?
>> Uh no layers. Well, then it becomes
problematic with even a neural network
because it could just be logistic
regression
>> one hidden layer.
>> Yes, thank you. I'm being a little
squishy about this because there are
some people who be like well even if
there's no hidden layers if you're using
relus and this and that and sigma that's
maybe it's a neural network and I don't
want to get into that how many ages in
the tip of a pin argument. So um so yeah
we need one hidden layer right in this
course we need at least one hidden layer
for it to qualify as a neural network.
Okay, so let's have a hidden layer and
we'll have a bunch of ReLUS as usual.
Okay, bunch of ReLULS and I'll ignore
all the arrows between them. It's kind
of a pain. U and then we come to the
output layer. And what should the output
layer be?
How many nodes do we have need in the
output layer? Three, right? Hip-hop,
rock, whatever. Pop. So we And then that
layer is called what? What activation
function?
Softmax. Perfect. Love it. love this
class. All right, three things. Uh,
rock, hip-hop,
and uh, pop, right? And this is a soft
max right there.
And then it's going to give us three
probabilities that add up to one because
it's a soft max. So that's our basic
network, right? Perfect. Yeah.
>> Why do you need those probabilities?
Again, if you just want to identify the
most likely genre, the soft max just
give you a way to kind of add them all
up once. Why do you need soft? Why don't
you just take the max value and say it's
that?
>> Oh, interesting question. Why can't we
just produce three numbers and grab the
maximum number? So, it turns out finding
the maximum bunch of numbers that
function
is not very it's not very friendly for
differentiation.
And ultimately you want to take this
output, run it through a loss function
like cross entropy and then be able to
run back prop on it. And so
fundamentally back propagation is just
differentiation and it requires
everything inside of it to have well-
behaved gradients. And so this little
max function is actually not well
behaved and which is why we have a soft
version of it soft max which makes it
easy to differentiate. So I can tell you
more about it offline but that's sort of
the quick synopsis.
So a lot of tricks you will see in the
neural network literature or ways to
avoid this the problem of having certain
the like the obvious choice of function
will not be well behaved for
differentiation. That's why you need to
go through all these other mechanisms
much like we couldn't just say accuracy.
Why don't you just maximize accuracy
instead of doing this cross entropy
business? Same reason.
All right. So let's come back here.
All right.
So that's what we created on the thing.
Right? Cats out of the mat vocabulary
thing and so on. And I you know I was
playing around with it uh earlier and so
I I found that you know eight relu
neurons were pretty good to get the job
done. So I'm just going to go with eight
rel
neurons in the hidden layer.
So I think that brings us to the collab.
Yeah. So let's switch to the collab.
All right. So um that's what we have
here. We you know there's a little bit
of verbiage here which just describes
what I just talked about. So we'll do
the usual things and upload everything
uh import everything we want. TensorFlow
and caras and the the holy trinity of
numpy pandas and mattplot lib. Uh set
the random seed as usual at 42.
This is our SDIE framework here. And the
nice thing is that all four of these
things SDIE are beautifully implemented
in Keras is a single simple layer called
the text vectorzation layer. Okay, which
is nice. Um, so we have the text vector
right here. And so in our first example,
what we'll do is we will use a default
standardization which will just remove
punctuation, convert to lowercase. We'll
use a default tokenization which just
means split on the space between words.
And then we will set the output to
multihart. Right? All the things we
talked about, KAS will just do it for
you automatically. And so output mode
multihart standardize this spread whites
space and boom, you run the text
vectorization thing. And once you do it,
KAS creates this textualization layer
with these settings and it's now ready
to swing into action. So what does swing
into action actually means? Well, now we
need to actually feed it a training
carpass so that it can do all the things
it's supposed to do and create the
vocabulary for you, right? So um so and
that thing is called the adapt method.
So we create a tiny training corpus for
us. This is our data set. Um right this
just a bunch of words from some of these
lyrics. And then what we'll do is we'll
take this layer that we just defined
here that we have set up here. And then
we will ask this layer to actually
create the vocabulary using this adapt
command. Okay. Index the vocabulary. And
it's done. And once it does it, you can
actually ask it for the vocabulary.
Okay, this is the vocabulary using the
get vocabulary command. And so first of
all, how long is the vocab? 17 17 words,
17 tokens. What are they?
And see here, and you can see these are
all the words and you can see it is
stuck in an in the very beginning,
right? It's sort of the default. By the
way, uh just a little programming tip if
you're not familiar with if you don't
have a ton of programming experience. If
you want to, you know, print these
Python objects like list and all in a
pretty way, one trick that often works
is just stick it into a data frame
and then print it. Usually, it'll print
it in a much better way. So, you can see
it like that.
So, you can see here ank arrays blah
blah blah blah blah. And you can see
integer zero assigned the ank token. By
the way, how come it picked the word
arrays as the second entry? Why not
something like an or um you know why
not? Why not a how come a is not chosen
as a second entry? Why why did it pick
arrays? You think
>> maybe maybe it tried like the words that
are most influential on the meaning of
the sentence to be on the
But it at this point it doesn't know
what we're going to use it for.
So it has no way to know what word is
useful because we haven't told it how
we're going to use it.
But but you're kind of on the right
track. So what KAS does is it'll
calculate it'll find all these tokens
and then it'll actually just sort them
by frequency.
So the most frequent as it turns out in
those four sentences we gave it happen
to be the word arrays. That's why arrays
is showing up on top. Um, and you can
actually confirm this by going to the
our little data set and you can see here
array shows up here and was up here
twice and that's why it came up on top.
Okay. All right. So that's what we have
and u and now now that we have populated
this we can run any sentence through it
easily. Yeah.
>> Does [clears throat] it matter that it's
on the top or is it just
>> it doesn't matter. It doesn't matter.
The reason why it's helpful later on is
because suppose you tell Kas hey don't
take every word you see here give me
only the most frequent 100 words I don't
want any more than that it can easily do
that that's the reason yeah
>> this is just a vocabulary so basically
you you give it all this phrases it
happens just four phrases in our example
and then it finds all the distinct words
and you know does all that stuff and and
then it has created a vocabulary. At
this point the the training corpus you
fed it will is forgotten and the only
thing has survived this processing is
just the vocabulary. That's it. Now we
have to start applying it to any kind of
text we want to use it for.
So here when you come back here u so
this is what we have and so what you can
do is you can take any sentence and you
can just run it through a layer and to
make sure that actually is doing the
right thing for you. So we'll take the
sentence, we will then run it through
the text vectorization layer by just
passing that sentence into it and then
we can just print it.
So now it's giving you a tensor. This is
a multihot encoder tensor with all these
ones and zeros. So note that this tensor
is 17 units long which is which is a
good check because our vocabulary is 17
long. So it's better match that. Uh now
recall that the ank token is at the
first location. It's at index zero and
it says that this encoded sentence does
have an unk word.
Okay. So
why is that? What is this UN word?
Anyone can guess?
Well, it turns out to be the word still.
Um I think yeah still is not in our
vocabulary because the four sentences
which is our training corpus used to
build vocabulary. They had a lot of
write and rewrite but there was no still
in it anyway. That's why there's an UN
ank for it. Uh we can just double check
that by asking Python is it is it
vocabulary? Nope, it's not. Okay. Now,
in the spirit of making small changes to
the code to understand what's going on,
which is a very useful tip for folks who
don't have a ton of programming
knowledge. Let's say that you send the
phrase Sloan Hodddle and DM, DMD. Uh I
think you will agree with me that none
of these words is in the training
corpus, right? So what will this what is
the multihot encoded vector for this
phrase sloan hodddle BMD
three
it's not count encoding it's multihod
encoding
right it's going to be 1 0 0 so you can
see here or in this case remember the
vocabulary is 17
right so each of these words is going to
be a one followed by 16 zeros
And then it's going to multih hot encode
them which means the three ones in the
column just become a one. So so you
still have only this one. Okay. All
right. Good. So now let's see that's now
let's actually get to the the the data
set. We have this 90,000 songs. Uh and
it's in this little thing here. Uh we
have grabbed the data and cleaned it up.
Cleaned it up meaning like formatting
wise not content wise. uh and then we
stuck it in this uh data frame and it's
we already have divided into train, test
and validation for your benefit. So you
don't have to worry about it. So turns
out we have 40 almost 49,000 songs in
the training set, 16,000 songs in the
validation set and 22 roughly 22,000 in
the test set. Okay, lot of songs. It's a
lot. It's a big data set. Um so let's
just look at the first few.
So oh girl, I can't get ready. We met on
rainy evening. Paralysis through
analysis.
Okay, that I can relate to as a data
science person. But anyway, u but uh by
the way this uh these things are very
useful for exploration of any uh data
frames that you might have. Collab is a
collab feature just check it out. Um so
anyway, that's the first few the first
few rows. Let's look at the last few
rows.
Okay,
you never listen to me as pop. Beamer
Benz is hip-hop. Yeah, of course.
So, okay. Uh, now to go back to the
question of, okay, um, what could be a
good baseline model? We need to
understand the proportion of these three
classes of songs. So, we'll do a quick
check. Turns out rock is 55%. So, if you
had to just guess something just
naively, you would just guess everything
to be rock and you'd be right 55% of the
time. Uh so now uh by the way the the
the target variable which tells you
whether which of these three genres it
is uh is is is a is actually a dummy
variable. So we need to one hot encode
that right. Um so we'll just turn that
this way using the pandas get dummies
function. And when we do that uh this is
y train which contains the dependent
variable. And you can see that is one
hot encoded now. Uh 0 1 0 0 1 0 0 1 and
so on and so forth. That's it. So I
think the first I forget it rock,
hip-hop, rock, pop or whatever. It's in
some order. We'll we'll get to that
later. So it's one hot encoded as well.
So that is as far as the data
downloading and setup is concerned. Any
questions?
>> Yeah.
>> Uh this kind of goes back to the
transfer learning concept. But do you
always want to build your corpus based
off of the vocabulary of your training
data or could you have like a
pre-ompiled like somebody's already made
like a list of the 50,000 words?
>> That's a really good question. Uh
unfortunately I'm going to punt on it
for the moment because um with modern
large language models a number of these
NLP tasks for which you had to sort of
roll your own and build your own thing
can now be very easily done using large
language models without even any further
training.
Case you pay for it is that you have to
use a large language model which means
you have to pay somebody an API call and
things like that and there are other
issues with it. uh but
we'll talk a lot about transfer learning
for text when we come to a little later
in the NLP sequence. So if I forget
please bring it up again.
>> Yeah.
>> Um quick clarification on the encode
factor. If I post it as floats not ins.
If it gets incredibly long wouldn't that
eat into compute time? Is there a reason
why it's floats?
>> Yeah. So uh question is that when when I
showed you that tensor the it is
actually is written as a continuous
number right a float floating point
number but we know these are one zeros
and ones so why can't we why do we have
to waste compute capacity by telling the
computer that these are all big
continuous numbers when it's just a zero
one there are ways to optimize that but
these problems are so small we just
don't worry about it but when we come to
something called parameter efficient
fine-tuning lecture maybe 10ish uh we
actually exploit that particular fact to
make things faster
Okay, so that's what we have. Uh, so
we'll we'll do the bag of birds model.
Um, by the way, there's a whole bunch of
stuff here. It just repeats what I've
been telling you in the lecture. So feel
free to read it again, but we can ignore
it for the moment. And now there's a new
thing we are doing here. So we are
basically saying, look, instead of
taking every word you see in these
49,000 uh songs in the training corpus,
uh, it's going to be too many words.
just pick the 5,000 most frequent words
and that's what this max tokens stands
for. Okay. And so we tell it uh all
right do this thing max tokens 5,000
sorry not 50,000 5,000 and still do
multihart and we are not explicitly
saying the standardization and all that
stuff because the defaults are what
we're going with. Okay. Yeah.
This is for making it more efficient.
Like this is like don't waste your time
on these thousand sports. Use them more.
Use them. Just focus on that to make
more efficient.
>> Make more efficient. But there is a
related and important point which is
that fundamentally the number of tokens
you allow this layer to have dictates
the size of your vocabulary and the size
of your vocabulary dictates the size of
the vector that you feed in. So shorter
vectors are better than longer vectors.
That's the efficiency point. The other
point is that the longer the input
vector, the more the number of
parameters the network has to learn
because the first layer itself is the
size of the input times roughly times
the size of the hidden layer. So this
thing becomes 10 times as long. You have
10 times as many parameters to learn and
given a finite amount of data, right?
The more parameters you have, the worse
it's going to do when you actually start
using it in the real world. It's going
to overfitit heavily. That's why you
need to be very careful.
Okay.
Yeah.
So, um, you downloaded the data set, but
are you still using the vocabulary the
17 words or did you
>> No, no, I'm that was just for fun. I'm
going to actually build a vocabulary
now. It's coming. Yeah, good question.
Yeah. So, all right, let's do that. Um,
so I first, you know, I defined this
layer. Uh, okay. I just defined it. All
right. Now we actually build the
vocabulary by essentially telling it to
adapt the layer using essentially the
full all 15 basically 49,000 songs in
the training data set right that's a
long list of songs as far as kas is
concerned you're just looking for a list
of strings so you just give it the list
of strings instead of four we're giving
it 49,000 the same uh philosophy applies
so we run it
it's obviously going to take a few
seconds to do that because it's 49,000
songs
five seconds. Uh, all right. Let's look
at the most common 20,
right? We get the vocabulary from our
layer. See, once you adapt the layer and
has built a vocabulary, the layer is
sort of been populated with all this
information. So, you can query it. So,
you can get the vocab top 20 words, the
most frequent word, no surprise, u, I,
blah, blah, blah. Uh, let's look at the
last few.
Dagger cheddar
verified
moving on
right and then we so once we have done
that now we actually can vectorize all
the data sets we have using this and by
vectorize you mean take every string and
create the multihot encoded vector from
it uh yeah
>> are we doing stie because we're keeping
stuff like d a etc. Yeah, we are not
strictly doing STI or to put it
differently the S stands typically S has
lower case uppercase strip punctuation
stemming stop word removal here the
default in KAS happens to not do
stemming not do stop word removal so
we're just going with the default thanks
for the clarification
and in fact in practice what I find
these days is that don't even bother to
stem don't even bother to remove the
stop words it's going to work well
enough
okay so all right uh okay so now Each
phrase is a vector. How long is this
vector? Each song is now a vector. How
long is that vector?
5,000. Correct. Because that is a size
vocabulary. Correct.
It's max tokens long, which is 5,000. So
if you actually look at X Oh, wait,
wait, wait, wait, wait. I haven't done
this thing yet.
It's going through 49,000. It's going
through another what? 23,000. Fine. So
let's run it.
Okay, now we can see X train which is
all the training data you have has is a
tensor is a table with 48 991 rows and
each row is a 5,000 long vector.
All right, good. Now we will try the
simple neural network that we wrote up
in class. So and now at this point this
code should be sort of second nature,
right? Isn't that cool? It's so easy to
write the write the thing the power of
abstraction. So uh we take kasin input
as usual input layer we tell it what is
the size of each thing that's coming in.
Well the size of each thing is a 50 max
tokens long vector. So we tell it the
shape is max tokens and then we run it
through a dense layer with eight relus.
Okay I'm hurrying.
So we get the outputs then we string the
inputs and the outputs into a model and
then we summarize the model. That's it.
So we go here and this has 40,000
parameters and you can see here right
when you go from the input the 5,000 * 8
that gives you 40,000 plus the eight
neurons have a bias coming in that's
another eight so you get 40,0008 okay
and we compile it as usual we use atom
as usual and because now the the output
y variable the y train variable is now
it itself is actually one hot encoded
right 0 1 0 0 1 depending on pop rock
and so on and so forth. We don't use
sparse categorical cross entropy. We
just use plain old categorical cross
entropy here. Okay. And this was
explained in lecture last week. So you
can revisit it if uh if it's if it's not
familiar. We again report accuracy,
right? So let's compile it. And we've
got a model. So we just run it for 10
epochs with a batch size of 32. And
because we have validation data already
supplied to us, we don't have to tell
Karas take the training data and keep
20% of it aside for validation. We can
literally tell it what validation to
use. That's what we're doing here. Okay.
All right. So, it's running.
Um,
it's pretty fast.
Any questions so far?
>> Yes.
>> The microphone.
>> How do we decide the max total? like
define the number of 5,000 here but we
do not know how many words would be
there in the entire text.
>> Yeah. So it's a good question. How do
you decide on this the maximum
vocabulary? What you typically do in
practice is that you actually you do it
without the max tokens and then you see
how long the vocabulary is and then you
actually get statistics on how
frequently the very infrequent words
actually show up. And then you'll
typically see like a dramatic fall-off
at some point and you pick that fall-off
point and then set that to be the max.
Uh all right. So perfect. Let's test it.
Uh accuracy is pretty good. 87% on the
training and 73 on the validation. We'll
do it on the test set. All right. 72%.
So we saw earlier the the largest class
of the three-way is a rock with around
50%. So the naive model is going to get
50% accuracy and this little neural
network model gets you 70 72% which is
pretty nice. Okay. So now let's actually
kick it up a notch and make it slightly
more capable. So the key thing here is
that uh as was has been observed in
class already when you go with a bag of
words model we lose all notion of order
right the word order clearly matters and
we're kind of ignoring it. So what we do
to get around it is um so actually this
actually really interesting uh sentence
here. Let's say this is a movie review.
Kate Vinclet's performance as a
detective trying to solve a terrible
crime in a P small pin tennos is
anything but disappointing.
Tricky tricky thing, right? Because if
you look at the word separately, the
word terrible and disappointing like
negative sentiment, right? But then if
you actually know that the word terrible
respon refers to the crime, not to the
movie or anything but disappointing
changes the meaning of the word
disappointing, you will see obviously
it's a positive review, right? So
clearly the the the words around the
word provide valuable clues as to how to
interpret that word. And so what we do
is how can we make our little model a
bit more capable of recognizing the
context around every word. And the way
we do it is something called bgrams.
Okay. And what for biograms what we
basically do is instead of taking
instead of just taking each word we take
each word and we further take every pair
of adjacent words
and those become our tokens and because
we take two adjacent words right it are
called bgrams you can take three adjent
words trigrams you get the idea engram
grams okay so that's the idea of bgrams
and so um so for example if you had the
cat matt sat on the cat sat on the mat
you will have the the cat cats sat you
get the idea right uh that's what we
have so let's do a little example and
kas makes it very easy you literally
tell it engram grams equals 2
bs and now by by from this you auto
immediately should know that engram
grams equals 1 is the default that's why
we didn't have to specify it okay so you
run it and then you do
cats on the mat is your training corpus
and then you get the vocabulary and you
can see here, right? It has created all
these nice biograms for you. And so
that's it. All right. Now, what we do is
we'll go back to the songs and we
actually tell Keras to not just take
each word, but take all the biograms as
well. And hopefully you'll do a better
job, right, of figuring out what the
sentiment is. And now because you know
when you have when you when you say,
okay, take the top 5,000 words, that's
great for single unigs as they are
called. But when you have biograms, you
have 5,000 possibilities for the first
word, maybe 5,000 for the second word,
right? That's a lot of possibilities. 25
million. Now, most of the 25 million
possibilities are not going to show up
in the data. So, you don't need to
actually make it much larger, but you
should make the vocabulary a bit more
than 5,000. So, here we go with say
20,000, right? Otherwise, it's the same.
Still multihart. So, let's run it. And
now we will run this. Now that the layer
has been set up with all the right
settings, we'll ask it to create the
vocabulary. Okay? again by doing exactly
what we did before. Create the
vocabulary
seconds
by triagrams all of them will get much
more computer intensive that's why
you're seeing this. So all right let's
look at the first 10 words. The first 10
words are all just single words and
that's not surprising because the single
words are going to be the most more
frequent right u
and then the last few
your mom your god you short you hell
all right let's just uh you know uh
index the whole all the data we have the
training validation test sets using this
vocabulary
Perfect. Now we come to our second model
where we say the shape the incoming
shape is now 20,000 long right because
we increase max tokens from 5,000 to
20,000. So each thing is a 20,000 long
vector otherwise it's the same and now
we will use this thing called dropout
for the first time which is a
rigorization thing that I have referred
to earlier that I never really described
and I will describe today if we have
time but I'll first run through the
whole demo. So just you know just you
can just you think of dropout as just
another layer you can insert and it's
essentially a great way to prevent
overfitting. So I just routinely will
use it and I'll talk more about it. So
for now you have this dropout layer in
the middle. It receives the input from
the dense layer and then sends it to the
output layer. The output layer is
unchanged. It's a three-way softmax.
Same model as before. Okay. And now uh
all right we'll come back to drop out.
So we'll compile it the same way as
before and then we will we will I will
just fit it for three epochs. Um if
you're interested after class later on
you can actually try it for more epochs
and see if it does better. Uh for now in
the interest of time we'll just do it
for three
right
I think 72% right was the uh the single
word unig thing we had.
>> If you're rerunning this code with the
same number of Do you ever expect the
accuracy to change?
>> Um if if you had to run this code in
your machine, you would expect it to be
roughly the same, but there are some
minute differences due to hardware and
device drivers.
>> If you rewrite it on your own machine
twice, would you expect a change?
>> That's actually a very tricky question.
Uh because it depends on what else I
have been doing in that notebook.
If I start fresh and do nothing but
that, typically I get the same numbers
typically. But for some reason I don't
get it exactly the right.
Okay. So we come to this. Let's evaluate
our little model.
Okay. 75%. So it went from 72 to 75.
It's actually a meaningful jump just by
using biograms. Okay. And I ran it only
for three epochs. If you run it for 10,
maybe it's going to do even better. All
right. So that is the beauty of this
thing. Now let's just actually do a
little demo. Uh we'll try to predict
some lyrics. Okay, I'll try another one.
Bites the dust.
It's a rock song. I think that's
correct. Yes. Okay. Okay, folks. Your
turn now.
Uh, somebody tell me your favorite song.
>> Dancing Queen from Aba.
>> I love ABBA. That's awesome. All right.
Okay.
Uh, Dancing Queen
Rex.
worse one intro. I don't like that.
Let's just go to something without all
this metadata.
Right.
All right. I'll just take the first
page. Okay.
Are we good?
All right,
down model. Let's predict
pop just about. Yay.
All right. So, uh yeah. So, that's
basically the model, but we have five
minutes. I want to get back to you can
play around and put your own lyrics in.
Uh typically what happens is that the
last two years that I've been doing this
particular lecture, I've noticed that
the songs are always rock songs for some
reason.
>> First time I'm getting a pop song from
the from a group that I actually like.
So thank you.
Uh all right. Uh let's go back to
dropout.
So the idea here in dropout is that you
know you have all these the input comes
in, it goes through a hidden layer and
so on and so forth. What the dropout? So
dropout is a layer and you put this
layer just like you use any other layer.
And what dropout does is that it takes
all the things that are coming into it
from the previous layer and randomly
decides to replace that number with a
zero.
That's it. It drops that number and
replace it with a zero. Okay? But it
does it randomly. It basically toss a
coin and the coin comes up heads zero.
If it comes up to us, let it through.
Pass it through. Okay? And the reason
why this is very effective is because
you can imagine all the neurons in a
particular layer when they overfit to a
particular data set the overfitting
happens because the neurons essentially
collude with each other right they sort
of collude with each other to actually
overfitit and predict things in sort of
a very accurate way. So you want to
break any sort of collusion between the
neurons, right? I'm obviously using sort
of like a you know again theoretic way
of describing it but the idea is that
any kind of speurious correlations in
your data neurons can pick it up by
being correlated themselves.
And so the way you avoid the spurious
correlation is by dropping neurons
randomly. You just kill the neuron
randomly which means that no neuron can
depend on another neuron being
available.
I know it's a bit grim but that's the
basic idea of dropout and apparently the
story goes that the the folk person who
the team that invented it Jeff Hinton
who won the touring for the stuff not
for not for dropout just for deep
learning um he said I don't know if it's
true but he said that apparently he got
the idea when he went to a bank and
realized that you know very often the
bank the folks who working in that bank
branch that he used to go to kept
changing
right they were never sort of the same
the people would be transferring in
transferring out and he was like why Why
can't they just leave these people
alone? Why does it keep changing? And
then he got the insight that maybe a lot
of fraud happens because the person
working in the branch colludes with the
customer, but by changing the staff
constantly, you break the the risk of
fraud happening. And that apparently was
the genesis for this idea. True,
apocryphal? I have no idea. But it's
sort of a fun story. Uh yes,
>> instead of random, if we go to the way
historical models are built, concepts of
multiple and all of that, would that
make it sharper as compared to this?
>> The problem is that um these networks
are massive, right? And for you to take
each layer and look at it correlation
with some other layer and so on and so
forth. First of all, investigating
multi-linearity is pro is a problem. The
second thing is okay, what do you do
then? Next uh in linear regression you
can do things like principal components
analysis to get around it. Here
everything is nonlinear. There is no
easy way to solve the problem. So we are
like we'll just solve the problem in one
shot using dropout. That's all right. Um
so I had uh some material on
something called bite pair encoding
which I will um which I will do when we
get to LLMs and I stuck it in the end
because I knew that we probably won't
have enough time to cover this anyway.
And that is a very clever tokenization
scheme used by for example the GPT
family and that allows them to do
beautiful punctuation, keep the case
intact and then use words that you just
made up and things like that. Okay. So
we have two one more minute. I'm happy
to answer any questions you might have.
>> And so initially when we are picking
like the hidden layer the number of
neurons and weed. So so far in all the
materials this is has been given to us
but initially how do you pick it? Is it
more of a trial and error type of thing
or
>> it tends to be trial and error. Um so
that's in fact what I did when I created
the collabs. So um and and you can
actually make it a bit more systematic
by trying lots of different values and
there is a particular package uh Python
package called Keras tuner. So just
Google Keras tuner and it comes with
very nice collabs and if I have a chance
maybe I'll just record a screen
walkthrough of doing that. But that's
that's a very efficient way to do these
things. And it comes under the broad
category of something called
hyperparameter optimization where the
number of neurons, the activation you
use, the learning rate, all those things
can all be tried. You can try lots of
variations and kas is a great way to do
it in the context of kas.
Other questions?
>> All right, I give you 30 seconds back.
Thank you. See you tomorrow.