Okay. So, let's get going. Today we're
going to talk about how do you actually
train a neural network, right? Because
that is sort of the heart of the game
here. Um so, just to recap, we looked
last class
at what it takes to design a neural
network, and we made this very important
distinction between the things that you
are handed by your problem and the
things that you have agency over, that
you have control over. And we noticed
that, you know, the input layer for your
problem, the input is the input. Uh the
output is the output. You got to do
something with the output, something
that's expected. But everything that
happens in the middle is actually in
your hands. And in particular, we
noticed that we have to decide how many
hidden layers we want. We have to decide
in each layer how many neurons to have.
And then we had to decide what uh
activation to use. Even though I'm kind
of cheating when I say that because I
told you very clearly on Monday that for
the hidden layer activation, just go
with the ReLU activation function. You
don't have to think deep thoughts about
this, okay?
But the other things are all choices you
have to make, and we will talk a bit
later about how do you actually make
those choices.
Okay. Now, the rule of thumb,
right? The rule of thumb always is to
start with the simplest network you can
think of.
And if it's if it gets the job done,
stop working on it.
If it's not good enough, make it
slightly more complicated. Okay? So,
that's sort of the, you know, like the
meta thing you have to remember always
when you're designing these things.
Okay. So, that's sort of, you know, what
it takes to design a deep neural
network. So, what we will do in this
class is we'll actually take a real
example with real data, and then we
we'll think through how we would design
a network to solve this problem.
And while doing so, we will cover a
whole bunch of conceptual foundations
such as optimization, loss functions,
gradient descent, and all that good
stuff.
Okay?
All right. So, the the case study or the
scenario here is we have a data set of
patients uh made available by the
Cleveland Clinic. And essentially, we
have a bunch of patients, and for all
these patients, the setting is that they
have come into the Cleveland Clinic, and
they have not come in with a heart
problem. They have come in for something
else. Maybe they just came in for a
physical. And we measured a whole bunch
of things about them, okay? And the
kinds of things we measured are, you
know, demographic information, like
what's their age, uh gender, whether
they have any chest pain at all when
they came in, blood pressure,
cholesterol, sugar, so on and so forth.
Right? You get the idea? Demographic
information and a bunch of biomarker
information. And then,
what the Cleveland Clinic uh did was
they actually tracked these people
and figured out in the next year,
did they get diagnosed with heart
disease or not?
Okay, in the next year.
Which means that maybe you can build a
model when someone comes in, even though
they didn't come in for a chest problem,
maybe you can predict that something's
going to happen to them in the next
year, right? It's a nice sort of classic
machine learning setup.
All right. So, this is the thing. So,
what we want to do is we can totally
solve this problem using decision trees,
neural network I mean, sorry, random
forests and gradient boosting and all
that good stuff you folks have already
learned from machine learning.
But we will try to solve it using neural
networks, okay? Um this is an example,
of course, of what's called structured
data because this is all data sitting in
the columns of a spreadsheet, right? Uh
so, working with structured data is the
way we warm up our knowledge of neural
networks. And then we will do things
like working with unstructured data
starting next week with images and then
later on with text and so on and so
forth. Okay, any questions on this?
Okay. Uh yes. Uh just connected even to
last time's class where we took uh the
same example and first it was a logistic
and then we did a neural network. So,
the probability in case of one was 0.85,
then was 0.22, and here as well, how do
you know when to uh
use what? Usually in textbooks, you know
when to use logistic or when to use uh
something else, but in this case,
uh
when do I complicate it to neural
networks visa-vis in this case maybe
just doing a random It's a great
question. Uh when do you use what? So, I
think there are two broad dimensions
that you have to think about. One broad
dimension is
uh how important is it that you need to
explain or interpret what's going on
inside the model to perhaps a
non-technical consumer.
The other dimension is how important is
sheer predictive accuracy.
In some situations, predictive accuracy
trumps everything else. In which case,
just go with it. In other cases,
explainability becomes a big deal
because if they can't understand, they
won't use it.
And those cases, it's probably better to
go with simpler models such as decision
trees and neural I mean, not neural
network decision trees, maybe even
random forests, certainly logistic
regression. Those are all a little more
amenable.
But that said, uh even complex black box
methods like neural networks, there is a
whole field called mechanistic
interpretability,
which seeks to try to get insight into
what's going on inside these big black
boxes. So, the story isn't over, right?
But that's just the first cut you sort
of analyze the problem.
Okay. So,
um let's get going. So, if you want to
design a network,
All right. So, we design the network. Uh
so, we have to choose the number of
hidden layers and the number of neurons
in each layer. Then we have to pick the
right output layer. So, here,
what I did is the simplest thing you can
do is, of course, is to have no hidden
layer.
So, if you have no hidden layers, what
is that model called?
Yes, logistic regression.
Okay? So, of course, we want to do a
neural network, so I'm going to have one
hidden layer because that's the simplest
thing I can do. And then, I'll confess,
I tried a few different numbers of
neurons in this thing, and when I had 16
neurons, it actually did pretty well.
Okay? So, there was some trial and error
that went on before I landed on the
number 16. Right? And for some reason,
people always use powers of two, so may
as well do that.
So, I tried like 4, 8, 16, and 16 was
really good.
And as it turns out, when I went above
16, uh it sort of started to do badly.
And it started to do badly because
something called overfitting,
which we're going to talk about later,
okay? So, yeah, 16.
Um and then by default, I use ReLUs,
okay? So, 16 ReLU neurons. And then
here, the output is a categorical
output, right? Heart disease, yes or no,
one or zero, classification problem,
which means that we want to emit a
probability at the very end. Therefore,
we'll use a sigmoid.
Okay? So, so far, so good, right? Any
questions?
All right.
So, we're going to lay out this network
visually.
Okay? So, we have an input, and so I
just have have an input. And as you will
see here,
X1 through X29, that's our input layer.
And you may be wondering, 29, where did
he get that from?
Because there doesn't seem to be like 29
rows here of independent variables. So,
it turns out there are only 13 input
variables here,
but some of them are categorical.
So, what I ended up doing is to take
each categorical variable and one-hot
encode it.
Okay?
And when you do that, you get to 39.
Sorry, 29.
All right? And when we actually do the
Colab later on, I'll show you exactly
how I one-hot encode encoded it, but
that's what I'm doing here.
That's why you have 29, not 13.
Okay? Now, obviously, we have decided on
these hidden units, 16 units,
with nice ReLUs here.
Okay? And then we have an output layer
with a little sigmoid.
And I got bored of trying to draw all
these arrows, so I just gave up and
said, "Assume there are arrows."
Okay, between all these things.
Good?
Yeah.
Yeah, I'm sorry. I think you already
mentioned this, but why 16 units? Why
16? Uh
I tried a bunch of different numbers of
units. Uh and at 16, the resulting model
did well, so I just went with that. And
the logic of why is a ReLU?
Oh, why a ReLU? Yeah, so there's a
there's just a mountain of empirical
evidence that suggests that uh ReLU is a
really good default option for using as
activations in hidden layers. There is
also a really great set of theoretical
results, and I'll allude to some of them
when we actually talk about gradient
descent.
Yeah.
Sorry, quick question. You mentioned um
in the input layer, how how did you get
to 29 again when you had like 13
variables? So, some of those 13
variables are categorical variables like
uh cholesterol low, medium, high. Right?
And so, I took them and one-hot encoded
them. So, if it had like five levels, I
would get five columns now.
Uh yeah.
And by the way, folks, um just like uh
is it can Yeah, just like did, please
use a microphone so that people on the
live stream can hear your question.
Yeah, go ahead. Uh sorry, just one
question. So, the vectors, since you
didn't represent them, are we assuming
like every X is connected to all the
units?
>> Correct. And this is also a parameter
that we have to decide or That ends up
being the default.
And we will see
deviations from that assumption when we
go to image processing and language
processing and so on. But when you're
working with structured data like we're
doing now, that's the default.
Okay. So, let's keep going.
So, this is what we have.
So, what Remember what I told you in the
last class? Whenever you're working with
these networks, right? Get into the
habit of very quickly calculating the
number of parameters.
Right? Just do it a few times, the first
few times, so that you really know cold
exactly what's going on. Okay? So, yeah,
how many parameters do we have here?
How many weights and biases? You can
work through it, okay? You can You don't
have to tell me the final number. You
can say x * y + z, stuff like that.
Yeah.
65. You have 48 weights and 17 biases.
Okay, and how did he come up with that?
So, for the weights, you have like for
the first layer it's 2 * 16 and for the
the second connection it's 1 * 16 and
then the biases are the 16 hidden plus
the outputs.
Okay.
Um any other views on this?
I think it's 29 into 16. 29, okay, 29
into 16. And then 16 into
uh plus I mean 16 there. Yeah. And then
biases 16 biases and one bias. Right.
So, the way it's going to work is we
have 29 things here, 16 in the middle,
so 29 into 16 arrows.
And then for each of these fellows,
there's a bias coming in.
So, that's another 16.
Plus, you have 16 * 1.
Which is here, plus there is one bias
for this one.
So, the total is 497.
So, you can see here there's something
very interesting going on, which is that
when you go from one layer to another
layer,
the number of weights is roughly on the
order of a * b.
The number of units and so that's a
dramatic explosion in the number of
parameters.
Right? And that's something we have to
watch for later on to prevent
overfitting.
Okay, that's where the explosion of
parameters comes from the fact that each
layer is fully connected to the next
layer.
Okay? But we'll revisit this later on.
Okay.
So,
what I'm going to do now is I'm going to
actually translate this network, right?
The one that we have laid out
graphically, into Keras code
to demonstrate how easy it is.
Okay? So, I will give a fuller intro to
Keras in TensorFlow later on, but for
now, just suspend your disbelief.
We'll just try to do it in Keras as if
we know Keras. Okay? So, let's try that.
Later on we'll get into all the gory
details and train it in Colab and so on
and so forth. Okay. All right. So,
So, the So, the way we typically do it
is that once we have a network like
this, we typically start from the left
and start defining each layer in Keras
one after the other. So, we flow left to
right. Okay? So, let's take the input
layer. The way you define an input layer
in Keras is really easy.
You literally say Keras.input.
Okay? And then you tell Keras how many
nodes you have in the input coming in.
In this case it happens to be 29, so you
tell it the shape. Shape equals 29. And
the reason why we say shape as opposed
to length is because, as you will see
later on, we don't have to just send
vectors in, we can send complicated
things in to Keras.
And those complicated objects could be
matrices, it could be 3D cubes, it could
be 4D tensors and so on and so forth.
So, it's expecting a shape.
Right? What is the shape shape of this
thing you're going to send me? In this
particular case it happens to be a nice
list or a vector, so it's 29. Okay,
that's it. So, we we write this down.
This creates the input layer.
Right? And we give it a name. Right? And
the name here means
this layer, whatever comes out of this
layer has a name input.
Okay?
Good. Next.
Let's make sure the shape of the input
as I mentioned.
Right there.
Then we go to the next one. And here and
we will unpack this. The way you define
a layer is typically a hidden layer
Keras.layers.dense
and all this stuff. Okay? So, what this
is is it first of all it says
I want a dense layer. By dense layer I
mean a layer that's going to fully
connect to the prior and the later
layers.
Fully connect, that's what the word
dense means. Okay?
Number two,
I want 16 nodes here in this layer.
Okay? Finally, I want to use a ReLU.
See how compact and parsimonious it is?
Right? And that is the appeal of Keras.
It's very easy to get going.
So, the moment you do that, you've
actually defined this layer.
But what you have not done
is you have not told this layer what
input is going to get.
Because as far as this layer is
concerned, it doesn't know that this
other layer exists.
So, you need to connect them. Yes.
Um do we need to define for the ReLU
where the the bends are? Like where you
take the max?
>> No, the ReLU the bend is always at zero.
Okay. Thank you.
Okay?
All right.
So, that's what we have here.
And then, what we do is we have to tell
it I you want to feed this layer the
output of the previous layer, so you
feed it by taking whatever is coming out
of this thing, which is called input,
and you basically
stick it in here.
So, the moment you do that, boom, it's
going to receive the input from the
previous layer.
And because this one's output needs to
go to the final layer, you need to give
a name to that output.
So, you give it a name. I'm just calling
it h for because it's coming out of the
hidden layer.
It's just a variable. You can call it
anything you want.
Now, what we do, we go to the final
output layer.
And this is what we use. The output
layer is just another dense layer.
That's why I use the word dense. But we
say, "Hey, give me just one thing
because I just literally just need one
unit here because I need to emit just
one probability.
And the activation I want to use is a
sigmoid."
Done.
Okay?
And once you do that, you
have to feed it the input from the
second layer. So, you stick an h here.
Now you have connected the third and the
second layers.
And after you do that, you give a name
to the output coming out of that. We'll
just call it output. You can call it y,
you can call it output, you can call it
whatever you want.
Okay? So, at this point, what we have
done
is we have mapped that picture into
those three lines.
That's it.
Okay?
But we aren't quite done yet. There's
one little thing we have to do.
So, what we have to do is we have to
formally define a model so that Keras
can just work with this model object. It
can train it, it can evaluate it, it can
use it for prediction and so on and so
forth. So, we tell Keras, "Hey, uh
create a model for me, Keras.model,
and basically where the input is this
thing here and the output is that thing
there.
And then the whole thing we'll just call
it model."
Okay? So, that's it.
We are done. That is the whole model.
That is It sounds really fancy, right? A
neural model for heart disease
prediction. That's pretty cool.
Four lines.
And we will show how to train this model
with real data and so on and so forth
and use it for prediction after we
switch gears and really get into some
conceptual building blocks.
Had a question.
Can you define a custom activation
function that is not in the list of
Keras library? Yes.
Yeah, you can define The question was,
can you define a custom activation
function? You totally can.
Uh in fact, I mean, the the kind of
flexibility you have here is incredible.
And this these innocent four lines
unfortunately sort of hide the the
potential that's possible here, but I
guarantee you in two to three weeks you
folks will be thinking in building
blocks like Legos.
So, you'll be, you know, I I I I'm so
happy when it happens. Students will
come to my office hours and say, "You
know, I want to create a network where I
have a little network going up on top,
one going in the bottom, then they meet
in the middle, then they fork again,
they split." I'm like, "Unbelievable."
It's fantastic. And you're going to be
doing this in two weeks, I guarantee
you.
Yeah, in the case of a multi-class
classification problem, are the output
nodes equal to the number of classes?
Correct.
So, we will come to So, this is binary
classification. And the question is for
multi-class classification, let's say
you're trying to classify some input
into one of 10 possibilities, we will
have 10 outputs.
But the way we define it is going to be
using something called a softmax
function, which we're going to cover on
Monday.
So, for now, we just live with binary
classification.
Uh
Is there a default activation method in
Keras or you have to put something? Ah,
that's a good question. I believe the
default might be ReLUs for hidden
layers, but I'm not 100% sure. Let's
double-check that.
Uh
Uh just to get a clearer understanding,
when you said that beyond 16 when you
tried working on those neurons, the
performance uh worsened.
So, that is where you were playing
around with initially two and then maybe
four and six and eight. Exactly. Right.
Could you use the mic?
Do we need to define each of the hidden
layer when the model gets more complex
when we have more than one layer? Oh,
like if you have like 25 layers?
>> consolidate, yeah. Yeah, yeah, yeah. So,
what we typically Good question. If you
have let's say 100 layers, right? Uh do
you actually write I have to type in
each by hand and cut and paste? No. You
can actually write a little loop which
will just automatically create them for
you.
And so, basically what's going on is
that this little output thing you see
here, this variable,
this output could be the result of a
thousand layer network with all sorts of
complicated transformations going on and
then finally it pops up as a little
thing called the output. And what Keras
will do is it'll be like, "Okay, this
model has this input and has this
output, but boy, this output came from
incredible transformations applied to
the input." And Keras will process all
that very easily for you. You don't have
to worry about it.
Right? It's really a beautiful example
of the power of abstraction.
And you will you will see that as we go
along.
Okay. So,
now let's switch gears and say once
you've written a model like that in
Keras, how do you actually train it?
Okay? Now, training is something you've
been doing a lot, right? So, for
example, when you have something like
linear regression, right? Where you have
all these coefficients you need to
estimate, you have this model, then you
have a bunch of data, then you run it
through something like LM if you use R,
and what it gives you is actual values
for these coefficients, right? 2.8, 0.9,
and so on and so forth. So, the the role
of the data is to give you the
coefficients.
Right? Or you can think of the
coefficients as really a compressed
version of the data.
Okay? Similarly, if you do logistic
regression, you have a model like that,
you add some data, you run it through
some estimation routine like GLM or
scikit-learn or statsmodels, pick your
favorite tool, then you'll come up with
something like that. So, basically
what's going on here is training simply
means find the values of the
coefficients that so that the model's
predictions are as close to the actual
values as possible. That's it. Okay? And
so and to find the one that is as close
to the actual value as possible, a whole
bunch of optimization is involved. You
didn't have to worry about the
optimization when you did the
regression, linear or logistic, because
it's all done under the hood for you,
but for neural networks, we actually get
to know how it's done.
Okay, because it's important.
Okay. So, training a neural network, a
deep neural network, even GPT-4, it's
basically the same process as what you
do for regression.
Right? You basically you're just a very
complicated function with lots of
parameters, but ultimately you have a
network with all these question marks,
you add some data, you do some training,
and boom, you get some numbers.
You may get into this, but are we
determining the architecture of the
network before we train it?
Okay. Yes, because if you don't define
the architecture,
um Keras doesn't know how to actually
calculate the output.
Given an input. And unless it knows
input-output pairs, it can't do anything
more with it.
Okay. So, um
so the essence of training is to find
the best values for the weights and
biases.
And the way we think of the best values
is that we basically set up a little
function, and this function measures the
discrepancy between the actual and the
predicted values. Okay? And I use the
word discrepancy because the way you
define discrepancy, there's an
incredible amounts of creativity in the
field.
In fact, a lot of breakthroughs in deep
learning come because people define a
very clever measure of discrepancy, and
then turns out it actually gives you all
sorts of interesting behavior. Okay?
That's why I use the word discrepancy as
opposed to the word error, because when
I say error, you might be just thinking
something like predicted minus actual.
That's too limiting.
Prediction minus actual is too limiting,
that's why I use the word discrepancy.
So, so we we basically define a function
that captures the discrepancy between
these the actual and the predicted
values, and these functions are called
loss functions in the deep learning
world.
And every paper that you read, you will
find interesting loss functions. There
are hundreds of loss functions, enormous
research creativity goes into defining
these loss functions. Okay?
All right. So, these are loss functions.
And so a loss function is a function
that quantifies a discrepancy. So, let's
say the predictions are really close to
the actual values, the loss would be
what?
It's close to zero. It's close to zero.
Close to zero. Right? Very small.
And if if you have a perfect model,
perfect crystal ball, what would the
loss be?
Exactly zero.
Right? Exactly zero. So, in linear
regression, we the loss function we use
is called sum of squared errors.
We didn't call it loss function because
we were not doing deep learning, just
linear regression, but that's basically
the loss function. Right? So,
the loss function we use must be very
matched very properly with the kind of
output we have.
Right? So, if your output is a number
like 23, right? You're trying to predict
demand like a product demand for next
week for a particular product, and uh
predicted value is 23, the actual value
is 21,
it's okay to do 23 minus 21, two as a
discrepancy, right? The error. Okay? But
for other kinds of outputs, it's not so
obvious what the correct loss function
is, what the correct measure of
discrepancy is. And so here,
for the simple case of regression,
right? Um
the YI, the I here, by the way, is a
superscript which stands for the ith
data point, the ith data point. So, what
I'm saying is that okay, for the ith
data point, this is the actual value, Y,
and this is what the model predicted.
Okay? I take the difference, square it,
and once I square it for each point, I
just average all these numbers to get an
average squared error, i.e. mean squared
error, MSE. So, this is sort of like the
easiest loss function.
Okay?
Now, let's crank it up a notch.
In the heart disease example, the heart
disease the neural prediction model,
the prediction is a number between zero
and one, right? It's because it's coming
out of the sigmoid.
It's a fraction. The actual output is a
zero or one, one of the two, right? It's
binary.
So, how would we compare the
discrepancy? How would we measure the
discrepancy between a fraction and the
numbers zero and one? Right? What is the
good loss function in this situation?
Right? Is the key question. So, let's
build some intuition around this.
And let's see if my little daisy chain
iPad thing works.
I'm doing it on the iPad so that people
on the live stream can see it, otherwise
the blackboard is a little tough for
them.
Okay. So, let's have a situation here.
Okay? So, let's say let's say that you
have a patient who comes in, and let's
say they have heart disease. Okay? So,
for that patient, Y equals one.
Right? The true value is one for that
patient. And now you have this model.
Okay? And this is the predicted
probability from this model.
Can people see my
handwriting okay?
Good.
I could never be a doctor, right? So.
So, zero, okay? One, it's going to be
between zero and one because it's
probability.
And then this is the loss we want to
sort of have, right? This is the loss.
So, for this this patient actually had
heart disease, Y equals one. So, let's
say that the predicted probability is
pretty close to one.
Okay? What do you think the loss should
be?
Small.
Close to zero.
Sorry?
Close to zero, exactly. So, here, if the
prediction comes here, you want the loss
to be you want the loss to be somewhere
here.
But if the predicted probability is
pretty close to zero, even though the
patient actually has heart disease, what
do you want the loss to be?
Really high.
Because it's screwing up badly, right?
So, you want the loss to be somewhere
here.
So, basically you want a function that's
kind of like that.
Right? You want the loss function shape
to be like that.
High values of probability should have
low losses, low values of probability
should have high losses. Yeah.
I understand like why it has to be
increasing or decreasing, but can you
explain why it has to be Yeah, yeah. So,
it can be linear, it can certainly be
linear, but basically what you want to
do is the more it makes a mistake, the
more harshly you want to penalize it.
Right? So, basically what you're what
what you really want is something where
if it basically says this person's
probability is say uh the probability
the predicted probability is say one
over a million,
basically close to zero, you want the
loss to be like super high.
So that the model is like it's like a
huge rap on the knuckles for the model.
Don't do that.
That's basically what we're doing, and
I'm sort of demonstrating that dynamic
by using a very curved and steep loss
function.
But you can absolutely use a linear
function, it's totally fine. It won't be
as effective for gradient descent later
on with a bunch of bunch of technical
details.
Are we good with this?
All right. So, now let's look at the
case where a patient does not have heart
disease.
Y equals zero.
Same setup, okay?
Predicted probability,
zero, one, loss.
So, for this patient, they don't have um
whatever uh they're not
uh they don't have heart disease. If the
probability is close to zero, what
should the loss be?
Close to zero. It should be somewhere
here, right?
And the more and more the probability
gets closer and closer to one, you want
to penalize it very heavily, which means
you want the loss to be somewhere here.
So, you basically want a loss ideally
that's kind of going up like that and
climbing higher and higher.
Are we good?
Okay, perfect.
Because we have a perfect loss function
for that.
So, just a recap.
Right? This is what we want.
People with for points with Y equals
one, lower prediction predictions should
have higher loss. You want something
like that. And then turns out
there's a very little simple loss
function
which just literally just uses the
logarithm, which will get the job done.
So, what you do is you literally do
minus log of the predicted probability.
That's it. And that thing it has exactly
that shape.
Okay? And in fact, you can see it
numerically. So, if the loss is one,
it's zero. If it's half, it's 1.0. And
if it's like one over 1,000, it's almost
10. If it's one over 10,000, it's going
to be like
much higher, right? Very high losses.
Okay? So, minus log probability, boom,
done.
Similarly, this is what we want for
patients for whom Y equals zero.
And turns out if you do minus log one
minus predicted probability, it does the
same thing.
Okay?
Mathematicians once again saved with a
logarithm.
So, see in summary
this is what we have.
Right? For data points where y equals 1,
we have this. Data points where y equals
0, we have this. But, it feels a little
inelegant
to say, "Well, if it's y equals 1, I
want to use this. If y equals 0, I want
to use that."
Right? There's There's like an if-then
thing going on here. And I don't know
about you folks, but if-then really irks
me
mathematically because you can't do
derivatives and so on very easily.
Okay?
But, no worries. This is MIT. We know we
have our bag of math tricks.
So, what we do is
we can actually combine them both into a
single expression.
Okay? Like this.
Okay? And here the yi again is the ith
data point. Remember, yi is either 1 or
0 always.
And this model of xi is the predicted
probability. Okay? So,
and I've just taken the minus log the
minus and I've just moved it here.
Okay? And I've taken the the minus that
was here and just moved it here. Okay?
That's why you see it like this.
So, this one is basically
you can convince yourself what's
happens. This single expression will get
the job done. So, let's say there is a
patient for whom y equals 1.
What's going to happen is that when you
plug in y equals 1, this becomes 0. The
whole thing will collapse to 0.
While here, y equals 1 just means it
becomes minus log probability, which is
what we want.
Conversely, if y equals 0, this whole
thing is going to disappear.
And this thing becomes 1 minus 0, which
is just 1. And so, it becomes minus log
1 minus probability, which is again what
we want.
Simple and neat, right?
So, in one expression, we have defined
the perfect loss. No if-thens, none of
that crap.
Good. So, now what we do is that was
true for every data point.
But, we obviously have lots of data
points. So, we just add them all up and
take the average.
That's it. We average across all the
data points we have. So, that we get an
average loss.
Okay?
We call this is the binary cross entropy
loss function.
Is there a way you can um edit the loss
function so that you penalize like false
negatives more strongly than false
>> you can do all of them. Great question.
Uh I'm just looking at the basic case
where we it's symmetric
loss. Um you can actually penalize
overestimates much more than
underestimates and things like that.
Um and if you're curious, you can just
Google something called the pinball
loss.
Okay?
Any other questions on this?
So, when you see this massive deep
neural network built by Google for doing
something or the other, if it's a binary
classification problem, chances are
they're using this thing.
Okay?
All right.
So, now let's figure out how to minimize
these loss functions because the name of
the game
is to find a way to minimize these loss
functions. So, now loss functions are
just a particular kind of function. So,
we'll first consider the general problem
of minimizing some arbitrary function.
Okay?
And once we develop a little bit of
intuition about that, we'll return to
the specific task of minimizing loss
functions.
How's everyone doing?
Yes, no, good, bad?
You have a bit of a
like a tough-to-interpret head shake.
It's more like um I kind of lost you
where you said that the loss function
and the predicted probability
uh how were they inversely because my
understanding was that the loss function
is supposed to be the sum of errors.
We're averaging the errors. And when you
said the heart patient
>> Sorry, sorry. Let me Let me just stop
there for a second.
For each point, you define the loss.
That's the whole point of the game. And
once you define it, you calculate for
every point and average it, right? So,
just focus on a single data point.
And so, now continue.
So, now when the heart patient has There
is more probability that they No. So,
when there is a person who has the heart
uh disease, you said that you want the
loss function to be high.
I think I'm going back to the graph.
>> You want the loss function to be high if
I'm predicting that they basically don't
have heart disease.
If the prediction is close to 0,
the predicted probability is close to 0,
then I'm badly wrong.
Because in reality, they do have heart
disease.
And that's why I want the loss to be
really high. Okay, so effectively, loss
is my way of finding out how good my
model is instead of saying, "Okay." Or
rather, how bad your model is. Yeah.
Right? How bad is it? That's really what
the loss function is. Got it.
>> And you want to minimize badness.
That's the whole point of optimization.
Okay.
Um I guess I don't have a fully like
similar to the point where I said but I
don't have a fully clear intuition of
why exactly a log function rather than
something that say
flatter for small and then really steep
later. Those are all fantastic things.
You can totally do it. Uh the reason we
picked the loss this function because A,
it's easy to work with. It has good
gradients. It's well-behaved
mathematically. But, there are many
alternatives to it. I don't want you to
think that this is like the only game in
town or it's the only choice for us. We
have many choices. This is really This
happens to be a very easy choice, which
also happens to be empirically very
effective.
And I'm happy to give you pointers to
other crazy loss functions, right? Which
can actually do all these things, too.
Okay?
All right. So, uh minimizing a single
variable function, we will warm up by
looking at this little function here.
Okay? Which is a
What do you call a fourth power?
What? Quartic, right? Yeah, thank you.
Quartic. So, yeah, it's a quartic
function. Um
right? And this is how it looks like.
But, you can see there is like a minimum
somewhere here, right? Between like one
minus one and minus two. Like maybe
minus 1.5. Okay?
So, we want to minimize this function.
It's obviously a toy function, little
function with one variable.
But, the intuition we use here is going
to be exactly what we use for GPT-4.
So, pay attention.
So, how can we go about minimizing this
function?
What will we do?
Yeah.
Take the derivative and set it equal to
zero. You take the derivative. Exactly.
So, you take the derivative, right?
Um so, when you So, let's look at what
the derivative does for us.
But, then
the second part of what said
Yeah. Second part of what said was set
it to zero. Setting it to zero becomes
problematic
when you have very complicated
functions. It's not clear at all what's
going to make them zero, right?
Unfortunately. But, the idea of taking
the derivative is in fact the right
idea.
So, we can go about this. We can
calculate the derivative. And that
actually happens with the derivative.
You can convince yourself.
And if you plot the derivative, it looks
like that.
And as you would hope, wherever the
minimum is, in fact, the derivative is
crossing
right? The derivative is zero here. It's
crossing the x-axis.
Right? In this case, you can actually do
that.
So, let's say you have the derivative.
How can you use it?
Like, what is the value of a derivative?
What does it tell you?
Yeah.
You use a gradient descent algorithm.
You are 10 steps ahead of me, my friend.
I just want the basic answer.
Like, what what what what good is a
derivative? What Like, what does it tell
you? When you calculate the derivative
of something at a particular point
>> you the rate of change of the function
at the place you are. Correct. Exactly
right. So, here, what the derivative
would tells us is that the slope tells
us the change in the function for a very
small increase in w, right?
And this is high school calculus. I'm
just doing a quick refresher.
So, what that means is that
if the derivative is positive,
what that means is that increasing w
slightly will increase the function.
So, if if you're here,
you calculate the derivative, the slope
is positive. It means that if you go
slightly in this direction, the function
is going to get higher.
Right?
Similarly, if it's negative,
let's say here, you calculate the
derivative, it's the the slope is like
this. It's negative, which means that if
you increase w, if you go in this
direction, it's going to decrease the
function.
Okay?
All right.
And if it's kind of close to zero,
it means that changing w slightly won't
change anything.
So, if you're here, changing it slightly
won't change anything.
All right?
That's it.
So,
So, what we do is this immediately
suggests an algorithm for minimizing gw,
which is let's start with some random
point w.
And then,
let's calculate the derivative at that
point.
And once we do that,
there are three possibilities.
It could be positive, negative, or kind
of close to zero.
And if it's positive, we know that
increasing w will increase the function.
But, we want to decrease the function.
We want to minimize it.
Which means that we should not be
increasing w. We should be doing what
here?
Decrease.
Yes. And similarly, if it's negative,
what should we do here? Increase.
Exactly. So, in the first case, you
reduce w slightly. In the second case,
you increase w slightly. And if the
thing is close to zero, you just stop
because there's nothing else you can do.
Okay?
This is the basic intuition behind how
GPT-4 was built.
Which is kind of shocking if you think
about it.
Right? Which means that all the the
heavy-duty optimization stuff that
people have figured out over the decades
is kind of not used.
Right? This algorithm is what's being
used with some, you know, flavors on top
of it.
So, yeah. So, back to this
uh and you you do that and then if
you've sort of run out of time or
compute
or right, if you run out of time and so
on, just stop.
Otherwise, just go back to step one and
try again. Of course, if it's close to
zero, you got to stop anyway.
Yeah.
Is there the um concern of a potentially
local minimum there? It's coming.
Okay? So, that's the function. It's
going to give find It's going to find
you some point where the derivative is
kind of close to zero. Okay?
So,
this is called gradient descent. Right?
This is gradient descent, this little
algorithm.
And this this
this very power pointy MBA table can be
collapsed into this little expression.
Basically says,
calculate the derivative,
multiplied by a small number which we'll
get to in a second,
and then change the old W to the new W
is the old W minus a little number times
gradient.
So, this little one-line formula is
basically gradient descent.
Okay?
And what you should do, just to build
your intuition, is to make sure that
these three possibilities here map
nicely to this. Like this thing will
actually capture these three
possibilities.
This is when gradient descent was
invented.
It has some historical fun, right?
The 19th century?
19th century. Yeah, okay. Good. Very
good. Excellent guess.
1847.
It was uh invented uh in 1847 by Cauchy,
the great mathematician. And in fact, if
you're curious, you can check out the
paper.
I have I gave you I give you the paper
here for handy reference.
So, 1847.
So, GPT-4 is built using an algorithm
invented in 1847.
Which I find like astonishing, frankly.
That this little thing is so capable.
Okay.
So, that's gradient descent. And this
little number alpha
is called the learning rate. And it's
our way of sort of essentially
quantifying the idea of let's not
increase or decrease W massively, let's
do it slightly.
Because the gradient is only valid for
small movements around your point. If
you take a big step, all bets are off.
So, this alpha tells you how how small a
step should you take.
Okay?
And in typically, it's set to very small
values like, you know, 0.1, 0.001, and
so on and so forth. And in fact, if you
read any deep learning academic papers
where they have trained like a big model
to do something,
right? More lot of researchers will very
quickly go to the appendix where they
have described exactly what learning
rates were used.
Because sort of the learning rate is
like part of the IP for how it's built.
A lot of trial and error that goes into
these learning rates.
Okay. So, that is gradient descent.
Um so, if we apply this algorithm to GW,
our original function,
right? We just keep on doing this thing
a few times.
Right? What you will find is that if
let's say we start with two point the
the
the point we randomly pick is a 2.5, we
set the alpha to one, we run this
algorithm, it starts here, then it goes
there, it goes there, bup bup bup bup
bup, and then finally ends up here.
In like four or five iterations, it
finds some minimum.
This is obviously a very simple,
well-behaved, nice little function, so
you can easily optimize it.
Okay? If you want, you can just go to
this thing. There's a nice animation of
this thing as well.
Okay. So, now
All right. Before we actually go to the
multi-variable function, I want to go to
the question that you posed about local
minima.
Um actually, you know what? I think I
may have some slides on it. So, sorry.
I'll come back to this.
So, let's actually see what you know,
what we looked at a toy example where
there was only one variable. What if you
have
uh what if it was GPT-3? GPT-3 has 175
billion parameters.
175 billion and GPT-4, they haven't
published it, so we don't know. It's
supposed to be eight times as much.
Okay? So, I mean, the number of
parameters is massive. So, basically,
our loss function has
billions of variables, billions of Ws
that we need to optimize over, minimize
over. So, we need to use this notion of
a partial derivative. So, let's take
baby steps and say, okay, what if you
have a two-variable function, right?
Something like this, very simple. So,
what we can do is we can calculate the
partial derivative of G with respect to
each of these Ws.
And the partial derivative, just to
quickly refresh your memories,
is you take a function, you pretend that
everything other than W is a constant.
Then the function becomes a
a function of just one variable W, W1.
And then you just differentiate it like
you do everything else. And you you get
you get something, and that is
this thing here.
And then you do the same thing for W2,
you get this thing here, and then you
just stack them up in a nice list.
Okay?
This is the vector of partial
derivatives.
So, how should we interpret this? The
same way as before. Basically, for a
small change in W1, keeping W2 and
everything else fixed, how does the
function change if you change just W1
slightly? And similarly for W2 and all
the way to W175 billion.
Same thing. Okay?
So, um
now, when you have these functions with
many variables, many Ws,
uh since we have a gradient for each one
of those Ws, we stack them up into a
nice vector
of derivatives, and this vector is
called the gradient.
And it's denoted
using
this uh Anyone know what the symbol is
called?
nabla
Yeah?
Laplacian
Maybe. Maybe that's a synonym. But the
one I'm familiar with is nabla.
Delta is the one that's upside down
triangle, but I think the upside down
triangle is called nabla if I if I
recall. Am I right?
Thank you.
He's my go-to.
So, yeah. So, the gradient, um we just
call it the gradient, and it's written
as this.
All right. So, what we do is we simply
do gradient descent on every one of the
Ws
using its partial derivative.
Okay? So, in a in a gradient step, we
update W1 using this formula, W2 using
this formula.
Finished.
We've just generalized gradient descent
to an arbitrary number of variables.
So, and of course, as before, this can
be summarized compactly as this vector
formula.
Let me just do this.
So, what's going on here is that
I have
W1
old W1 minus alpha
times
the function G
of W1, then W2
W2 minus alpha
G by W2. And then all we're doing is
we're just stacking them up into a
vector
like that.
minus alpha, and this vector
like that.
So, this can be written as just this
vector W, the new vector
old vector minus alpha
and the gradient. Finished.
And you can see if it is, you know,
GPT-3,
this vector is going to be 175 billion
long.
Okay? But whether it's two or 175
billion, who cares? It's the same thing,
right?
Okay.
So, yeah. So, that's what we have here.
I'm really thrilled by the way this
whole iPad business is working out.
I was a little worried about it. Okay.
Um so, if you look at two dimensions,
this function, and if you actually look
at if you plot the function, this is W
the first W, the second W, and then you
actually This is actually the loss
function. That's the function GW. And
so, you're trying to find the minimum
here, and so this is how the gradient
descent will do do do do do. It will
progress if you're starting from this
point.
Or you can also sort of look at it from
up top down into the function, and
that's what this picture is, and it
shows gradient descent starting from
there and working its way down
um from here all the way to the center.
Okay. So,
All right. Local minima. So, now
gradient descent will just stop
near uh hopefully a minimum,
right? But the problem is it may not be
a global minimum. It may It may not even
be a minimum.
So, um
so, let's see what what I'm talking
about here.
Here are some possibilities.
So, let's take a simple function.
Okay? Let's take This is GW.
This is W. And turns out this function
is actually looks like this.
Okay?
So, you can see here
Well,
um this point
this point here
is a local minimum.
This is a local minimum.
It's a local minimum.
These are all
lots of local minima here.
Okay? And yeah, there's a lot of local
minima here, too.
So, these are all places in which the
derivative is going to be zero.
So, if you run gradient descent and it
stops because the gradient is reached
zero,
you could be in any of these places.
Right? So, there's no guarantee. So,
this in this picture happens to be
maybe the global minimum because it's
the lowest of the lot.
Right?
But, there's no guarantee you're
actually going to get there.
Okay, there's not even a guarantee
you're going to be in any of these
places because you could literally be in
this thing here
where it's sort of taking a break and
then continuing on down.
That, by the way, is called a you know,
a saddle point. I drew it badly, but
this sort of coming in sort of taking a
break and going down again is called a
saddle point. So, gradient descent can
stop at a saddle point. It can stop at
some minima. There's no guarantee it's
going to be global.
Okay?
But, it turns out it has not mattered.
So, it has not mattered. And there are a
whole bunch of reasons why it has not
mattered because when you have these
very complicated neural networks,
they're very complex functions. Even
finding a decent solution, right, to
these complicated networks is actually
really good for solving the problem.
You don't have to go to the best best
possible solution. And in fact, if you
go to the best possible solution, you
actually run the risk of overfitting.
So, that's one reason. The other
interesting reason and by the way, this
is a very hot area of research to figure
out exactly
So, it's sort of like this. Empirically,
what we have seen is that not worrying
about local minima, global minima, all
that stuff has not hurt us because these
is things are amazing.
GPT GPT-4, probably they just stopped
somewhere. They probably it wasn't even
a local minima. They're like, "All
right, we've It's been running for 6
days. We've spent 2 million dollars.
Let's stop."
Right? Because these are very expensive.
So, but that's still so magical.
You don't need to get anywhere close to
local minimum. But, there's another
interesting point which I've which which
I read about.
People basically hypothesize that
for you to be at a local minimum, just
think about what it means. It means that
you're standing at a particular point,
in every direction that you look,
things are just sloping upward.
Right?
Everything is sloping upward. Only if
everything is sloping upward all around
you, could you be at a local minimum
by definition. But, if you have a
billion dimensions,
what are the odds that you're going to
be standing at a point where every one
of those billion dimensions is going
upward?
The odds are really low.
Chances are some of them are going to go
going up, some of them are going down,
others are sort of coming down and going
another way. It's going to be crazy.
So, in some sense, the best you can hope
for in these very high-dimensional
situations is probably a saddle point.
And it turns out it's good enough.
So, for those reasons, we are content
with just running gradient descent with
some tweaks which I'll get to in a
second. Um and it just performs really
admirably.
Um how does alpha depend on like how
much compute you have? Like, would you
set the learning rate based on that or
not really?
>> No, the the learning rate is really
is a measure of It's sort of like this.
When you're at a point where you think
that the gradient is looking nice and
right, if you take a step in the
direction it's going to go down. And if
you further believe that it's going to
keep going down in the direction for a
while,
then you're very confident about taking
a big step.
But, if you're like, "I I don't know
because the maybe I take a little step,
maybe I have to go this way. I can't go
straight anymore." Then you don't want
to take a big step because then you have
to backtrack.
So, those kinds of considerations go
into the learning rate. Um and so,
that's sort of the rough answer to your
question. It's not so much determined by
compute and bandwidth and things like
that.
But, again, it's very it's a sort of a
complicated thing because sometimes with
a given amount of compute compute, if
you have a particular kind of data, you
can have very aggressive learning rates.
So, it tends to be a bit sort of, you
know, jumbled up complicated. So, but
that's sort of the the quick surface
level idea of what's going on.
Um okay.
9:31.
Anyway, folks, this lecture is like
probably one of the driest in the like
semester because of like I have to go
through all the concepts. Um once we
start doing collabs, you know, things
get a lot more lively.
Okay.
Um all right. So, now let's talk about
minimizing a loss function gradient
descent. So, here is our little binary
cross entropy loss function that we saw
from before. Right? This is what we want
to minimize. So, if you look at this
thing,
where are the variables we need to
change to minimize this function?
Folks, don't look at your phones.
Okay, with laptop and iPad use, don't
look at your phones.
Sorry, we've kind of abstracted um the
variables W, but just to bring it back,
those are actually the weights in the
neural networks, right? Yeah, the
weights and the biases. I'm just calling
them as weights. So, the output of these
uh minimization functions are going to
be the actual weights in your model,
right?
>> Exactly. Exactly right.
The whole name of the game is to find
the weights.
And so, for example, when you see in the
press that uh Meta has essentially um
made the weights of Llama 2 or something
available, that's basically what they've
done.
They basically published the weights.
Reason that's so valuable is
>> Microphone, please. Go.
Cuz if you have a billion parameters,
the compute time on that is horrendous
and expensive. That's why it does
weights are so valuable.
>> Correct. The weights are the crown jewel
because they are the result of a lot of
money and time and smartness being
spent.
There is a separate question of why are
they making it open source,
which
I'm happy to chat about offline.
All right, cool. So, what are the
variables we need to change change to
minimize? It's basically the parameters
and they're hiding inside the model
term.
Right? Because what is the model? The
model is some function like that, right?
If you look at the simple GPA and
experience thing we looked at in the on
Monday, we finally figured out that the
actual thing that comes out here is
going to be this complicated function of
all the X's and the W's and so on and so
forth, right? And that complicated thing
is showing up inside this thing.
So,
you know, and the W's here are the
variables we can we need to change to
minimize the loss function. And it It's
important for you to to note and
understand that the values of X and Y
and so on are just data.
You're not optimizing anything there.
You're just data.
What you're optimizing is the W's.
The weights.
Okay. So, so imagine replacing the model
here with the mathematical expression
above whenever this appears the loss
function. And once you do that, your
loss function is just a good old
function of the W's.
The fact that it's a loss function is
kind of irrelevant.
It's just a function.
And since it's just a good old function
of the W's, you can apply gradient
descent to it as we normally would.
It's no big deal.
Which brings us to something called
backpropagation.
Um
Um if you remember nothing else about
backpropagation, just remember this.
Never use the word backpropagation
again. Only use the word backprop.
You're
hip and cool to the deep learning
community.
Backprop.
Okay. All right. So, what is backprop?
Backprop is a very efficient way to
compute the gradient of the loss
function.
So, when you have this loss function,
and let's say you have a billion W's
and you have 10 million data points. So,
the little n we saw was 10 million.
That is a lot of computation.
And that is just for one step of
gradient descent.
Right? So, backprop is a way is a very
efficient and clever way to compute the
gradient of the loss function, which
takes advantage of the fact that what we
have here is not some arbitrary model.
It's a model that came from a particular
kind of neural network, which has layers
one after the other, and then there was
an output at the very end.
So, what backprop does is
it organizes the computation in the form
of something called a computational
graph, and the book has a good
discussion about it. And so, what we do
is we start at the very end.
We calculate the gradient of the loss
with respect to the output.
Then we move left. We calculate the
gradient of that output with respect to
the output of the just the prior hidden
layer.
Step to the left. Calculate the gradient
of the current thing with respect to the
previous layer. You get the idea, right?
It's iterative and it moves backwards,
and by doing so, you never repeat the
same computation twice wastefully.
That's the big advantage. You calculate
once and reuse it many many many many
times.
The second advantage is that if you
organize it this way, it just becomes a
sequence of matrix multiplications.
Okay.
And
because it's a sequence of matrix
multiplications and eliminates redundant
calculations, and best of all,
there are these things called GPUs,
graphics processing units, originally
invented to accelerate video game
rendering.
Uh and as it turns out, to accelerate
video game rendering, the core math
operation you do is basically a matrix
multiplication. Right? Some linear
algebra uh
sort of operations. And so, someone
really at some point had the bright idea
for deep learning, calculating gradients
and so on, we need to do matrix
multiplications, and here is some
specialized hardware that does really
that does a fast job of matrix
multiplications. Can't we Can we use
this for that?
And they did it. And all hell broke
loose.
That's literally what happened.
And that's why Nvidia is valued at what,
1.5 trillion or something.
So, yeah. So, they are really good. And
so, backprop
the way you do backprop plus using it on
GPUs leads to fast calculation of loss
function gradients.
If this thing were not true, this class
would not exist.
Because there won't be any deep learning
revolution.
This is a fundamental seminal reason.
All right. So, the book has a bunch of
detail
um
and I actually did like a I work I hand
worked out an example
of calculating a gradient like the
old-fashioned way and calculating it
using backprop.
So, take a look at it. I'll post it on
Canvas and you will understand exactly
where the savings come from, where the
efficiency gains come from. Okay?
Because of time, I'm not going to get
into it now.
All right. Any questions so far?
Yep.
Sorry, I followed up to and so, we've
done gradient descent, which is
different than calculation of the
gradient of the loss function. What What
is the purpose of the calculation of the
gradient of the loss function? You
calculate the gradient because the
fundamental operation of gradient
descent is to take your current value of
W
and modify it slightly and the
modification is old value minus learning
rate times gradient.
It'd be cool, right, if I say, "Go mo-
go back five slides to this thing." and
it just goes back. Product idea. Anyone
startups?
So.
So, this one.
So, this is the fundamental step of
gradient descent.
So, this is the current value of W.
You calculate the gradient at that
current value
multiplied by alpha do this thing and
you get the new value.
And you keep repeating.
Right, but GW
that's not that's not the loss function.
>> It is the loss function. That is the
loss function.
>> Yeah, right. Here, I'm just using G as
an arbitrary function
to just to demonstrate the point. But
when you're optimizing, when you're
training a neural network, what you're
actually doing is minimizing a loss
function. Right.
>> Loss of W. Sorry, I got things mixed up.
Thank you.
>> Yeah.
Uh how do we define the initial weights
for the neural network?
>> Ah.
So, yeah, the initial weights um
So, there's a there are many ways to So,
first of all, they are initialized
randomly.
Uh but randomly doesn't mean you can
just pick any random weight. There are
actually some good ways to randomly pick
the weights. Uh those are called
initialization schemes. Um and there are
a bunch of very effective initialization
schemes people have figured out over the
years and those things are baked into
Keras as the default.
So, the Keras, I believe, uses something
called the
uh He initialization, H E
initialization, or the Xavier Glorot
initialization. I wouldn't worry about
it. Just go with the default
initialization.
The reason why they have to be very
careful about how these weights are
initialized is because if you have a
very big network and if you initialize
badly then
the gradient will just explode as you
calculate it.
The earlier layers, the weights will
have massive gradients or the gradients
will vanish.
So, they're called the exploding
gradient problem or the vanishing
gradient problem. To avoid all those
things, researchers have figured out
some clever way to initialize so that
it's well-behaved throughout.
Yep.
If using um backprops and GPUs was so
critical, I'm just curious like who
first did it and when? Was this like a
couple years ago? Was it a company? Was
it a Yeah.
>> Yeah. Well, GPUs have been used for deep
learning, I want to say um
I think the first uh case may have been
in the mid 2005, 2006 sort of thing.
But I would say that it sort of burst
out onto the world stage and made
everyone take notice when uh a deep
learning model called AlexNet
in 2012 won a very famous
computer vision competition.
Uh and it beat the and it set a world
record for how good it was.
Uh and that's when everyone was like,
"Hey, what is this thing?" And that's
really when it burst onto the world
stage. I'll talk a bit more about it
when I get into the computer vision
segment of the class.
But you can Google AlexNet and you'll
find a whole bunch of history around it.
I believe that if you do this, is it
true that could get to a global minima
that would mean there would be no
hallucinations?
Aha, good question.
So, if it is perfect
if you get to a global minimum. First of
all, global minima doesn't mean the
model is perfect, right? It may still
have some loss.
Um
but global minima is going to be on the
training data.
You can imagine that the test data,
future data has its own loss function,
right?
So, what is minimum here may not be
minimum there. That's the problem.
Is that a comment? No, okay.
Just saying that
uh that would mean that also you can be
over-fitting for
>> Correct. Exactly. Exactly. So, if you
overdo, if you find the best thing in
the training function, chances are it
doesn't match the best thing of the test
data.
So, on the test data, you're actually
doing badly.
Okay. So,
uh come back to this.
Okay. Now, uh the final uh twist to the
tail here uh we're going to go from
something gradient descent to something
called stochastic gradient descent. And
stochastic gradient descent or SGD is
the workhorse for all deep learning.
Okay?
And funnily enough, SGD is simpler than
GD.
Okay? Just when you thought it couldn't
get simpler, right?
Okay. So,
So, for large data sets, computing the
gradient of the loss function can be
very expensive. Right? Needless to say.
Because it has to be done at every step
and the cardinality of the data set is
really big. Right? And you may have, I
don't know, billions of parameters. It's
just very, very
tough to compute it even with backprop.
So, the solution is at each iteration,
when I say iteration, I'm talking about
this step of gradient descent.
Instead of using all the data
instead of calculating the loss function
by averaging the loss across all N data
points and then calculating the gradient
of that thing, what you do is you just
choose a small sample randomly. You
choose just a few of the N observations
and we call it a mini batch.
So, for example, the number of data
points you may you may have 10 billion
data points
but in every iteration, you may
literally grab just like 32 or 64,
something really small.
Like absurdly small.
Okay?
And then you pretend that okay, that's
all the data I have. You calculate the
loss, find the gradient and just use
that here instead.
Okay? So, this is called stochastic
gradient descent. So, strictly speaking
theoretically, SGD uses just one data
point.
But in practice, we use what's called a
mini batch, 32, 64, whatever.
Uh and so, mini batch gradient descent
is just loosely called stochastic
gradient descent, SGD.
So, and SGD, as it turns out
you can see it's clearly very efficient,
right? Because
it's just processing a few at a time.
Uh and in fact, if you have a lot of
data
and you calculate the full gradient of
the loss function, it may not even fit
into memory.
Right? It's really problematic. But with
SGD, it says, "I don't care whether you
have a billion data points or a trillion
data points. Just give me 32 at a time."
Okay? And you just keep on doing it.
And
turns out, because not all the points
are used in the calculation this only
approximates the true gradient. Right?
It's only an approximation. It's not the
real thing. It's only an approximation.
But it works extremely well in practice.
Extremely well in practice.
And there's a whole bunch of research
that goes into why is it so effective?
And you know, people are discovering
interesting things about SGD, but we
don't have like a definitive theory as
to why it's so good yet. We have some
interesting, you know, uh research
threads that have happened.
And very tantalizingly, very
tantalizingly
because it's only an approximation of
the true gradient
SGD can actually escape local minima.
So,
in the in the true loss function, you're
at a local minimum
but in SGD's loss function, when you're
doing SGD, you're reaching the the
minimum of the SGD loss function
which actually may not be the actual
loss function. So, as you're moving
around, you're actually jumping from
local minima to local minima of the
actual loss function.
I know that's a mouthful. I'm happy to
tell you more. It's just a side thing
that I just wanted you to be aware of.
Okay?
One of the reasons why SGD is actually
effective. It's almost like you work
less and you do better.
How many times does it happen in life?
This is one of them.
Okay? Now, SGD comes in many flavors.
Uh many siblings. It's got a lot of
siblings and variations. It's a big
family. Uh and we're going to use a
particular flavor called Adam
as our default in this course and I'll
get back to it when we get into the
co-labs and things like that.
All right.
Um
By the way
you know how you know all these pictures
I've been showing you a nice little
function like that, a little bowl and so
on.
This is a visualization
of an actual neural network loss
function.
You can see like the hills and valleys
and the cracks and so on and so forth.
Okay? And you can check out the paper to
get more insight into how they actually,
you know, came up with this
visualization. It's crazy.
It's complicated.
Yep.
So, for for SGD, do you perform the
iterations until you minimize the loss
function for each mini batch and then
move to another mini batch? Yeah, so
what you do is you take each mini batch
and then
you calculate the loss for the mini
batch, you find the gradient.
And use the gradient and update the W.
Then you pick up the next mini batch. So
you don't you don't pick a mini batch
and try to perform the iterations on
that mini batch until you reach the
You Each mini batch, one iteration. Each
mini batch, one iteration. Because if
you do a lot of iterations on one mini
batch,
first of all, you'll never be sure that
you're going to find any optimal
solution because you're not guaranteed
of any global minima. And secondly, it's
much better for you to get new
information constantly because what you
can do is you can revisit that mini
batch later on.
Right? And that gets into these things
called epochs and batch size and so on,
which we'll get into a lot of gory
detail when we do the collab.
So let's revisit that question. It's a
good question.
Yeah.
When you do the backprop process, Very
good. Backprop. Not backpropagation.
Nice. I made sure.
>> Yes.
Well, it's it sounded like you started
from the layers that were closest to the
output and you went backward. Okay. And
um my question is are you doing that
once or is it looping multiple times and
then
>> do it once. Just once. Yeah. So for each
gradient calculation, you do it once.
Why does it Why does it want to start
from the layer that's closest or why do
you want to start it from the layer
that's closest to the output?
>> Yeah. So basically what happens is let's
say that just for argument that you go
go in the reverse direction.
You will discover that a lot of paths to
go from the left to the right will end
up calculating certain intermediate
quantities including the very final
gradient sort of item
again and again and again.
Same thing is going to get calculated
again and again and again. So by
starting from the end and working
backwards, you just reuse stuff you've
already calculated.
So that is sort of the rough idea. But
if you see my PDF, I've actually worked
out the example and you and that will
demonstrate what I'm talking about.
By the way, this gradient the backprop
is just a sort of a
Like in calculus, we have something
called the chain rule.
To calculate the derivative of a
complicated function, you calculate the
calculate derivative of like the outer
function then the inner function and so
on and so forth. The backprop is
essentially a way to organize the chain
rule to work with the neural network
layer-by-layer architecture. That's all.
So is it Is it fair to say that once we
are finding like the local minimum, we
are not optimizing to all the GWs
because like this local minimum is
coming like from different curves, from
different lines. So
Is that fair to say? When we are using
stochastic gradient descent, yes. So for
in stochastic gradient descent, when you
take say 32 data points from a million
and you're calculating the loss for that
32 data points, you're basically trying
to do a gradient step.
Right? The W equals W minus alpha
gradient thing. You're doing it for that
that 32 points loss function.
Right? Which is not the 1 million points
loss function.
That's why it's approximate.
But the approximation, instead of
hurting you, actually helps you because
it helps you escape the local minima of
the global loss function.
So it's it's sort of an interesting and
somewhat technically subtle point, which
is why I'm not getting into it too much,
but I'm happy to give pointers if people
are interested. Yeah?
Uh when you say you initialize the
weights, you initialize for the whole
network or just the end layer and then
go backwards like you
>> No, you initialize everything in one
shot.
Because if you don't initialize
everything in one shot, what's going to
happen is that you can't do like the
forward computation to find the
prediction.
Uh and so they are done independently
and the initialization schemes will take
into account, okay, I'm initializing the
weights between a layer which has 10
nodes and on one side and 32 on the
other side and the 10 and the 32
actually play a role in how you
initialize.
Okay. So um so the summary of the
overall training flow
is that, you know, you have an input.
It goes through a bunch of layers. You
come up with a prediction. You compare
it to the true values and these two
things go into the loss function
calculation. You get a loss number.
Right? And you do it for say 10 points
or 32 points or a million points. And
this loss thing goes into the optimizer,
which calculates the gradient. And once
it calculates the gradient, it updates
the weights of every layer using the W
equals W minus alpha times gradient
formula, gradient descent formula. And
then you keep it doing this again and
again and again.
This is the overall flow.
This is how our little network is going
to get built for heart disease
prediction. This is how GPT-4 was built.
And this is how AlphaFold was built.
And AlphaGo was built.
You get the idea.
I mean, it's astonishing, frankly.
If you're not getting goosebumps at the
thought that this simple thing can do
all these complicated things, we really
need to talk offline.
Uh there was a hand raised here. Yeah.
Sorry. Just quickly, this is for each
mini batch, right? So
my question is if you came with
different weight for each mini batch,
how do you
add it up?
The like, okay, this weight has is the
perfect combination for this mini batch,
but you have weight different
weight for another mini batch. How do
you combine those two? No.
On each point, what you do is you you
find the you find you you you start with
a weight.
You run it through for a mini batch. You
come up with the loss function. You
calculate the gradient.
And now using the gradient, you've
updated the weight. Now you have a new
set of weights, right? Which is the
updated weights. Call it
W2 instead of W1.
Now W2 is is your network and when you
take the next mini batch, it's going to
use W2 to calculate the prediction.
And this this whole flow will become a
lot clearer when we do the collabs.
Okay. So we have 3 minutes.
I don't want to go into
regularization overfitting in 3 minutes.
So let's have some more questions.
Yeah.
Can you use any activation function as
long as it gives like positive values?
For like X squared or mod X or
something. Um you can use a variety of
activation functions.
Um
There is uh but yeah, there's a whole
literature on, you know, the pros and
cons of various activation functions
that you could use.
But in general, you have to make sure of
a couple of things. One is that when you
do backprop,
the gradient is going to flow through
the activation function in the reverse
direction.
And the activation function should
actually sort of make sure the gradient
doesn't get squished.
It shouldn't get squished. It shouldn't
get exploded.
So those are some considerations and
these are technical considerations, but
those all those considerations have to
be taken into account. If you can take
those into account, then you're okay.
That's sort of the key thing to keep in
mind.
And that's in fact why the ReLU is
actually very popular
because as long as the value is
positive, the gradient of the ReLU is
just one. Right?
Uh because
So if you look at something
Oops.
Was it frozen?
I jinxed it.
So sorry, livestream.
If you have something like this,
the ReLU is like that, right?
So the gradient here
is always going to be one.
Which means that as long as the value is
positive, whatever gradient comes in
like this, it just like gets multiplied
by one and gets pushed out the other
side. So it doesn't get it doesn't get
harmed or squished or anything like
that. Um so that's one reason why the
ReLU is very popular because it
preserves the gradient while injecting
almost like the minimum amount of
non-linearity to do interesting things.
Um yeah.
If you have a high number of dimensions,
can you do mini batching on like
features dimensions instead of just
observations and keep the same number of
observations, but just take a small
sample of the number of features that
you're actually using? Oh, I see. I see.
So you're saying let's say you have 10
features.
Um instead of taking all data points of
10 features, what if you have choose
five features and just use them and do
the thing
as long as you can actually compute the
prediction.
To compute the prediction, you may need
all 10 features.
Right? Or you need to have some defaults
for those features.
And by if you define defaults for those
other five features, you're basically
using all all features.
So that's the key thing. Can you
actually calculate the prediction
by manipulating? And typically, you
can't.
All right?
Okay, folks. 9:55. I'm done. Have a
great rest of your week. I'll see you on
Monday.