Okay. All right. Let's get going. Uh
[clears throat] today is going to be
packed. uh I'm going to spend the first
roughly half of the lecture on uh
actually building a model a car corass
model in collab to solve the heart
disease problem we saw earlier and then
switch gears halfway and then talk about
uh how to solve image classification
okay so we're going to do two collabs
today uh I've been talking about collab
collab right I've been teasing you we'll
actually do collabs today all right so
summary of baby by the way I've shut off
the lights in the top because when I
switch to collab it's going be much
better for you folks particularly the
folks in the back to be able to see it.
Okay, but I hope you can see the slide
right now. Yes.
Okay, great. So this is just a quick
recap of what we did last class. U you
know broadly speaking training a neural
network essentially is no different than
training other kinds of models. We have
a bunch of parameters i.e weights and
biases and we need to use the data to
find good values of those weights. And
what does good mean? Typically it means
that we define some measure of
discrepancy between what the model
predicts for a given set of weights and
what the right answer is what the ground
truth answer is and then we try to find
weights that minimize this discrepancy
that's it and this notion of a
discrepancy is called a loss function
right so the broadly speaking the
overall training flow is that you define
some network it has an input it goes
through a bunch of layers you come up
with some predictions you take the
predictions you take the true values and
then those two go into the loss function
i.e i.e. the discrepancy function and
then you come up with the loss score and
then you send it to the optimizer which
then proceeds to calculate the gradient
of this loss function with respect to
all the parameters and then it updates
all the weights using that gradient and
then this process repeats. That's it. So
that is the training flow. Okay, quick
recap. Now we also talked about the
optimization algorithm we're going to
use which is called gradient descent.
and gradient descent. As you noticed in
each iteration, every data point is
being used to make predictions and
therefore to calculate the loss and then
to calculate the gradient. And then we
pointed out that gradient descent is
actually not as good as something called
stochastic gradient descent. Stoastic
gradient descent where we instead of
choosing taking all the points, we just
randomly choose a small number of
points. Pretend for a moment as if those
are the only points we have. make
predictions, calculate loss, calculate
gradient and go on. So that was the
basic idea behind stochastic gradient
descent, right? Two different kinds of
things. Now what it means is that when
we actually start training the model, as
we will in a few minutes, the way
because we only take a few points at a
time, we have to be a bit careful in
what's going on. And I want to make sure
you clearly understand what the
differences are before we actually get
to the collab. Okay. And
all right. So there is the notion of an
epoch.
An epoch essentially just means that we
make one pass through the training data.
All the training data we make one pass
through it. Okay. And so what is one
pass is that if you have something like
gradient descent, one pass means every
data point is sent through the network.
We calculate its predictions, calculate
the loss, calculate the gradient, right?
We run every training sample through it.
we calculate the gradient which is just
this thing here right I mean I will
sometimes say d of loss time dwerative
of loss with respect to w sometimes I
might use this naba symbol these are all
interchangeable okay so we'll calculate
the gradient and then we update using
some version of this okay but we just do
it once at the end of the epoch because
if you have 10 billion data points every
one of them flows through you get 10
billion outputs and then we calculate
the epoch just one at the end of this
thing we calculate the gradient and
update once one update per epoch. Yes.
Now in stoastic gradient descent what we
do is that we process the data in
batches
small numbers of points at a time right
and these are called technically
speaking they're called mini batches I
don't know about you I just get tired of
saying mini batches I'm just going to
say batches from this point on okay and
in fact that is widely done in the
literature so we'll so we'll have to
process it in batches so we take the
training data and then we divide it up
into batches
batch one, batch two all the way till
the final batch. And so what we do is we
for each batch we basically do gradient
descent for each batch we take batch one
and then we run just the training
samples in that batch through the
network to get predictions. We calculate
the gradient we update the parameters
and then we go to batch two then we go
to batch three and so on and so forth.
So pictorially this is how it's going to
look like
right let's say the first batch is say
32 points we take those 32 points we run
it through the network get all the stuff
out we calculate the gradient update the
weights so when we now get to batch two
the weights have changed
they have been updated and then we do
the same thing for batch two batch three
and all the way till we get to the end
of the thing and when we are done with
this thing this whole thing is called a
what
an epoch [clears throat]
This whole thing is an epoch. Okay.
All right. Now, so the question of
course is that if you have a bunch of
data points and you're going to run
stoastic gradient descent on it in in a
in a particular epoch, how many batches
are going to be there? Okay, how many
batches are going to be there? Now,
Keras is going to calculate all this
stuff. You don't have to worry about it,
but you just need to understand exactly
what happens. Okay, so my philosophy, by
the way, is that you have to know the
details of what's going on. If you don't
know the details, if you haven't figured
out at least once, you will not actually
be able to think new and creative
thoughts for a new problem. Okay, it's
because the concepts are not manipulable
in your head yet. Okay,
please use the microphone.
So when we talk about SG, so and we
talking about uh we are only taking some
part of it. Is it what we are saying is
that we only take some variables or we
only taking some part of the data.
>> We are taking some rows.
Okay. We taking only right. So that data
points that means a batch.
>> Exactly. So for example, let's say you
have a thousand data points, right?
Thousand rows of observations, thousand
patients in the heart disease example or
a thousand images that you're trying to
classify. You take let's say 32 of those
images, 32 of those patients and that's
a batch. Then you go to the next 32.
Then the next 32 and so on and so forth
till you run out of patients or run out
of images.
>> And each iterative time you are updating
with the weights new weights that you've
got.
>> And it means you keep connecting it or
keep moving towards
>> you're basically updating the weights as
you
>> updating the weights
>> and what we calling the epoch is
ultimately the equation of loss function
that we are trying to do.
>> No an epoch. See the the thing to
remember is that here this whole thing
is called an epoch because we have to do
one full pass through the training data.
Okay. But within that epoch we update
the weights many times. Basically we
update the weights as many times as we
have batches.
All right. Um
so to go here let's say for example
basically the idea is that you take the
training tech you divide it by the batch
size and you choose the batch size okay
you choose the bat size and we'll talk
about well how do you choose that later
on you choose the batch size and once
you choose size just divide it and round
it up so for example as you will see in
the collabing set is going to be 194
patients and then we're going to choose
a batch size of 32 and we typically tend
to choose batch sizes of 32 64 and
things like that because it actually
aligns very well with the nature of the
parallel hardware we're going to use.
Okay. And so here 32 and so on. So
divide 194 by 32 you get 6 point
something. You round it up to seven.
Okay. And so what that means is that the
first six batches will have 32 samples
each. And then the final batch has only
two samples left. And that's okay. It
can be a nice little small batch at the
end.
There's nothing that says that every
batch has to be the same size.
>> That's it. Epoch batches.
>> And are you like for each batch you run
through the whole network like all the
layers or like each layer is one batch?
>> No, for a batch you run it through the
entire network. So the way I think about
it is that you take a batch right just
momentarily you assume that's all the
data you have
just run it through the network because
unless you run it through the every
layer of the network you can't get a
prediction and unless you get a
prediction you can't calculate the loss
and unless you calculate the loss you
can't calculate the gradient unless you
calculate the gradient you can't update
the weights
>> last thing but if you're using like all
the data just doing the gradient descent
then you just go through the network
once right
>> okay exactly so in Gradient descent one
epoch is one pass and one weight update.
In many in stoastic gradient descent the
number of updates you make is equal to
the number of batches you have which
ends up being you know some the training
set divided by the batch size rounded
up.
>> So just to confirm so initially when we
introduced like the concept of batches
the whole purpose was not to run through
all the data and be able to do some
prediction from a subset. So now like
the advantage is that like after batch
one we are using more accurate
coefficient to run through batch two and
so on. That's really the advantage of it
or there's something else to it.
>> Perfectly set. That's exactly the
advantage. So we take a small amount of
data and we say hey we know this is not
all the data. It's just a small subset
of the data. So therefore it's not going
to be super accurate. It's going to be
approximate but it's okay. So we'll
still tend to move in the in the right
direction. So instead of waiting for the
whole thing to get done and then
updating it, we're just going to update
it as we go along.
All right. Uh yes,
>> building on to her question, is it that
uh doing this process for SG will uh
render us a more better solution or
requires less compute power?
>> Both
>> both and the reasons for both are in the
previous lecture. Yeah. And I'm saying
that instead of repeating it just
because I'm like very pressed for time
today. That's why uh all right cool so
that's what we have uh are we good
okay so now we come to the last step
before we actually fire up the collab
which is overfitting and regularization
um so if you remember from your machine
learning background um when your model
gets more and more complex
right if you you know using
use a simple model then you use a more
complex model and so on and so forth
what happens to the error on the
training data Typically what happens to
the error on the training data? So let's
say you have a simple regression model,
you get some error and then you have a
regression model in which you use all
kinds of interaction terms. You use
logarithms and this and that and make it
super complicated. What do you think is
going to happen to the error on the
training data?
>> Right? Basically it's going to go down
as the model get more gets more complex.
Correct. Now of course comes the punch
line which is what what do you think is
going to happen to the training data? I
showed you the answer.
Right? Basically, what's going to happen
typically, at least conceptually, is
that it's going to get better and better
at some point. It's going to bottom out
and it's going to start climbing again.
And so, we typically refer to this
phenomenon here when it starts to climb
again as overfitting because the model
is essentially fitting to the
idiosyncrasies of the training data as
opposed to generalizing patterns. And
then in this thing we call it
underfitting because it can still
there's a lot of potential to improve
and we really are hoping to find the
sweet spot in the middle right that's
the basic idea of overfitting
underfitting and the way we and to to
relate this to neural networks as you
see as you as you've learned so far you
have to learn smart representations of
the input data and to do that we I have
argued that you need to have lots of
layers in your network the more layers
you have the better things get. GPT3 for
example has 96 layers if I recall right
more layers the better but more layers
means more parameters more parameters
means more complexity to the model and
therefore more chance of overfitting
okay so it's really important in neural
networks that we think about
regularization and regularization you
will recall from your machine learning
background is the way we handle the risk
of overfitting and try to find models
that fit just right okay and so several
regularization methods have been
developed over the years and we are
going to use only two of them. The first
one is called early stopping. uh and
this is this has been famously referred
to uh by Jeff Hinton who's one of the
pioneers or as he's more colorfully
known one of the godfathers of deep
learning um who won he also won the
touring a few years ago as the own sort
of a beautiful free lunch right that's
what he calls it so the idea is very
simple we take a validation set we take
the training data we split into a
training and a validation set and then
we just keep you know doing gradient
descent boop b the training will
hopefully keep on getting better and
better lower and lower error
And then we just keep track of what's
going on in the validation set. And then
at some point if it starts to flatten
out and start to climb, we just say,
"Okay, that's when we stop training."
Right? And what we're going to do in the
collab is actually run it through the
whole thing, see where it flattens out,
and then we say, "Okay, that's why we
should stop." But of course, you don't
want to go all the way to the end and
then go back and say, "Well, I want to
stop at the 10th epoch." And there are
ways you can use Keras to be very
efficient about this. But the
fundamental idea is you take the
training data, split it into training
and validation and just track what's
going on in the validation set to see
whether this kind of bottoming out
happens. Okay. So this is called early
stopping. And the other way we're going
to do right this called early stopping.
We're looking for this part. The other
thing is called dropout. And I'm going
to come back to dropout when we do when
on Wednesday's lecture because that's
the first time we're going to use it.
And so I'll come back to draw port and
tell you exactly how it works. It's a
very very clever strategy. But we will
not use it today. We'll use it on
Wednesday. Okay. So in summary, uh what
do we do? We get the data ready. We
design the network, number of hidden
layers, number of neurons and so on and
so forth. We pick the right output
layer. We pick the right loss function.
Uh we choose an optimizer. As I
mentioned earlier, SGD comes in lots of
flavors, lots of variations on the
theme. And empirically much like for
hidden layer neurons we t tend to use
relu as the activation function for
optimization we tend to use a flavor of
HGD called Adam okay as sort of the
default because it's really good so
we'll use Adam as you'll see we
typically use either uh early stopping
or dropout and then you just fire it up
and start training in terasen tensorflow
all right so that is the training loop
now I'm going to switch gears and give
you a quick intro to teras and teras
TensorFlow. Okay. Keras and Tensor. No,
TensorFlow and KAS. Thank you. Um, and
then we'll actually fire up the collab.
So, first of all, what's a tensor?
>> Yeah, I just quick question on the
previous thing like if you're looking at
the validation set to avoid overfitting,
but aren't you actually like over
actually overfitting because like you're
kind of using the validation set as a
training set or not?
>> Uh, no, no, no. The validation set is
never used to calculate any gradients.
It's only used to calculate accuracy and
loss.
Yeah. Yeah. It's kept aside and only
used for evaluation, not for training.
That's what keeps you honest.
>> Right.
>> And this will become clear when we
actually go to the collab. So what's a
tensor?
>> All right.
Okay.
Tensor is the input data which you're
giving to the system. It could be in
various formats like it's image it could
be like we call it a 4D tensor. If it's
a time series data, it's 3D. And
typically, if you just send numbers in,
it becomes a vector which would go
inside which each each it gives the
value of the
uh uh the variable as well the values of
the variables associated to it as well
as
uh as well as the I mean information you
want to get to.
>> You're kind of on the right track, but
not entirely, right? It's actually a
simpler concept than that. So, uh
>> it's like a matrix but generalized with
higher dimensions.
>> Correct? That's also actually correct
but incomplete. The reason is because it
can be simpler than a matrix. It's not
matrix or higher. It's actually could be
simpler. In fact, you take a number,
it's actually a tensor.
All right? The simplest case of a tensor
is a number. The next simpler case is a
vector which is a list. The next higher
case is a table.
Okay, so these are all tensors. So
tensors basically are a generalization
of the notion of both a number, a vector
and a table to higher dimensions.
Okay, so you can think of a tensor as
having what are called every tensor has
something called a rank, right? So a
number is just a number. It doesn't have
a dimensionality to it. So it has got
rank zero. Okay. While a vector it's a
list of numbers. You can sort of write
it down top to bottom and it's one
dimension. Right? So that dimension that
one dimension is called a rank. So it's
called rank one. A table is 2D
two-dimensional. So it's called rank
two.
And you can have a rank three which is
just a bunch of tables.
A bunch of tables is a rank three
tensor. We also think of it as a cube.
Okay. So these things are very useful
because obviously we are all familiar
with vectors. Uh as you will see very
shortly later in this class black and
white grayscale images are usually
represented using tables of numbers like
this. Color images are represented using
three tables.
Okay. Can you get think of what might be
representable as you know a tensil of
rank four? Meaning every element of a
tensor of rank four is actually a color
picture.
Just shout it out. Video. Exactly. What
is a video? A video is basically a
stream of black color images. A color
video. So each element of that stream,
right? What the first dimension of the
tensor is which frame it is and then
everything else is the actual frame. So
the way I u think about these tensors
always is
tensor you can just think of it as a you
can think of a tensor as being this
array which has all these axes or
dimensions. This is the first one. This
is the second one. This is a third swan.
Right? This is a tensor of rank four.
Okay? 1 2 3 4. And so if you have a
vector, right? So you can imagine if
it's just a vector, you can imagine the
vector actually living like this, just a
list of numbers, right?
But if it's just if it is just
a 2D a rank two tensor right which is
just like that right which is just like
that
so this thing becomes you know like that
and that thing becomes like that. So for
example if this is a 7a 3 that means
that there are
seven rows and three columns.
So you get the idea. So the way you
think about tensor is always as if this
open square bracket a bunch of things a
closed square bracket and that's really
what a tensor object is. So what that
means is that anytime you have a tensor
right anytime you have a tensor however
complicated it is you can always create
a more complicated tensor by if you want
to take a list of those tensors let's
say that you have a list of videos
each video is a rank four tensor so
which means a list of videos is what
rank
Exactly. So a a tensor of rank say 10 is
just a list of rank nine tensors.
So that is this that is the most
important thing you need to understand
about tensors. So at any point in time
if I give you a tensor you can just
iterate through the first dimension of
it the first aspect of it and as as you
go through each one of these values. So
for example here um
yeah that can do it.
So
so if you have this tensor here
and if you want to create a more
complicated tensor no problem.
So you add another dimension here. Okay.
Now it just becomes this dimension let's
say has nine values.
one on the nine. So you put zero here
and then what do you get? This whole
tensor is a rank four tensor. And you
put a one here, it's another rank four
tensor. You put a two here, another rank
four tensor. So every tensor, you take
the first element, it's just a list, but
it's a list of the next downrank tensor.
Okay. Now this tensor concept is
actually something Einstein came up
with. Um and so u it's simultaneously
kind of easy to understand and also
slippery. So I would actually encourage
you to read the book which has a really
good discussion of tensors and the more
you practice with it the easier it'll
get. Okay. So if you feel you kind of
understood but not quite you're not
alone. It happens to all of us right?
You have to pay the price or go through
the crucible. Okay. Okay. All right.
So to come back to this
that's what we have
and we already talked about a rank four
tensor it's a video so 2.2 two the text
has a lot more detail. You should
definitely read it. U so here tensorflow
is a library and as you can imagine
neural networks tensors come in and go
through the network and go out the other
end right and since tensors capture
everything numbers lists uh tables and
so on and so forth it's just tensors
flowing from input to output hence it's
called tensorflow and it gives you a
couple of things which are really really
important which is why we use it. The
first one is that it'll automatically
calculate gradients for you of
arbitrarily complicated loss functions.
You don't have to calculate the gradient
because calculating the gradient is very
painful, right? It'll automatically
calculate the gradients for you. That's
the best part. You don't have to use the
chain rule. You don't do anything. The
second thing it'll do, it gives you all
these optimizers including SGD and all
its variations. So you don't have to
worry about the optimization itself.
It'll just you can just pick and choose
what you want. Third, if you have a lot
of servers, it'll actually take the
computational load and distribute it
across all those servers. People here
with the CS background know that
parallelizing computation is actually a
very difficult problem, right? There are
things which are called embarrassingly
parallel. Many things are not actually
quite tricky to figure it out. We don't
know how to figure it out. TensorFlow
will figure it out. Okay? And then
finally, I talked about the fact that
there are these things called GPUs,
graphics processing units, which are
parallel hardware. uh and so it'll even
if you have just one computer but it has
GPUs there's a particular way in which
you have to take your computation and
organize it to really exploit the fact
that you have a GPU and so TensorFlow
will actually do it for you out of the
box automatically you don't have to
worry about any of that stuff okay so
those are all the advantages of this
thing by the way TPU is called a tensor
processing unit it's something that it's
kind of you can think of it as Google's
GPU right they came up with their own
variation on the theme okay now keras
sits on top of TensorFlow, right?
TensorFlow, this is the this is the
hardware you have. TensorFlow sits on
top of the hardware. Keras sits on top
of TensorFlow and it basically gives you
a whole bunch of convenience features.
So, for example, it gives you the notion
of a layer, right? We already saw
Keras.dense is a dense layer, right? It
gives you the notion of a layer. It
gives you the notion of activation
functions and so on and so forth. It
gives you easy ways to pre-process the
data, easy ways to train the model,
report on metrics, you know, calculate
validation loss, validation accuracy,
training loss, all the metrics we care
about. And then it also gives you a
whole library of pre-trained models that
you can just use and adapt for your
particular problem. So it gives you a
whole bunch of conveniences and that's
why it's very popular. And by the way,
you know, many of you might also be
familiar with PyTorch, which is a
fantastic framework as well for deep
learning. And the reason we chose to go
with TensorFlow for this course rather
than PyTorch is because we wanted to
make the course uh sort of accessible to
folks who don't have a ton of
programming background before coming to
the class. And Pyarch is a bit more
demanding from a CS perspective. It
requires more knowledge of
object-oriented programming. Uh which is
why we decided to go with TensorFlow and
KAS because I think it's actually as
powerful uh in many ways and it's a
little easier to get going. Okay, so
that's what we have here. And one other
thing I will mention is that there are
three ways in which you can use kas.
There are three kinds of APIs.
Sequential, functional, subclassing. And
we'll almost exclusively use the
functional API. Okay. And in fact, the
model we built for heart disease
prediction uses the functional API. And
so just read 722 of the textbook to
understand in detail how the API works.
I find in my own work, the functional
API is basically all I need. I don't
need to do anything more complicated
than that. Um and and as you will see as
you work on the homeworks uh and on your
project that it's is it's sort of a
beautifully designed Lego block
environment for doing these things and
you can create very complicated models
very easily. Okay. Uh there's a whole
bunch of stuff here on these websites.
So check them out. There's lots of
collabs or uh uh are available. So now
if you go back to the neural model for
heart disease prediction, this is what
we came up with in the last class,
right? uh we had an input layer, one
dense layer with 16 neurons, rel
neurons, an output layer with the
sigmoid and then boom, that was a model.
So let's train this model. Uh and so the
training checklist is that uh we have
already done this hidden layer of 16
neurons uh sigmoid. We need to use an
appropriate loss function based on the
type of output. What loss function
should we use?
What is the output here?
It's a binary classification problem. So
what should the the loss function be?
Kind of heard it somewhere. Get shout it
out.
No, the output is a sigmoid. The loss
functionary
cross entropy.
Okay, remember if if you're predicting a
number an arbitrary number, you can use
something like mean square error. If
you're predicting a probability which
has to be compared to a 01 output, which
is what binary classification is all
about. we use binary cross entropy.
Okay, so that's what we do here. So we
do binary cross entropy
and then we will go with Adam, right?
And then we'll use early stopping to
make sure we don't over fit. Okay, I
know this like okay I promise this is a
lot literally the last slide before I go
to the collab. I feel like one of those
used cars here but wait there is more.
So anyway, u so uh don't worry if you
don't understand every detail of what
I'm going to go through. I'm going to
link to the collab as soon as the class
is over. But once you get your hands on
the collab, make sure you actually go
through every line in the collab. What I
typically do when I'm trying to learn
something new is I'll actually cut and
paste, right? I won't do that. I won't
actually cut and paste the code and run
it myself. I will retype the code. If
you retype the code as opposed to
cutting and pasting, trust me, you'll
learn a lot more. Right? So I strongly
encourage you to do it that way.
Um and so all the collabs you're going
to publish in the class, uh the first
thing you should do is you should just
make your own copy of the notebook,
right? Copy to drive. And then if you're
using anything other than today's
collab, uh right, anything involving
natural language processing or vision,
you probably should use a GPU. So just
go into go in here, choose the runtime
to be a GPU. Um and then you start your
notebook and you're done. And the second
time onwards, you can just go directly
to this step. You don't have to do all
this stuff for that particular notebook.
And there are numerous tutorials like
five minute videos and so on on how to
use collab. Just just do that. I'm not
going to spend time on it here.
All right. Okay. So, uh I just ran it um
a few hours ago. I'm not going to run
every cell now because it's going to
take some time. It's going to get in the
way of the class time, but I'm going to
just like, you know, go through it
slowly and explain what's going on. So,
here this is just an introduction to the
data set. We already saw this
introduction in the last last week. We
have whatever 303 patients, hot
patients. We have a whole bunch of uh
variables here, age, demographics, and a
whole bunch of biomarker information.
And this is a target variable. Okay? Uh
zero or one, heart disease, yes or no.
And so, by the way, just some technical
prelim preliminaries here. Basically,
every time we load these things, we're
actually going to load these packages.
So you can see here these are the two
key things we need to do. We import
tensorflow first and then from within
tensorflow we import keras. Okay that's
what these two lines do here. Okay. And
then and folks who have done data
science and machine learning a bit
before you you'll know this. We will in
in sort of we will actually load like
the three packages that were just most
commonly used right which is numpy
pandas and mattplot lib. Uh numpy
because it's very easy for manipulating
matrices and arrays and tensors. uh
pandas because often times you get some
data in from somewhere you need to
massage it and wrangle it to a point
where we can actually feed it into ketas
so you need pandas for that and mattplot
lib because you just want to plot you
know uh these loss curves and accuracy
curves to see whether early stopping is
needed okay so that's why we use it uh
so we import all these things and then I
guess the other thing you have to
remember is that when we are training
these deep learning models uh there is
randomness in the process which enters
in a few different places so clearly the
starting values for the these weights
are going to be they're going the
weights are going to be randomly
initialized. Uh and therefore that
that's obviously a source of randomness.
Uh now we talked about how you take if
when you're doing stoastic gradient
descent you take all the data and then
you randomly choose batches right from
this data till we finish a whole pass
through it. Well that immediately raised
the question well well what do you mean
by randomly choose? So typically what we
do in practice is that and kas will take
care of all this for you. um you
basically take the data and just shuffle
it once randomly and then you just go
first 32 next 32 next 32 next 32 like
that okay but it is a source of
randomness and then when we split the
data into train validation testing and
so on uh particularly if you want to
look for early stopping and overfitting
uh we need to again split the data
randomly and that's another source of
randomness and then when we do dropout
which we'll talk about on Wednesday
again dropout has a little bit of a
random element to it and so that's
another source of randomness this. So
all of it all this means is that if
you're working with these models and if
you want to build a model and you want
to hand it off to someone so that they
can reproduce your results well you
better make sure that you sort of you
know make it easy for them to replicate
what you have and the way you do it is
by sending a setting a random seat for
all these things okay and the way you do
it is by having this little handy
function here set random seat uh and of
course you know I use 42 tool like just
like everybody should right so okay so
that's that uh by the way just that's
just a popculture reference to this book
called The Hitchhiker's Guide to the
Galaxy.
>> Number 42 and you'll know what I mean.
Okay, so by the way, um the question
inevitably comes at this point, okay, if
we do exactly this, will you actually
get the exact same numbers that you have
in your version uh of the notebook? And
the answer is hopefully most of the
time, but it's not guaranteed. So this
is called bitwise reproducibility. It's
not guaranteed due to certain hardware
things and device drivers and stuff like
that. So we won't get into all that
stuff. uh and which is why as you see
here uh I have a bit of a fingers
crossed thing. Okay. All right. Cool. So
that's what we have. Um so as it turns
out uh Frantois Shallet who wrote the
book uh the textbook he actually made
this data available in a pandanda's data
frame. So we read the CSV file into this
data frame right there. Uh and then it's
uh and it's 303 rows 14 columns right
and you can see here we'll take a look
at the first few rows. Uh and these are
all the rows. age, gender, cholesterol,
blah blah blah blah blah. And then this
is the target variable right there. U
and the one of the first things I always
do when I'm working with a binary
classification problem is to quickly
check whether the positive and negative
classes are balanced or not. And so what
you can do is you can just quickly check
to see what percent of the data points
is zero versus one. And you can see here
uh 72.6%
of the patients don't have heart
disease. That's a good thing of course.
Uh and then 27.4 have heart disease. So
it's not bad. It's not 50/50 or roughly
50/50. It's a little thing. So, by the
way, quick question. What is a a b good
baseline model for this problem? Suppose
you couldn't use anything any
complicated thing. What's a good
baseline model?
>> Yes. Just predict zero.
>> Yeah. And why would you do that?
>> Uh, it would give you a 72.6% accuracy.
Exactly. Because 72.6% 6% is the sort of
the higher class higher class with the
higher percentage you just predict it
you'll be right on those 72.6% of the
cases you'll be wrong on the rest which
means that your accuracy of this model
is going to be 72.6%.
Okay. And so any fancy model we build
better do you know it's got to do better
than this otherwise it's not worth its
weight uh in layers. Um so all right so
we'll come back to this later. So the
first thing we want to do is we want to
pre-process it because this data set has
both categorical variables and numeric
variables. Um and so it's usually
convenient to just to group them into
two different groups. So I have listed
all the categorical variables here and
the numeric here. Uh and then we have
the pre-processing here. We have to take
the categorical variables and we have to
one hot encode them. And the reason is
that unlike say a decision tree model, a
neural network cannot handle uh
categorical inputs directly. It can only
handle numeric inputs. Which means that
we have to numericalize every
categorical thing that comes in. And the
st there are many ways to do it but the
standard way to do it is one hot
encoding. Um and for the numeric
variables we need to normalize them and
I'll come to that in a second. So pandas
has this get dummies function here and
you can just run this thing and it'll
just hot encode the whole thing. So once
you do that this is what you have. So
you can see here previously um let's say
tal was had three values fixed normal
reversible or something and then you go
to the one hot encoded version u and now
we can see here tal fixed tal normal tal
reversible that's three columns right
that's the one hot encoding in action
okay now the other thing to remember is
that neural networks work best when the
numeric inputs you send them are all in
a relatively small range they shouldn't
have a wide range of variation
Um and so the standard practice is to
standardize the numerical variables. By
standardize, I mean typically subtract
the mean, divide by the standard
deviation. Um we should do that. But
before we do so, we should split the
data into a training set and a test set,
right? And why do we want to split into
a test set? Because at the very end once
we've built the model and done all the
things we want to do with it, we finally
want to take out the test set and
evaluate it once so that we get this
true measure of how it's going to
perform in the wild after you deploy it.
Okay. Uh so you want to divide it 80 80
say 80% training and 20% test set. So
the question is why should we do the
splitting now before we do the
normalization? Why can't we just do the
normalization and then do the splitting?
Um all right
>> because then your uh validation set is
also somewhat dependent on your test set
results as well as the mean of the test
set.
>> Correct? Because the test set has now
essentially sort of has been influenced
by the training set. Right? That is the
the the modeling process part of the
modeling process the splitting and the
splitting also the the the
standardization
if if the standardization which is part
of the process uses information about
the test set well the test set not
really kept away from anything is it
that's why we want to split it lock away
the test set somewhere and then proceed
with the modeling this again this is
like machine learning 101 which is why
I'm going through it pretty fast uh okay
so we we do this uh sampling function
take 20% of the data and make it the
test set and the remaining is going to
be the training set. And when we do
that, you can see the training set is
now 242
um rows while the test is 61 rows. Uh
and any of these data frames, you'll
know that the the shape attribute gives
you the dimensions of the number of rows
in the columns. That's what we're doing
here. And now that we have done that, we
have done the split, we can calculate
the the the mean and the standard
deviation. So I calculate the mean here.
I calculate standard deviation. And
these are all the means. And once I do
that, I just do you know each column
minus the mean divide the standard
deviation. And then once I do that I get
I save them in the train and the test
data frames. And you can see here now
all the numbers are all very sort of
smallalish 0 1 minus one kind of around
that range and that's kind of ideal when
you're network training. Okay. All
right. Right. So at this point the data
is entirely numeric and then uh we are
ready almost ready to feed it into KAS
and the way you do it is you take a
numpy array u you you take a pandas data
frame and then you convert it into a
numpy array and then keras is happy to
take it happy to receive it. So the so
we use this thing called two numpy which
I think is as descriptive as it gets in
programming. Um and then you save it as
train and test. Now train and test on
two numpy arrays with exactly the same
information and now we can fit it into
kas. All right. Now I guess there's one
other thing we need to do which is that
um in this data frame train and test our
independent variables all the features
as well as the target the 01 target.
They're all in this
right and we need to now take it and
just take the the dependent variable the
01 column and split it out and keep the
x and the y separately. Right? That's
the whole point of it, right? Because
you need to feed the X, do the
prediction, and then compare it to the
actual Y and calculate the loss and so
on and so forth. So, uh, so the target
column is our Y variable, and it's
column number six from the left. If you
count it, you can see it. So, we just,
you know, uh, we we delete it from the
the train and test. Um, and now we have
242 rows and 29 columns, 29 features.
You will recall from the network that we
made way back, it had 29 inputs, right?
29 nodes in the input layer. And that's
where the 29 is coming from. And so now
uh we just select the sixth column which
is the target and make it the Y variable
right train Y and test Y. And that is of
course a vector which is 242 long in the
training set and 61 long in the thing.
So at this point all we have done is to
be honest boring pre-processing. Okay,
we haven't actually gotten to the action
yet. Finally, let's do something. So um
and we start with a single hidden layer.
Since it's a binary classification
problem, we'll use sigmoids as we saw
earlier. And this is the model we
created in class last last class. This
is the model we created. Okay. The only
difference between that model and this
model is that I've actually given names
to these layers. And this name thing is
totally optional. Right? If you want to
give a name, give a name. It's just a
little easier to interpret later on.
Okay? It's just cosmetic. Okay? So, uh,
but I've just put it here. U and once
you build the model u you should
immediately run the model dots summary
command because it gives you a nice
overview of the model right what are for
each layer it tells you what the layer
is it tells you what's coming into the
layer meaning the shape of the tensor
that's coming in and what's going out
and how many parameters the layer has
and it turns out this layer has sorry
this network has 497 parameters okay uh
and I have told you repeatedly the first
few times just hadn't calculated the
number of parameters to make sure it
verifies. So we should just make sure
that it is in fact 497. So let's hand
calculate it. And you do basically it's
basically what's going on here. 29
inputs time 16, right? All the arrows 29
* 16 arrows, right? And then you have a
bias of another 16. That's why you have
this expression. And then the next one
is 16 * 1 plus one bias for the output
sigmoid and you get to 497. Okay? Just
make sure you follow this later on when
you work with the collab. We we did this
in class last week and you can visualize
the network graphically as well by using
the plot model function. So we do that
here. Um and let's say it gives you the
same information but in a slightly
easier form to consume and when we work
with larger networks starting on
Wednesday you will see that being able
to visualize the topology of the network
is actually quite handy. Okay, we
finally come to uh actually trying to
train this thing and so what loss
function should we use? uh we need to we
need to use binary cross entropy right
there. What optimizer to use? Well, as I
mentioned earlier, uh we'll use Adam.
Adam.
All right, Adam. Uh and then uh and then
the the final thing is you can ask Keras
to report out whatever metrics you care
about. These metrics are not going to be
used in any optimization. They just it's
just reporting it to you. And the most
common thing people report out for
binary classification is accuracy. So
we'll just go with that metric. Um and
so so what we do is we tell Keras take
the model we just built and compile it
with this choice of optimizer this
choice of loss function and these
metrics. And this compilation step what
it does is it essentially Keras will
take this information and take the model
you have built and it'll reorganize the
model in such a way that the parallel
computing uh distribution of computing
across many servers and so on. That's
that's what's happening in the compile
step. Organizing it so that reorganizing
the model so that it becomes amendable
to parallelization and distribution.
That's what's going on. That's why you
actually have to do something called the
compile step. Okay. And once we do that,
we have finally finally ready to train
the model. And to do that uh we have to
decide what the batch size is that we're
going to use. Remember, we're using some
flavor of SGD, which means we have to
choose what is the bat size. And
typically what people do is that uh 32
is a good default for the batch size.
Like if you don't if you're not just
getting started with something, just use
32. Uh and there's a whole bunch of
literature on what the right batch size
should be for the number of data points
you have, the size of the network and so
on and so forth. My philosophy is start
with 32. Um and you can always try 32,
64, 128. It's kind of like, you know,
oftenimes what people tell me,
researchers tell me is that just use the
biggest batch size that doesn't make
your machine die.
Right? If you can fit into memory, it's
probably good. Just try the biggest
size. We'll just start with 30. It's
just a tiny problem. It's not a big
deal. And then we also have to decide
how many epochs through the data do we
want to go through, right? How many
epochs? And uh you know, usually 20 to
30 epochs is a good starting point. Um
and then because this is a tiny problem
just for kicks, I decided to run it for
300 epochs. Uh just to see if anything
any overfitting is going to happen. Uh
and then whether we want to use a
validation set. Of course, we want to
use a validation set. Uh right. So we
will use 20% of the data points as a
validation set so that we can look for
overfitting underfitting.
All right. So with these decisions made
we finally uh we use the model.fit
command. Model.fit is what actually
trains the neural network. Okay. And you
have to tell it what the x
tensor is. You have to tell it what the
dependent variable y tensor is. We need
to tell it how many epochs to do this.
What the bat size to use. Verbos equals
1 just means like just you know put a
lot of descriptive output as you do this
thing and then validation split means
you know take 20% of the training data
and set it aside as your validation data
set. Don't use it for training because I
want to measure overfitting using that.
So that's it. So you do that thing it
it'll run for 300 epochs and this is the
reason why you know I decided to just
not actually run it in class. Um and so
you keep on doing it gives you a lot of
output and finally
we reach the end.
Okay. Now let's take a moment to
understand what's being reported. So
I'll just take this one line here. So
this there is a there is these two there
is a pair of lines for each epoch. And
then here it's telling you uh you know
it it actually uses in the in this 300th
epoch it used seven batches seven out of
seven batches right so it used seven
batches and if you you will recall from
the math we did in the class that it's
actually seven batches where the first
six batches are 32 and the last batch is
just a couple of examples but we have
seven batches right this is the 193 by
32 rounded up okay so that's why we have
seven here and then it tells you how how
long it took it for that and then it
this is the loss value. This is the
binary cross entropy loss value on the
training set right on on that particular
batch right uh that it calculated this
is the accuracy that you asked you to
report out 98.4% 4% 98.5% accuracy on
that batch and and then at the end of
this epoch using whatever weights were
available in that network it actually
calculate the loss on the validation set
which is the 20% of the data we have set
aside and then it this is the accuracy
on that validation set okay so that's
what each of these numbers mean now
looking at these wall of numbers is kind
of painful so usually you just plot it
um so and the way you do that is if you
if you notice here Uh okay, I'm not
going to go back here. So I said history
equals model.fit blah blah blah blah
blah. And that history object has a lot
of information that we can use for
plotting and diagnostics and so on. And
that history thing uh history object has
another object called history
history.htistory which is a dictionary
with all these values and that's what
we're going to plot. Was there a
question here? Yeah.
>> Uh so you prompted it to keep the size
for validation but didn't we already
keep a test set? So that's going to be a
secondary validation, right?
>> So basically we have a training uh and
then a validation and a test. The role
of the validation set is to figure out
things like early stopping. Should we
stop here? Should we go back? And as you
will see later on, if we use
hyperparameters, you know, we we'll try
different values of the hyperparameters
and figure out use the validation set to
figure out which one is the best one.
But once we are done with all that, we
will finally have a model. At that
point, we open the safe, take out the
test set and use it just once with your
final final model. Not because you want
to improve the model, but because you
want to have a realistic idea how it'll
do when you actually deploy it out in
the real world.
>> Uh yeah.
>> Uh can we use can we instead of accuracy
could we use other metrics uh to
evaluate whether to
>> absolutely like a confusion matrix let's
say?
>> Yeah, you can you can do whatever you
want. You can use like I said it's not
used for training so there is no
mathematical implication what you choose
right you can choose error rates
accuracy f1 fb beta you can do whatever
you want and keras as you will see has
this dizzying list of possible metrics
you can use for reporting the key thing
to remember is you're just reporting
these metrics you're not actually using
them for any training
yeah
>> uh my question is with respect to
validation like uh we've got a training
data set so when we take out 20% This is
the validation uh data for validation.
Are we taking out from the training set
or correct from there that level or we
go to each batch and take out 20% from
the train?
>> No, we're taking it out from the
training set.
>> So it means the batch size the number of
batch number of data would be available
for calculating the batch size will
reduce.
>> Correct. And in [snorts] fact once we
validate take out the validation set
whatever remaining is 193.
>> Okay. And then we divide that into
batches and then that every info uh that
validation and the data gets different
added. Now once you take out the
validation set at the very beginning you
keep it aside and then you only evaluate
at the end of each epoch what your loss
and accuracy is on that validation set.
>> So you don't have cross validation.
>> No no we're not doing any of that stuff.
We're just taking it out once and we're
just evaluating the end of every epoch.
>> Okay. So
yeah. Okay. So I know we both asked
similar questions but
>> so I know both have asked similar
questions but just to reconfirm. So here
my training model is giving me say a
loss of 0860.
My validation model is giving me 660.
That means I've already crossed the U.
So when I have to actually test the
model that is the midpoint which I take
and that will model which will get
deployed in production.
Correct. And as to okay, what do we do
to get that model? Do we actually have
to go go back to the beginning and run
it for a few epochs or can we do
something smarter than that? We'll get
to that.
>> Yeah.
>> Is the validation set different for each
APO or is it the same?
>> It's the same. So what you do is you
have a training set before you do any
training. You take out 20% of it, keep
it aside. You take whatever is left over
that you divide that into mini batches
and then start running it through each
epoch. But at the end of each epoch, you
just evaluate the quality of that
resulting model using the validation
set.
>> What's different between each epoch? Is
it just the way
>> weights have changed?
>> It's the it's the division into the
different
>> uh no so in the difference in each epoch
is the weights have changed.
>> So after every mini batch, the weights
have changed. At the end of one epoch,
you've gone through all the data points
you ever had, right, in the training
set. And then you come back to the
beginning and you do it again.
How do you identify the sweet spot?
>> It's coming.
>> Yeah. All right. So, I'm going to keep
going. So, we have this here. And so,
you just I mean there's a little bit of
mattplot lip code. So, what we do is we
just plot the training loss and the
validation loss as a function of the
number of epochs. Okay? And as you can
see here, the training loss is these
things here. And it's steadily going
down as you would expect. The validation
loss goes down here. And then at some
point it kind of flattens out and then
maybe gently starts to rise. Okay. So do
you think there's overfitting?
>> Right. There seems to be some level of
overfitting here. But the thing you have
to always remember is that the binary
cross entropy loss is a loss function
that is convenient for you because it
sort of captures the thing you want to
capture the discrepancy but also because
it's mathematically convenient but what
you may actually care about in practice
is something like accuracy right so I
always that's why you're reporting out
the accuracy when we do these things so
you should also plot the accuracy to see
what's going on and really you should
look at the accuracy and figure out
overfitting and underfitting and all
stuff. So let's just do that. So I have
here uh overfitting.
Uh okay. So this is how it looks like
for accuracy. Accuracy of course as the
model gets you know as you do more and
more epochs hopefully it get better and
better for training. So you can see here
accuracy actually climbs all the way up
to the mid 90s uh right there small the
low 90s here. the validation gets to
this point after like I don't know 50
epochs maybe and then it kind of
flattens out and then strangely it
climbs up again a bit later right so now
the fact that the accuracy actually got
better at the very end suggests that
maybe we can live with this overfitting
>> okay
>> right it's not the end of the world
right so you can so you can certainly
what you can do is you can go back and
say you know what no I'm going to be a
purist about this around 50 epochs or
so. I think that's when it actually
flattened out for loss. So you can just
go back and just restart the model and
run it only for 50 epochs, not 300 and
then stop and just use that model for
everything from that point on. Or you
can say, you know what, it's okay. I can
live with this thing. Uh and so that's
what we're going to do here. Let me just
stop for a second. There was a question.
>> Yeah,
>> for originally when we were starting
out, we were saying 20 to 30 pods, but
we were going to do 300. 50 is over 20
to 30. So when it comes to validation of
if you run enough epochs, are you doing
like derivative calculations?
>> Oh, I see. No, that's a great question.
So the question is I said start with 20
and 30 epochs as a rule of thumb here,
I'm just going with 300. And because I'm
going with 300, I can actually see some
potential evidence of overfitting. But
if I had done only 20 to 30, maybe I
wouldn't have even seen that. What
happens next? Right? Is that the
question? Great question. So what you
should do is when you look at these
curves if at the end of 30 epochs you
find that the validation loss continues
to drop then you know maybe there is
more room for it to drop. So you you
continue from that point on. The thing
about keras is that you can actually run
the the the fit command at that point
and it'll continue where it left off. It
won't go to the beginning again.
Right? So you can run 10. Okay. The
validation is still getting better and
better. Okay. Run for another 10. It's
getting better and better. Run for
another 10. Getting better and better.
Run for another 10. Oh, it starts to
climb up again. Okay, now I'm going to
back off. That's what you do.
All right. Now, all this manual stuff
I'm going through it just because to
build intuition, there are these things
called callbacks in KAS, which we'll get
to later on in which you can actually
tell it, hey, when the validation loss,
you know, uh, stops improving, stop
everything or when it stops improving,
save that model for me somewhere. So,
they don't have to go back and rerun
everything. It'll just it'll have saved
it for you and you can just pick it up
and use it. Uh yeah.
>> What's the intuition behind um the
accuracy continuing to improve when the
loss is getting higher?
>> Because accuracy and loss are related
but they're not the same thing. Uh in
particular, so it's a really good
question also kind of a profound
question because accuracy is a very
discrete measure, right? So if a
particular point we predicting its
probability to be say 049 we're going to
say okay that's a zero no heart disease
but if it goes to 0.51 we're going to be
oh that's heart disease. So when you go
from 049 to 0.51 the binary cross
entropy loss will change very very
slightly but the accuracy will go from 0
to one dramatic jump. So it's very jumpy
and discreet and that's why it tends to
be a proxy but sort of a crude proxy for
loss. That's part of the reason and I
can talk more offline.
Okay. So yeah,
>> you mentioned that if you are a purist,
you could stop up 50. In this case, I
was want and run it and stop it there. I
was wondering if you could see the
history of the model, take the weight at
EOC 50 and input it your model and it
will be roughly the same or it would be
certain differences.
>> You could try it. Yeah, you should just
try it because what happens is that
ultimately what we care about is how it
performs on the validation set. Right.
Here it appears to perform better on the
validation set. right? If you stop at 50
but only for the loss for accuracy
actually if you wait till the very end
it gets better. So my thrust tends to be
what is the measure that's closest to
the real world deployment.
It's accuracy. So I tend to go with
accuracy.
Binary cross entropy is a beautiful
proxy but an imperfect proxy for the
thing we actually care about in the real
world which is error rate and accuracy.
That's why I tend to plot both and if
accuracy is telling me one thing I kind
of tend to believe that
all right so um here that's what we have
so once we do all this we have a model
and we now we may to evaluate to see
okay if you actually deployed how good
is going to be so you use this thing
called the model evealuate function so
you take the modelealate function now we
use the test and the the test x and the
test y data set which we split at the
very very beginning and never used from
that point on uh we run it And when I
ran it uh last night, it came up with a
83.6% accuracy for the model. And
remember our baseline model which just
predicts everybody is a zero is going to
have a 72.6% accuracy. And this little
neural network gives you 83 83.6 which
is pretty good right so it's actually uh
few it's beating the model the baseline
model which is nice. Uh and I guess
there is something here about you know
the fact that we did a bunch of
pre-processing outside Keras and then we
send stuff into Keras. You can actually
do all this pre-processing inside Karas
automatically and there are layers for
that and I have linked to a bunch of
stuff here. So that's it as far as this
model is concerned. I know we went
through it really fast but please go
through it afterwards and make sure you
understand every single line. Change
each of these lines, rerun it, see how
the output changes. That's how we build
some intuition. Okay. All right.
computer vision
>> as I do
>> just one question and for is there a way
to build a model just to have less false
positive or less false immediate or you
don't know that
>> oh yeah yeah you can do that um but
there are so you can report on all those
things very easily but there are more
complex loss functions which will take
the the asymmetry between the false
positive false negative into account u
you know yeah so the short it's possible
yeah
All right. So, first let's just talk
about how do you represent an image
digitally. Okay. Uh and so these are how
gay grayscale images are represented.
Black and white images. So the basic
basic idea is very simple. Every picture
you have it's got a every location in
that picture is a pixel and the pixel
pixel basically has a light intensity.
The amount of light at that location and
that light level is measured from zero
no light to blinding white light which
is 255. And so all the numbers here, if
you take this five for example, you can
see a lot of no light like all the black
regions, those are all zeros. Okay? And
then wherever there is white light,
there's a number and more the amount of
light, the closer it gets to 255. Okay?
In fact, if you just step back and
squint at this, you can actually see the
five.
Okay? So that's it. That's how that's
how black and white image represented.
Very simple. Okay. Now, yeah.
microphone
>> just when you say amount of light what's
the unit that's being measured like what
do you mean
>> so here basically what we have is uh the
the computer takes whatever so when you
send an analog you take an analog
picture there is an there's a process by
which you take that analog picture and
read it in and it gets mapped to a scale
between 0 and 255 that's it that's all
so you can think of it as like a
relative scale a normalized scale
between 0 and 255 and so um it just
roughly maps to amount of light in that
location the exact like lumens to the
number mapping I don't know how they do
it my guess is there are a dis number of
variations on that but the for our
purposes just think of it as it's a
normalized scale which runs from 0 to
255
all right so uh if you look at u so
that's what's happening every is a
number between 0 to 55 boom boom okay so
if you have a color image each pixel of
a colored image is represented by three
numbers uh And these numbers measure the
intensity of red light, blue light and
green light because red, blue and green
if you mix them in the right proportion
you can get whatever you want. Okay. So
uh and so each light density is still a
number between 0 and 55 and that's what
you have. Which means that now you have
three tables of numbers instead of one
table of numbers. And by the way just
some lingo here uh in the deep learning
world these uh colors RGB, red, blue,
green are sometimes referred to as
channels. Okay. All right. So this is
what we have here. This is a picture of
Kian Cord U and then if you take that
little thing here red the red table the
green table and the blue table. So for
this picture these three tables is a
tensor of rank what?
Good.
All right. Any questions on this?
So the key task in computer vision
obviously the the important thing is
image classification right uh the most
basic task if you will uh when you're
working with images is you you have an
image and you want to take whatever you
take the image and figure out okay you
have a list of possible objects the
image could contain and you're figuring
out okay which of these possible objects
exists in that image right the doc cat
classification is like the the canonical
example right that we all know and love
uh and that's what we will solve uh
later today and on Wednesday but there
are many other tasks that you need to to
be aware of. So when you actually not
just classify an image, but you also
localize where in the image is it,
right? It's not just enough to say
sheep, you want to figure out where is
the sheep, right? And that's called
localization. And the way you do
localization is you put this little box
around it. And then you output not just
whether it's a, you know, sheep, yes or
no, but the coordinates of this box, the
top left, uh, and the bottom right, for
example, if you put the coordinates, you
can actually draw a box around it. So
you you output the numbers the
coordinates of where this box is in the
picture. Okay, this called localization.
Now this is object detection where you
may have lots of objects going on and
you want to pick up every one of them
and you want to localize it.
Okay, this is object detection. So here
we have gone in there and said okay
sheep one, sheep two, sheep three and
each of these sheep has a little box
around it. Okay.
>> By the way, u you know, self-driving
cars, the the camera vision system is
constantly scanning what's coming in
through the cameras and doing object
detection constantly, many times a
second,
>> right?
>> Pedestrian box, you know, zero crossing
box, doggy box, stroller box, and so on
and so forth.
And then we have this thing called
semantic segmentation where we take
every pixel in the picture and classify
every pixel. We are not classifying the
whole picture, we're classifying every
pixel. So we are saying okay all these
gray pixels road all these pixels are
sheep and all these pixels are grass
every pixel is being classified.
So we are taking a an image instead of
giving one classification for every
pixel we are solving a multiclass
classification problem.
Okay, every pixel is classified. And
just when you think it can't get more
complicated than this,
we have something called instance
segmentation where not only are we
classifying every pixel, we are
distinguishing between the different
sheep.
So every pixel is classified and
different instances of the same category
need to be identified.
Okay. So these are all some of the most
sort of uh I would say popular most
prevalently and useful most prevalent
and useful categories of image
processing problems that are aminable to
a deep learning system.
All right. So let's go to image
classification and we're going to work
with this application called fashion
emnest. Um
so the idea here is that you have 70
70,000 images of clothing items across
10 categories. you know like boots and
sweaters and t-shirts and you get the
idea 10 categories of clothing. Um we we
have 70,000 images like this u and then
we'll build a network from scratch to
classify all these things uh you know
with pretty high accuracy. So these
classes by the way you know this is a
very balanced data set. So 10% of the
data is you know sweaters 10% is boots
and so on and so forth. So a naive
baseline model would give you what
accuracy
10%. Exactly. So we need to build
something that's better than 10% and I'm
glad to report that a simple neural
network can actually get you close to
90%.
Right? So so this is the simple network
that we have. The input in this case is
a 28x 28 picture.
It's a 28x 28 picture. Uh and
so far we have been feeding vectors into
our neural network. Now we have a
picture which is 28 by 28. It's a tens
set of rank two, right? It's a table of
numbers. What do we do? How do we feed
that in?
It's a temp. No, each image is a table
of numbers. Let's just take a single
image.
Like what do we do? How do we what do we
do with this table?
Convert it into a vector. Exactly. And
that's called flattening. So we take
this table of numbers and we flatten it
into a vector. And so so what we do is
uh let me just
Okay. So we have um
28 by 28.
So what we can do is we can take each
row right take this row and then write
it like that.
We take the second row oops
write it like that.
third row is here
like that. You get the idea. So you take
each row just rotate it and stack it all
up, right? And string them up. It
becomes one long vector. So this called
flattening. Okay? So that's how you take
this thing and make it into one long
vector.
So when you do that 28 by 28 is what is
it? 7
784. So we get 7. So we get a vector.
This is the flattened input and you get
784.
Uh it's a vector that's 784 long.
Okay. After the flattening, we have not
done anything complicated yet. We have
literally taken the numbers and just
reorganized them in a different way.
Okay. And once we do that, now we are
back in our familiar neural network
territory, right? We know how to work
with vectors. So, we just need to pass
it through a hidden layer, right? And
this hidden layer, we're going to use re
neurons. And I tried a few different
values. And it turns out that 256
neurons does a really good job.
Okay? And so, I'm going to use 256
neurons here. And then we need to now
think about what the output layer should
be. Now, the now we run into a problem
because the output layer before we saw
for the heart disease example, it's just
zero or one. Right? Here there are 10
possible outputs. It could be a you know
boot, a sweater, a shirt and so on so
forth. 10 possible categories. So we
need some way to handle something with
many more than you know one binary
output many possible outputs. So the way
we do that
this is by the way pay attention to this
because this is actually how GPD4 works.
Okay. So what we do is here's what we
have. We know how to output 10 numbers,
right? If you want to output 10 numbers,
no problem. We just, you know, we have,
we can easily output 10 numbers by just
using a linear activation. We also know
how to output 10 probabilities,
right? Each one just needs to be a
sigmoid. But here we can't use 10
sigmoids as the output. Why is that?
Why can't we use 10 sigmoids?
>> Because the probability to one,
>> right? So here when the output comes we
need to figure out okay is it a boot, a
sweater, a shirt and so on and so forth.
There's only one right answer. Okay,
which means that we need to actually
figure out which of these 10 is the
right answer which means that we need to
produce probabilities but they have to
add up to one because only one of them
can be true.
So that's the key thing. They have to
add up to one. That's the wrinkle. If
not for that we can just use 10
sigmoids, right? And the way we do that
is something using something called the
softmax function or the softmax layer.
And the idea is actually very simple. We
have these 10 outputs in the very final
layer which is just linear activations.
And then we take each one of these
numbers and then run it through the
exponential function and then divide by
the total. So when you do that two
things happen. The first one is when you
take these numbers and run it through
say you take a1 and do e raised to a1
you now get a positive number
and now you have a positive number
divide by the sum of a bunch of positive
numbers and they're all you can see here
you can confirm visually that they will
add up to one because you're literally
divide taking each number dividing by
the total so they will add up to one
there's no other option right so this is
called the softmax function which means
that you can take any set of 10 numbers
that's coming out of the network and
convert them into probabilities that add
up to one
and So, by the way, the GPD4 reference
when you actually put a prompt in GPD4
and it starts giving you the output.
Every word it's emitting, right? It's
actually a token, but we'll get to that
later. You imagine it's a word. Every
word it's emitting u is actually it's
doing a 50 52,000 way softmax.
Think of it as every word in the
language is a possible output. So it's a
vector which is 52,000 long but it's
actually a softmax and it just picks the
most probable word and emits that. So
this notion of a softmax is actually
very powerful.
Okay but we'll come back to that uh
later. So, so to summarize, if you have
a single number, you can use a s simple
output layer, a single probability, a
sigmoid, you have lots of numbers, just
have a stack of these things. And when
you have a lot of numbers that have to
add up to one, that have to be
probabilities, use softmax,
>> right? So uh yeah
>> why do we choose probabilities instead
of just number
one
>> sorry
>> then we know it's only going to be one
>> because you can't force the network to
give you ones or zeros
it's going to produce what it's going to
produce
>> you can't force it to be exactly one or
zero
it'll give you some number you can do is
to tame that number so that it comes
into a range that you like like between
zero and
So here very quickly um we have a b when
we have a binary classification example
like yes or no this is the one hot
encoded version one or zero this is what
we saw in the heart disease example when
you have something like this example
fashion mn list where you have all these
different possibilities then you can
encode it in one of two ways you can
encode it just using integers like 0 to
9 right this is called the sparse
encoded version or you can do a one hot
encoded version of the output right you
can have a one hot encoded version of
the output and depending on how your
data comes in to you comes into your
collab right just pay attention to this
and depending on what it is you have to
pick the right keras loss function so
data comes like a one zero thing which
is exactly what we had in the how this
example we use binary cross entropy if
your data comes in this form where it's
sparse encoded you use sparse
categorical cross entropy and then if it
comes in this form form you use
categorical cross entropy, right? These
are all equalent things. It just depends
on the data that you get how it happens
to be encoded by the people who sent it
to you. If they send it this way, use
this loss function. If you send that
way, use that loss function.
Now, as it turns out in our example
here, the data is actually coming in in
this form. So, we'll use this thing
called the sparse categorical cross
entropy. And categorical cross entropy
is a generalization of binary cross
entropy which I'm not going to get into
the mathematical details but the in the
the intuition is basically roughly the
same.
Okay so this is what we have. Um if this
is your output layer use mean squared
error. If this is your output layer use
binary cross entropy and if you still
have a stack of these numbers you can
still use mean squed error. And if your
output is a soft max, use categorical
cross entropy or sparse categorical
cross entropy.
Okay. So let's actually run this in
collab. Um
right. So this is what we have. Can
folks see this? Okay. All right. So this
is the data set we saw earlier. Uh down
here as usual, right? We have we load
tensorflow and kas. We load our usual
three packages and then we set the
random seed for reproducibility. And it
turns out that the fashion mnest data is
actually available in keras. You don't
have to go find it somewhere and bring
it in. It's actually available in kas.
It's one of the standard data sets. We
luck out. So we just actually load the
data right using this load data command.
And then you do that and conveniently
for us keras has not only made the data
available it has already split it into a
training and test set. So we don't have
to do the splitting. Okay. And the
reason they do that, why would they do
that?
They do that so that different people
who are building algorithms for that
particular data set can all be evaluated
using the same test set.
Otherwise, if I split it one way and
say, "Hey, look how well I did that like
I don't know how did you split it."
>> That's the reason.
>> Okay. So here and you can see here that
uh we have
the input data is a tensor of rank
three. The first and basically another
way to think about a tensor of rank
three is just a list of rank two
tensors. Right? So here you have 60,000
images. 60,000 images and each image is
a 28x 28 square of numbers. Each image
is a 28 x 28 table. Uh and then of
course the output uh is just what
category it is a number between 0 and 9.
So you just have 60,000 numbers. It's
just a vector of 60,000 numbers. Okay.
Uh so there are 60,000 in the training
set. Oops. Uh and then there are 10,000
in the test set. Same structure 28 by
28. Uh that's what we have. So if you
look at the first 10 rows of the
dependent variable Y, you get these
numbers 9 0 33 like that. There are
numbers from 0 to 9. So if you look at
the fashion mnest GitHub site, this is
what it refers to. Zero is a t-shirt,
one is a trouser, and so on and so
forth. And nine is an ankle boot.
All right. So, uh, whenever I'm working
with multiclass lab classification
problems, I always, you know, do a
little thing here to help me figure out
that nine corresponds to an ankle boot
and so on and so forth. It just makes it
a little easier to to work with this
stuff. So, I create this little list. Um
and then uh turns out if you okay what
is the very first data point? What is
it? What is its y- value? Turns out to
be an ankle boot. Um so you can actually
look at the raw data for that image
which is just a 28x 28 thing and these
are the numbers you have.
See all these 250 233 lots of zeros and
so on and so forth. So you can actually
look at the first visualize the first 25
images. I have a little bit of code here
which visualizes that just matt plot lip
code and you can see these are all the
images they're kind of smallalish this
my friends is an ankle boot
right it's like okay can the network
really make any sense out of this thing
right it looks very blurry and I don't
know
this is uh
oh this is actually a better ankle boot
look at that okay sorry I'm getting
distracted so so this is what we have
here
uh okay we are at 955
I'm going to stop um so you folks are
not late for your next class. So we'll
continue this journey on Wednesday and
then we'll go on to color images the
next class as well. Thank you folks.
Have a good one.