Right folks, good morning.
Welcome back. I hope you all had a nice
weekend.
Uh, and I hope you had a chance to watch
the the video walk-through I posted
yesterday. Um, it's going to save us
some time today. So, let's get right in.
Today is going to be super packed. Um,
you're going to go from not knowing
anything about convolutions perhaps for
some of you to actually knowing how
convolution networks work and actually
to build one and demo it in class, okay?
And uh, this demo has actually worked
pretty well for the last few years that
I've taught the class, but you never
know because it's a live demo, it may
not work. We'll see.
Um,
Valentine's Day gods, maybe they maybe
be with us.
Okay, so let's get going. So, Fashion
MNIST we saw previously, um, i.e. as in,
you know, in the in the walk-through,
the video walk-through, that a neural
network with a single single hidden
layer can get us to some an accuracy in
the the high 80s, okay? Uh, and that
thing that network actually didn't know
what was coming in was an image, right?
It literally took this table of numbers
and just took each row and then
concatenated all the rows into one giant
long vector and then sent it in. So, the
neural network did exploit the fact that
the input data was sort of known to be
of a certain type, okay? Which is the
clue for how can we do better?
Right? So, let's just spend a few
minutes on why what is it about images
that we have to really pay attention to,
okay? As opposed to any arbitrary vector
of numbers that's coming in.
Okay? So, when we flatten the image into
a long vector and feed it into a dense
layer,
several undesirable things can actually
happen.
What are some of them? Any any guesses?
Uh, yeah.
I think you lose the proximity of one
pixel to other ones that would be around
it.
Right. So, if you take a particular
pixel, then let's say that the picture
shows a t-shirt, um, if there's a little
pixel at in the center of the t-shirt,
knowing that the surrounding pixels are
related to the pixel in a way because
they are all part of this concept called
a t-shirt, would certainly be helpful,
right? So, so to put it more
technically, spatial adjacency
information is very important. And we
need to somehow take that into account.
Okay? Um, all right. What else? What
else might be going on here?
Uh,
Yeah,
you have some metadata about it like the
relative match into the resolution
Oh, I see. So, if you actually had
structured data about the image such as,
you know, various characters about that
might be helpful. True. Now, but let's
just focus on the case where you only
have the raw image and nothing else.
And under that constraint, what else
might go wrong?
Or what else might be suboptimal?
Okay. Well, the first thing that might
happen is that
we have we may have too many parameters.
So, let's take So, this is, you know,
this these numbers are from my, you
know, older iPhone. Uh, I noticed that
when I take a color picture with my
phone, it's a 3,000 * 3,000 roughly uh,
grid, right? So, the picture is actually
3,024 pixels on this axis, 3,024 on that
axis, okay? So, that gets us to roughly
9 million pixels, but remember there's a
color picture, which means there are
three channels,
which means there are 27 million
numbers,
each of which is between 0 and 255 from
that little picture, okay? And now let's
say we connect it to a single
100 neuron dense layer.
A single 100 neuron dense layer. How
many parameters are we going to have?
Just in that one little part of the
network.
Could the mumbling be louder?
Yes, roughly 2.7 billion because 27
million parameters times 100,
right? Roughly, of course. Forget about
the biases for a moment, right? It's 2.7
billion.
2.7 billion parameters,
right? Do you think we can actually get
2.7 billion images to train any of these
things?
So, then you're going to overfit.
Right? Too many parameters. We have to
do We have to be smarter about this.
It's not going to work.
Right? That's the first problem.
The So, this clearly is computationally
demanding, very data hungry, and
increase the risk of overfitting.
Okay?
Next,
we lose spatial adjacency.
Right? We literally are ignoring what's
nearby.
So, that's a huge huge factor. There's a
third factor,
right? That we have to worry about,
which is that
let's say that, you know, the picture
has a vertical line
on the on the top left side and it has
some other vertical line on the bottom
right side.
What this sort of dumb approach is going
to do
is going to it's going to learn to
detect that vertical line on the top
left and it's going to independent of
that, it's going to learn to detect the
vertical line on the bottom right.
Okay? Which doesn't make any sense. What
do you A vertical line is a vertical
line. So, you want to be able to detect
it wherever it happens.
Detect once, reuse everywhere.
That's what you need to do.
So, this, by the way, is called
translation invariance.
Translation is math speak for move stuff
around.
Right? You take a line and it moves
around,
it doesn't matter, it's still a line.
Let's Let's Let's figure it out.
So, these are the the three things we
need to worry about. So, we want to
learn once and use all over the place.
We want to take spatial adjacency into
account, number two. And number three,
let's just find a way to make sure that
we don't have billions of parameters for
simple toy problems.
Any questions?
Yep.
Um, is this a problem
just because we are compressing the
image or would it have happened anyway?
It would have happened So, the question
was is it a problem because we are
compressing the image uh, or would it
would it have happened anyway? The
answer is it would have happened anyway.
You can take any picture, this is going
to happen, right? Because I'm not making
any assumptions about how the image is
coming in to me,
whether it's compressed or not and so on
and so forth.
Okay. All right.
So, convolutional layers
were developed to precisely address
these shortcomings and they're amazing
solution, as you will see. Very elegant.
All right.
So, the next, I don't know, half an hour
is going to be me defining a whole bunch
of stuff
before we actually get to the fun
collabs and so on and so forth.
Um, so just to put in perspective, I I
have a PowerPoint,
two collabs,
and an Excel spreadsheet, and maybe even
a notability file to cover today.
Okay? So, but hang on for the next 30
minutes because it's going to be a
little concept heavy
before we get to the fun stuff. So, stop
me, ask me questions because we do have
time.
All right. A convolutional layer is made
up of something called a convolutional
filter.
Okay? That's the atomic building block.
A convolutional filter is a nothing but
a small matrix of numbers like this.
It's just a small square matrix of
numbers. That's a convolutional filter,
okay? Now,
a layer is just composed of one or more
of these filters.
All right?
Filters and layers.
Now,
the thing about the convolutional filter
that makes it really magical
is that if you choose the numbers in a
filter carefully
and then you apply the filter to an
image, and I'll get to what I mean by
applying the filter,
if you choose the numbers carefully and
you apply to that image,
this little humble thing has the ability
to detect features in your image.
It can detect lines, curves, gradations
in color, circles, things like that,
okay? It's pretty cool.
And so,
I'm going to claim and I'm going to
prove shortly that this little humble
filter with the ones and zeros, it can
detect horizontal lines in any picture
you give it.
Okay?
This thing here is going to has the
ability to detect vertical lines.
All right? So, I will demonstrate how
this thing actually detects all these
things and then we will ask the big
question that's probably in your minds
already, where are we going to get these
numbers from?
That all sounds great, Rama. Where are
we going to get the numbers from? Okay?
And we have a beautiful answer to that
question.
All right. So, let's go. Um, now I'm
going to first explain to you what I
mean by applying a filter to an image
and then I'm going to give you examples
of how the filter works for detecting
vertical and horizontal lines. So, all
right. So, let's say that this is the
image we have.
Okay? Again, an image. Assume it's a
grayscale image. So, you just have a
bunch of numbers between 0 and 255,
okay? So, that's that This is the image
we have. It's a little tiny image.
And this is the filter that's been
magically given to us by somebody.
And what we are trying to do now is to
apply it, okay? So, what we do is that
we literally take this filter,
the little one, and then we superimpose
it on the top left part of the image.
So, you have the image here, you take
this little filter, and then you move it
to the top left so that they are sort of
right on top of each other.
Okay?
Once you have it right on top of each
other,
you have these matching numbers. You
have three numbers in the image, there
are three numbers in the filter, and
they're all matching each other right on
top of each other, right? So, you have
nine pairs of numbers.
And then what we do, once we overlay it,
we literally just multiply all the
matching numbers and add them up.
Okay? You just multiply all the numbers
and match them up, and you can confirm
later on that you know the the
arithmetic I'm doing is actually
accurate. Okay?
And once you do that you'll go get some
number.
Right?
Um
once you get that number
what we do is we go to our good old
friend the relu
and then we just run it through a relu.
Now, in this case all that effort comes
to nothing because it's zero. It's okay.
Okay? So, zero and this number becomes
the top left cell of your output.
So, this is called the convolution
operation.
Okay?
And we won't get into why it's called
that and so on and so forth. There's a
long and rich and storied history of
these things.
But this is the convolution operation.
And once we do that you sort of can now
predict what's going to happen, right?
We take the same exact operation and we
just move it to the right.
We move this little 3 by 3 thing to the
right and repeat the exact same process.
Matching numbers
uh to you know multiply all of the all
the matching numbers together, add them
up, run them through a relu.
Okay?
And then boom, you get the you get the
second number here.
And you keep doing that till you reach
the very end. You fill up all these
numbers then when you then you come to
the top of the second row.
Okay?
And you keep on doing that till you
reach the very bottom.
So, this is what I mean when I say apply
a filter to an image.
Okay?
Any questions?
Okay.
Microphone, please.
Microphone.
What happens when
the heart of the
and you stop
the remaining
but the filter doesn't perfectly match
Yeah, so you start from the left and
then you keep on going. At some point
the right edge of the filter is going to
match the right edge of the image and
then you stop.
Yeah. Now, there are some nuances here.
So, for example, you can actually pad
the whole image
on its borders so that you can actually
go outside the image and it'll still
work.
Okay? Number one. Number two, nuance.
Instead of just moving one step to the
right every time you finish, you can
move two steps to the right.
Right? And that's something called a
stride. Okay? So, there are a bunch of
pesky details here. But I'm just
ignoring them because this basic default
approach works well amazingly well
almost all the time.
Okay? All right. So, that's that's
that's the mechanics of how this
operation works. Um all right. Now, I'm
going to switch to a spreadsheet which
shows this really beautifully
courtesy of the fast.ai people.
All right. So, what I'm going to do here
because the big spreadsheet I'll upload
the spreadsheet after class so you can
see it. So, all I have done here, rather
all they have done here
thanks to them, is that they have
essentially created a table of numbers
in Excel as you can tell.
And they have just put some numbers.
Most of the numbers are zero. But these
some of these numbers are all more than
zero. They're like 0.8, 0.9 and so on.
Basically, all they have done is instead
of working with numbers between zero and
255, they're just dividing all the
numbers by 255 so you get fractions and
they just put the fractions in the
table. Okay? And then then they have
used Excel's very cool conditional
formatting
to essentially mark in red all the
values that are high. Right? If the
number is closer to one, the more
reddish it gets.
Okay? And when you do that the three
obviously pops out.
So, there is a three in the image. Yes?
Okay, good. So, now
what we're going to do is we're going to
move to our little filter here.
You can see the filter.
Right? And I'm claiming this detects
horizontal lines. And so and this table
here
Sorry.
This table here is the result of
applying that filter to the three.
Okay? And you can see here I'm looking
at the top left cell here.
Um
This is
Look at this top left cell. The formula
is nothing more than
you know, multiply all those things and
add them up. And then once you add it
up, run it through a max of zero comma
that which is just the relu.
Okay? Basic arithmetic.
So, we do that.
And this is the output and the output is
also conditionally formatted to show you
where things are lighting up.
And you can see only the horizontal
lines of the three are lighting up.
Everyone see that?
Right?
So, you So, now you you understand the
filter in fact is living up to the claim
I made for it.
Right? Similarly,
if you look at what's going on here,
this is a vertical filter, the same
thing, you apply it, only the vertical
line is lighting up.
Right? Now, what you can do is
uh I would encourage you to do this, you
know, um after class, is you can look at
all these numbers here, for example, and
then ask yourself, "Okay, why is that
lighting up?"
Right? And you will discover that what's
actually going on is that it's looking
for edges.
It's looking for you know, s- you're
looking for rows in the table where
there is some nonzero thing in the first
row and zeros in the second row.
And by choosing the numbers carefully,
you multiply the ones with positive
numbers and you multiply the zeros with
zeros and then you'll come up with a
positive number and thereby you detect
an edge.
Right? So, what I would encourage you to
do is use the this Excel thing here.
All right. So, here is here is a cell we
have. So, let's uh trace its
coincidence.
Okay.
So, you can see here
these numbers
Right? Th- This is what it's processing.
Right? That is this grid is being
processed to come up with that big
number. And you can see here in this
grid these are all these numbers are
here and then these numbers are a lot
lower than these these numbers because
there is an edge.
Right? The numbers are a lot lower.
That's why you can see the horizontal
part of the three.
And so, what this filter is doing, it's
basically saying, "Well, the stuff
the row that I'm catching here has the
ones, the middle has zeros, the rest are
all minus ones."
Right? So, the small values are going to
get very small.
The big values are going to get very big
and the overall thing is going to be
emphasized.
So, that's the basic idea of edge
detection.
Spend some time with it with the Excel
and it'll you'll become clear to you
what I'm talking about here.
All right, cool. So, that's that.
All right. Uh by the way, I also have uh
th- there is a little very cool site
here
in which you can actually go in and
punch in your own numbers and see what
it detects.
Right? Lot of edges and curves and this
and that. It's very cool. So, I
encourage you to try it out.
So, the key thing here I want to say is
by choosing the numbers in a filter
carefully and applying this operation
different different features can be
detected. All right.
Now,
I mentioned earlier that a convolution
layer is composed of one or more of
these filters. So, one or more of these
filters. And so, you can think of each
filter as a sort of a specialist for a
particular feature.
Okay? So, it's a specialist. Maybe it it
specializes in detecting vertical lines,
horizontal lines, you know, uh
semicircles, quarter circles, you don't
know. Right? You can imagine either them
as being specialists.
And given that modern images could be
very complicated, they may have lots of
interesting features going on, you
probably want to have lots of these
filters.
Okay? But the key the key is that you
don't have to decide up front, "Hey, you
filter, you better specialize in
detecting vertical lines and you on the
other hand do not stay in your lane. Do
vertical lines." Right? You're not going
to do that.
You will let the system figure out what
it wants to figure out.
Okay? So, there is no human bottleneck
in doing this.
And I mentioned this because there used
to be a human bottleneck, you know,
before deep learning happened.
And so,
Now, let's just um make sure we
understand the mechanics of what happens
when you have two of these filters, not
one. So, this is the input image as
before. This is the filter we saw
earlier and this is another filter we
have.
The thing is we just run them in
parallel. We take each filter, do the
operation, come up with an output. Take
the other filter, do the operation, come
up with its output. And then when you do
that, the first one gives you that, the
second one gives you that. And this
output is a table of some it's it's a
it's a it's actually not a table. What
is it?
Louder, please.
It's a tensor. Thank you. It's a tensor.
And so, these two 5 by 5 matrices can be
represented as a tensor of what shape?
And there are two right answers.
5 by 5
into two, correct. So, it can you can
either think of it as 5 by 5 * 2 or 2 *
5 by 5. They're both fine.
Which one you go with is actually ends
up being a matter of convention.
Okay? So, now you begin to see why we
care about tensors.
Imagine if instead of having two
filters, we have 103 filters.
The resulting tensor is going to be 5 by
5 by 103.
Okay.
Good.
Um all right. Now,
let's now look at the slightly more
complex situation where you have not a
black and white image, a grayscale image
with just a little table, but an actual
color image.
Okay? So, So, we know how to apply a
filter to a 2D tensor like this and to
get that. But let's say we have
something like this where it has
three, right? It's got three channels,
red, blue, green, RGB. It's got three
tables of numbers.
So, this is a a tensor of shape 6 * 6 *
3, let's say, and you want to apply this
3 by 3 filter just like before to this
thing. You want to apply the convolution
operation. How's that going to work?
Do we just like apply this to each
We first apply it to the red, then we
apply it to the to the green, then we
apply to the blue. Should we do that?
Or is there a
a problem with that approach?
Yeah.
Could you use the microphone, please?
Uh the problem with the approach, I
think, would be the same as what you
said earlier, that it would learn the
lines probably the same each channel,
right?
Like the location of the lines are
probably the same each channel.
Yes, the location of the line is going
to be the same thing because that line,
if you will, is sort of the the
aggregation of information from the
three different channels. Right. But the
problem here
is sort of slightly different,
which is that
If you do them independently,
the network has not been informed that
these things are all part of the same
underlying concept.
As far as it's concerned, it's just like
three things. It's just going to process
them independently. So, we need to
somehow change the filter so that it
understands like what is at this pixel
location, the three numbers under it,
RGB, they're actually the same part of
the same thing, underlying thing.
So, what we do is actually very simple.
We just take this filter and make it 3D.
So, we take this filter, instead of
having just one of them, we just make it
a cube like that. Three times.
And once we do that, you can imagine
taking this thing here and essentially
doing that.
Okay. Now, instead of having, you know,
nine numbers in the image and nine
numbers in the filter,
you have 27 numbers in the image, 27
numbers in the filter.
But you still match them up, multiply
them, add them up, run them through a
ReLU.
By the way, I tried to get ChatGPT to
give me a picture like that.
It just completely bombed.
I like three, four, five different
variations. It just gave up. And then I
found this nice picture at in the
deeplearning.ai and I used it.
So, then if you put different numbers in
each of the layers, is that like color
processing? Like it could be doing a
different thing to green and blue. I'm
sorry, say that again. If you put
different numbers in each of the layers
of your knowledge, in each of the
different like depth dimensions of your
convolution filter, would that be like
color processing?
Uh yeah, you you will in
Yeah, you will put different numbers. In
fact, you you have 27 numbers now,
but we haven't gotten to the question of
where these numbers are coming from. So,
just hold the thought till we get there.
Okay. Um so, any questions on this?
Okay. You literally take the 2D thing
and make it 3D.
You basically give it depth and the
depth just matches the depth of the
input.
So, if the input is like, you know, 10
deep, your filter is going to get 10
deep.
Okay?
Yes.
Rather than
increasing the rank order of the tensor
by one, is there any instance where you
would create a subtraction layer where
you would run an operation across the
different layers to come up with a
intermediary layer that you would run a
lower rank tensor of a filter over?
Yeah, so there is a lot of stuff in the
research literature which tries to do
things like that. Uh I'm just describing
like the the the most basic approach to
doing this. And as it turns out, this
basic approach is actually extremely
powerful, right? And of course, uh
researchers try to, you know, go from
the 95th percent thing to the 95.1%.
So, they invent like all sorts of crazy
complicated stuff, which is all good for
us, humanity, but for practical use,
this is good enough.
How do you convert the 3 by 3 layer into
a single 4 by 4 layer? 4 by 4 is
understood, but what about the 3 layers?
How do they work?
Yeah. Um so, we are coming to that. I
think we have a slide here. Actually, we
don't. Never mind. We'll answer that. Um
so, so here you have one filter, right?
You have one 3 by 3 by 3 filter, which
plugs into this thing here, and then it
gives you the 4 by 4 at the end.
Right? So, for one filter, we know that
by doing this operation, we get
we get this 4 by 4.
Let's say that you have another filter,
which is also 3D.
You do that thing, you'll get another 4
by 4.
And if you have 10 filters, you'll get
10 of these 4 by 4s, which then gets
packaged up into a 4 by 4 by 10 tensor.
Remember, whether it's 2D, 3D, 10D,
what is coming out is always 2D.
Because ultimately, when you apply all
this operation, at each position, you
just have one number.
And then ultimately, you just do all
those things, you just come up with a
table of numbers always. So, the what's
coming out is always a 2D number table
like that.
But when you have lots of filters, you
have lots of these 2D tables one after
the other, and there therefore, they get
packaged up into a tensor.
All right.
Um so,
textbook chapter 8.1 has a lot of detail
and intuition, which I think is really
good. So, please uh try it out. Okay.
And folks, by the way, this convolution
stuff, um it's sort of it grows in the
telling. So, I would encourage you to
revisit it, revisit it
a few times, and then it slowly becomes
part of your muscle memory.
Don't expect to just understand all the
nuances like one shot.
Do it a few times.
And it will become, you know, wired into
your into your head.
Okay. So, all right. The big question.
These seem excellent, but how are we
supposed to come up with these numbers?
Now, in fact, traditionally,
uh these filters actually used to be
designed by hand.
Uh computer vision researchers would
invest, you know, prodigious amounts of
time and effort and talent to figure
out, you know, the kind the right kinds
of filters to use for various specific
applications. So, if you wanted to build
an application which would look at, say,
MRI images and figure out, okay, what
kind of features should I extract from
this MRI thing to be able to say, you
know, predict the the evidence for a
stroke, they would actually, you know,
hand design the filter. They'd try lots
of different values and then come up
with, "Ah, I got the perfect filter for
this thing here." Right? So, that's the
way it used to be done.
Um and now,
I but as we figured out how to train
deep networks with lots of parameters,
right? We figured out things like ReLU
activation, stochastic gradient descent,
GPUs, backprop, things like that, you
know, uh this big idea emerged. Why
don't we think of the numbers in the
filter as just weights?
And why don't we just simply learn them
from the data using backprop?
Right? Just like we learn all the other
weights. What's the big deal?
And this simple idea,
and it feels a bit, I don't know,
blindingly obvious in hindsight.
I'm sure it was not obvious in
foresight.
Um right? This was the breakthrough.
This was the key breakthrough. And now,
it's actually possible to do this
because a convolutional filter that we
have seen is actually just a neuron.
And the underlying arithmetic of it is
just a neuronal arithmetic. And so, it
just happens to be a slightly special
one. It's actually even simpler than a
regular neuron. And in the interest of
time, I have a one or two slides in the
appendix which tells you exactly why
it's a neuron. So, check it out. But
just take my word for it. It's just a
particular kind of neuron. And because
it's a particular kind of neuron, and we
know how to work with neurons,
right? You know how to work with
neurons, which means that our entire
machinery,
layers, loss functions, gradient
descent, SGD, blah, blah, everything is
immediately applicable.
We don't have to invent any new stuff to
make it work.
Okay?
All right.
Do you initialize the layers differently
in applications or just because the
network has different sizes? Like
computer vision versus uh medical
imaging. Is it just because the network
has different numbers in them?
Yeah, so the initialization
So, let's It's a good question. Let's
come back to it when we get to something
called transfer learning, which I'm
going to get to by about 9:30.
All right. So,
that's it. All right. So, this turned
out to be a huge turning point in the
computer vision field, and this was the
massive unlock in the year 2012. This
computer vision system that used this
technology called AlexNet burst out onto
the world stage because it crushed the
competition in a, you know, in in a
competition called ImageNet, and uh the
previous best score was 26% error rate,
and this thing came in and had 16% error
rate. Right? It's the kind of thing
where if you see it, you'll be like,
"Oh, that must be a typo."
Right? Because every year, the
improvements in error rate were like
very little, half a percent, 1%, and
then this year was 10%, and that that
was because of this approach.
And so, all right. Now, one other thing
I want to cover talk about is that with
every succeeding convolutional layer,
uh this particular convolution any
particular convolutional filter, it's
basically implicitly seeing much more of
the input image as we go along.
Right? Which means that if in the very
beginning, if this is the input, right?
This little convolutional filter this
number here
in the first layer, let's say, only sees
like the top of the chimney or whatever
of this house.
But then the next layer, remember, the
next layer is input is this particular
layer.
And so,
this particular little thing here is
getting information from this whole
square here.
And every one of the points in that
square is actually something big in the
original picture.
So, with every additional layer, you're
seeing more and more and more of the
image.
All right? And this is a key part of why
these things work because you're
essentially hierarchically building a
better and better understanding of the
image.
It is the hierarchical understanding,
the hierarchical learning, that's a very
key part of the unlock.
And so, if you look at networks and what
they're visualizing, this actually a you
know, a face detection deep network
visualizes of what it's learning, you'll
see that the first layer is just
learning lines and edges and so on,
lines.
And the second layer is actually
learning edges. Look at this thing,
right?
It's it's learning to put these lines
together
to get some sort of an edge here,
another edge here. This looks like three
three quarters of a of somebody's ears.
And then, these things are now being
assembled
to get whole faces out.
Can you imagine the researchers who did
this work? They built the network, it's
doing really well on detecting faces,
and they turn around, "Okay, let's see
what it's actually doing."
And then, this picture pops up.
I mean, goosebumps.
Okay, so pooling layers, the next one.
So,
so far you've talked about convolutional
layers, this is the second thing, second
building block, and then we'll again go
go to the collapse. So, pooling layers
are also called subsampling or
downsampling layers.
So, the idea is that every time a tensor
is coming out of these convolutional um
layers,
we try to make it slightly smaller
because the act of making it smaller
will force the network to try to
summarize and learn what's going on in
this complicated thing it's coming into
it, okay? So, I will describe the
mechanics first. Um
So, let's say that this is the output of
a convolutional layer.
Okay?
Is this four of them? A 4 by 4.
So, what we do is that there are two
kinds of pooling, max pooling and
average pooling. This is called max
pooling, and the idea is really simple.
In this max pooling layer, there are no
weights parameters to be learned. It's
just a simple arithmetic operation. We
basically take
we take this we basically superimpose a
2 by 2 empty grid
on the top left, and then we say, "Hey,
what's the biggest number on the among
these four numbers?" Well, the biggest
number is 43. Boom. Okay, I'm going to
stick a 43 here.
Then I move my 2 by 2 to the right
so that it overlaps with these numbers
in blue, and I say, "Hey, what's the
biggest number here?" Okay, that's 109.
And I move it down, what's the biggest
number here? 105. Stick it in here.
Biggest number here, 35, and I stick it
in there. That's it. This is max
pooling.
Similarly, there's this thing called
average pooling, but instead of taking
the maximum of these four numbers, we
just average the four numbers.
Okay, the average of these four things
in yellow,
am I done?
Average of these four numbers is 32.2.
The average of blue numbers is 25.5, you
get the idea.
That's it. Max pooling and average
pooling. Now,
as you can see, when you go when you
apply pooling, the number of entries
drops significantly.
Right? The number of entries drops
significantly.
And the output from this layer is just
fed to the next layer as usual.
Okay? There's nothing, you know, crazy
going on.
So, it's a way to shrink the output from
one convolutional layer before it passes
on to the next convolutional, you
interject with a pooling layer.
Now, I have actually a
even if I say so myself, a very nice
handwritten explanation of what pooling
does, the the effect of pooling.
And unfortunately, I can't get my iPad
to actually show up on my laptop.
So, I'm not going to be able to do it,
but I will record a walk-through.
Yeah, and I posted check it out, okay?
But the intuition that I tried to convey
with that thing is that oh, um Sorry,
I'll come back to this.
So, max pooling acts like an or
condition. It basically says, "I have
this big picture.
So, in the four things that I'm looking
at, if there's any number which is
really high,
that means that some feature is being
detected, right?
The number is really high coming out of
a convolutional layer, that means that
something somewhere fired up,
lit up.
And so, I'm just looking to see if
anything lit up in that part. If it did,
I'm going to say, "Yep, something lit
up."
If nothing lit up, then I'm going to
say, "Oh, nothing lit up."
So, in a in that sense, what it's it it
think you can imagine it's like acting
like an or condition.
Anything fired up? Anything fired up?
Anything fired up? Anything up? Yes,
okay. Otherwise, no.
And so,
sadly, I can't switch to Notability.
So, it acts like a feature detector. So,
if you have lots of things going on in a
particular picture, you want to be able
to summarize and aggregate all the
things that are going on so that you can
say you if you may have a big picture
with lots of things lighting up here and
there, but you want to step back and
say, "You know what? In this picture,
the top left, nothing lit up. The top
right, something lit up. Bottom left,
something lit up. And the bottom right,
nothing lit up."
So, you're operating at a higher level
of abstraction.
That's the effect of pooling.
But don't you lose spatial information?
Uh you don't because the
what you're actually saying is the top
left has this thing.
You already know it is in the top left.
And you already moved up to that level
of abstraction.
So, the fact for example, if if the top
left there is a human eye,
and there is a circle detector, it's
going to fire up and saying, "Hey, in
the top left there is an eye."
Yep, lit up. So, you're not looking at
the pixels anymore, you're already
operating at a higher level of
abstraction, and that's how we get
around it. But this proceeds slowly and
incrementally, which is why you have
these big networks.
All right.
So, now as we saw, some successive
convolution layers can see more and more
of the original image,
the max pooling layers that follow them
can detect if a feature exists in more
and more of the original input as well.
So, by the time you get to like the
seventh and eighth, ninth and layers and
so on, this thing is actually really
smart. It's operating at a very high
level of abstraction.
Right? It It is You can think of it It
is basically like tagged all the
features in that image at various
resolutions, and it can work with it.
Is there a trade-off between doing
pre-processing as opposed to adding
additional convolutional layers? I'm
thinking if you have a video turning
into a black and white static images in
a sequence as opposed to
shoving in a color video with a ton of
noise.
The greater the time expanse, is there a
trade-off element? There is a trade-off.
Um if your particular data set and input
has has some there is some very
important domain knowledge that you want
to encode
into the network so that the network
doesn't waste its capacity learning
things that you know have to be true,
then yeah, modify the input.
But if you're not sure,
right? Then you want to just let network
learn whatever it can as long as it's
focused on predicting accuracy as well
as possible, then just let it be.
Uh all right. So, that's the basic idea.
And I again, I'm sorry this is
Notability thing is is it's not working.
Uh but take a look to really understand
um how this max pooling thing business
works. Okay. Oh, uh I think I skipped
over this.
So, when you have something like this,
so this, let's say, is a tensor coming
out of some convolutional layer, and its
size is 224 by 224 by 64, then you apply
something like a pooling. The thing I
want to point out is that the pooling
will work with every slice of the
tensor.
Okay? So, if the tensor is 224 by 224 by
64, it has a depth of 64,
which is basically like saying it's got
64 tables of 224 by 224, and the pooling
will work on every one of those tables.
Which means that
the 64 will that you'll still have 64
things at the very end. It's just that
every one of the things of the 64, the
224 by 224, will shrink to 112 by 112.
So, each table shrinks due to pooling,
but the number of tables does not
change.
Okay. So,
uh by the way, this
link here
has a beautiful explanation of all these
things with a little bit more complexity
as well from a course taught at Stanford
in like 2018 or 2019 or something, I
forget. Uh so, just check it out if
you're curious about this stuff. It's
really good.
Okay. Um
All right. So, that brings us to the
architecture of a basic CNN.
Um and so, what we do is we have an
input.
Okay? We take that input, we run it
through a bunch of convolutional and
pooling layers. So, there's a
convolutional layer, and then we pool
it, which is why it has shrunk
in size,
and then it goes through another
convolutional layer, then we pool it,
which is shrunk again,
and then it keeps on doing it. So, we
have a series of these these called
these are called convolutional blocks.
So, a convolutional block is typically,
you know, one to two convolutional
layers followed by a pooling layer.
Okay.
So, you have a series of convolutional
blocks.
Okay? And the thing to notice is that
as you go further and further in the
network,
the blocks will actually get smaller and
smaller because of
uh max pooling, right? They'll get
smaller and smaller, but they'll get
longer they'll get deeper and deeper.
Okay.
And we have empirically figured out that
that actually that model of reducing the
size, the height and height and the
width, but then making it deeper, tends
to work really well in practice.
And so,
in fact, uh and I apologies to the live
stream that I can't use iPad, I'm going
to do it on the the board.
So, let's say that you have a picture
which is
coming in as 224
224
and then you have
say three of them
because it's a color picture, so you
have three of them.
Can you folks see this okay?
All right. So, right? Let's say this is
the input coming in. And ResNet, which
is a very famous network that we're
actually going to work with in a few
minutes,
then it actually gets done with all this
convolution pooling business.
The final tensor that it it has is
actually of shape
7 by 7.
But it is 2048 long.
Okay? So, it it has gone it has
processed something which is 224 224 * 3
to much smaller height and width just 7
by 7, but it's gotten much deeper, 2048
layers.
This is a this is a numerical example of
what I'm talking about there in terms of
as you go along, things get smaller but
deeper.
All right.
Uh
Yes?
Is the reason that it gets deeper
because each
Like it it gets deeper because each
layer has a single feature that is
picked up and then it gets stacked on
top
It's not so much that each layer has
picking up a single feature, it's more
that
uh
basically
the way I think about it is that
the the the the number of atomic
features that you may want to detect are
probably not that many, right? Lines,
curves, gradations in color and things
like that. But the way in which you can
combine these atomic features
to depict real world things
is combinatorial.
It's sort of like I have 10 kinds of
atoms, how many molecules can I make
from it?
You can make a lot of molecules from
those 10 atoms, which means that you
better give the network more the ability
to capture more and more of these
possible things that the real world can
come up with.
And so every as the depth increases, you
have more filters and every filter has
now has the ability to pick up some
combinatorial combination of what's
coming in.
Uh sorry, quick question related to
this. So, right now like our model is
being trained to detect certain specific
features like a line, a color, or
something of this sort. But still it
doesn't have meaning to this, right?
Like still they don't know if that
arc is a sun or is an eye, right?
So, yeah. So, we we don't tell it what
to learn, it just learns.
All we tell it is make sure that you
minimize the loss function. Now, once it
is finished learning, if it's a good
network, it has good accuracy, then we
can introspect. We can peek into the
internals and try to understand what is
it learning,
right? And sometimes you like you saw in
the face detection example, it's
actually learning interesting things
like basic lines and edges and then
slowly, you know, more complicated
shapes and then finally like entire
human faces. Sometimes it may not be
understandable.
And the way it's doing this is by
constructing features of my brain.
Like how do you figure out what it's
learning?
>> Yeah. Oh, oh, I see. So, I'm going to
give a reference in just a few minutes.
Read the paper. That was one of the
first ones to actually visualize what it
what these things are learning and
that'll give you an idea of how it
actually works. And I'm also happy to
talk about it offline. It's a bit of a a
tangent, but it's a really rich tangent,
so if if I keep talking about it, I'll
end up spending 10 minutes on it, so I'm
going to back off.
Okay.
Um all right.
So, now once we do that,
okay? Now we are back in familiar
territory where we take whatever tensor
is coming out from these convolutional
operations and pooling operations and
then we just flatten them only now into
a long vector. And once we flatten them,
we can connect them to some good old
dense layers
like we know how to do and then we
finally connect them with whatever, you
know, output layer you want, right? In
this case, this example is using some
multi-class classification of
classifying images to what kind of
automobile or whatever it is. So, it's
like a softmax. So, this is a general
framework.
Okay?
Any questions?
Yeah.
Can you explain again how the depth
increases exactly like Oh, the depth
increases because you decide what the
depth is.
So, when you add a convolutional layer,
you decide how many filters it has. So,
you just keep adding more and more
filters the later on you go in the
network.
So, it's in your control. So, remember
the number of neurons in a hidden layer
is in your control, right? Similarly,
the number of filters is in your
control. It's a design choice.
And we design it so that the later we
go, the more depth we have. So, you have
you stack
um layers with each of those layers has
a different filter applied to the end
Yeah, a layer is made up of filters and
so the depth just comes from having lots
and lots and lots of filters. And you
get to choose what they are.
All right. So, now let's go to the
fashion MNIST collab um that I did the
video walk-through on and then actually
solve it using a convolutional network.
All right, cool. So, uh at this point
I'm going to zip through some of the
stuff because you know the preliminaries
have to be done. Import all these
packages, set the random seed here.
Great. And then the we will load the
MNIST data set just like I did in the
collab yesterday. Uh we create these
little labels.
Uh and then we just have these standard
functions to plot accuracy and loss that
we've been using so far. All right. Now
we come to the convolutional thing and
so as before, we're going to um
we're going to divide it by 255 to
normalize everything to a zero to one
range. Uh let's confirm to make sure
that the data nothing has gotten
tampered with. Yep, we have 60,000
images, each one is 28 by 28 in the
training set. Now,
convolutional networks um they expect
the input to have
three channels or it expects to have
like a an additional thing which is like
a channel,
right? Uh the color images have three
channels,
but black and white images have only one
channel, right? One table of numbers.
So, instead of saying 28 by 28, we tell
this the convolutional layer expect 28
by 28 by one.
It's the same thing conceptually, but
that's the sort of the format that it
expects.
And so,
uh we go here and then we say, all
right, there's a thing called expand
dimension. I'm just telling it to expand
its dimension and once I do that, you
can see here it's still 60,000, but
instead of 28 by 28, it has become 28 by
28 by one. Same thing.
Okay? Now, let's define our very first
CNN.
So, all right.
As as before, the the input is just
Keras.input as before, no difference
here and we tell it the shape and the
shape is of course just 28 by 28 by one.
Okay? That's what I have here.
And then we come to the first
convolutional block.
So, and this is the key thing.
If you want to tell Keras to use a
convolutional a layer,
you use this keyword layers.Conv2D.
And from this you can probably also
figure out that there's a Conv1D and
there's a Conv3D and so on and so forth,
which, you know, uh explore. It's really
good stuff.
But for image processing, Conv2D is all
you need. And now we tell it how many
filters you want. Okay. So, uh we decide
on the number of filters. So, I've
decided to have 32 filters. Okay? And
then I I we also have to decide the size
of the filter, right? The simplest size
is 2 by 2. So, I'm just going to go with
that.
Right? Kernel size is 2 by 2.
And then the activation is of course
ReLU. I give it a name, convolution one,
and then I feed it the input. And then
once I do that, I follow it up with a
little pooling layer which I where I use
MaxPooling2D.
And MaxPooling2D, you just literally
pass the input, you get the output back.
It just
shrinks everything using pooling.
So, that is the first convolutional
block.
And you know what?
I know how to cut and paste. Boom, cut
and paste, I get the second
convolutional block.
Okay? Here is the second convolutional
block. And I know in in I just lecture I
mentioned that as you go deeper, you get
more depth to it, but this is this is
just a starting point. I'm just going to
use the same depth. Not a big deal. It's
a simple problem. So, which is why in
the second convolutional block I'm still
using only 32.
But you can totally go to 64 for
instance to make it much deeper.
Okay?
Uh and once I do that,
I finally come to the point where I
flatten everything to a long vector,
then I connect it to one dense layer of
256 neurons.
And then finally, I come to the softmax
where I have 10 outputs, right? 10
categories of clothing, softmax, and
then I tell Keras, okay, take this input
and the output, string them up together,
define a model for me.
So, that's it. That's a convolutional
network. The new concepts we are seeing
here are Conv2D for the convolutional
layer and then MaxPooling2D for the max
pooling layer.
Okay? That's it.
Uh
coming. So, let me just run this thing.
It runs. Okay, good. Yeah.
Uh how do you decide when to flatten and
would there ever be a situation in which
we just kind of use the method that we
used before and not use a CNN?
Well, we already tried it with MNIST,
right? We didn't use a CNN. We just
flattened right away.
>> work. It it was it's not bad, but we are
like, you know, can we do better than 85
or 88 or whatever the percent was,
right? So, but we are working with
images, it's typically a good idea to
just start with a CNN straight out the
back because you're not losing anything.
You're not giving up anything.
So, uh in terms of how many uh layers
you should have, my philosophy is start
simple and if it works, stop working on
it. If it doesn't, add more layers.
Uh yeah.
Yeah, just to uh is it the architecture
design, the number of filters, kernel
size, number of layers, convolution
pooling, is that just all based on trial
and error or what's sometimes? Yeah, so
typically it's based on trial and error,
Um to answer your question. But as you
will see in the transfer learning
discussion we're going to have soon,
you can actually, instead of doing
anything from scratch, it's much better
to just download a pre-trained model and
just adapt it for your particular
problem. That is actually the norm by
which people do these things. The reason
I'm doing it from scratch is because you
should know how it was done.
Like you it should not be a black box to
you. That's my goal.
Yeah.
Just for what notation perspective, I
noticed you named all of these layers X.
Is that a habit we should get into
naming them all the same or is that just
a
>> Actually, I'm not naming the layers as
X. What what's going on here is I'm
feeding it X.
And whatever is coming out of it, I'm
just calling it X.
That's all. It's just a notational
convenience for me to I'm I'm just
calling the input and the output and
Keras under the hood will track
everything and make sure the right thing
happens. Otherwise, I'd have to be like
X1, X2, X3, X4 and then if I want to add
a new layer somewhere in the middle
between X3 and X4, I have to call that
X4 and then I'll change everything to 5,
6, 7. Complete pain in the neck. That's
why I do this.
All right. So, model.summary
It has got 302 thousand parameters. I'll
just plot it.
Great. And I encourage you to hand
calculate it later on and make sure the
numbers tally, okay?
For now, let's just go. So, as before,
we'll just use the same compilation.
We'll use Adam and then we'll train it
for, you know, just 10 epochs. We'll use
a validation split again, as usual, of
20%. So, let's just run it.
So, it's actually going to run. And as
you will see,
convolutional networks there's a lot
more going on, so it's going to be a bit
slower to run. Hopefully not too much
slower.
While it's doing, other questions?
So, if we have a task other than image
classification, do we still flat the
model like first and then it's
segmentation?
Yeah, so this is for image
classification. For other kinds of
applications,
typically you run it through a bunch of
convolutional layers and so on and so
forth.
But the output side of the equation gets
much more complicated because if instead
of classifying just
the whole picture into, you know, dog or
cat, if you have to take every pixel and
classify it, right? Then, well, you
better have an output shape that is the
same dimensions as the input shape.
So, for that we use a different
architecture. It's called U-Net
and so on, which unfortunately I won't
be able to get into. But I know I am
planning to post another video
walk-through where I show you how to use
the Hugging Face Hub
to very quickly build models for the
other applications like segmentation and
so on. I'm hoping to post that tomorrow.
It's an optional viewing thing that
might help with that.
Okay. So, is it done? Okay, good. It's
done. All right, let's plot the
thing here.
All right, so it seems like training is
going down nice and nicely. Validation
is sort of flattening out somewhere here
around the eighth epoch. Let's look at
the accuracy.
Same situation here. The accuracy is in
the 90s. Of course, the final question,
of course, is how it will will it does
on the thing.
Whoa, 90.5%.
Pretty good.
By the way, if you're not impressed that
we went from 88 to 90,
this is the These applications are the
proverbial sort of diminishing returns
problems, okay? So, what you should
always think of is look at the amount of
error that's left and ask yourself how
much of that error am I able to reduce?
So, you we had 12% roughly of error left
when we did the simple collab yesterday.
From that 12% we have knocked off two of
the 12% to get to over 90, which is
amazing.
Okay?
And in fact, I think the state of the
art on this
um
is 97%.
So, I invite you
to take this thing and try different
filters and so on and so forth to see if
you can get to the the mid-90s.
It's not easy, but try it. Yeah.
Does the number of epochs have to be
related to the number of batches?
Because you did 64 batches and 10 No,
the epochs is an independent
the epochs is just the number of passes
through the whole data.
But within each pass, within each epoch,
the num the batch size tells you how
many batches you're going to process.
So, it is basically the number of
examples you have in your training data
divided by the batch size that you have
chosen,
right? That number rounded up is the
number of batches within each epoch.
And here I'm just choosing 10 because,
you know,
Siri found something on the web. Okay.
I chose 10 because it's going to be fast
to do for me to do it in class. And 10
is actually more than enough because you
can see it's already beginning to
overfit.
Yeah.
This is more of a conceptual question,
but is it always the case that a neural
network will have better accuracy than
this like machine learning algorithm?
And I'm asking more on the case of like
the heart disease problem. Oh, yeah,
yeah.
Great question. So, neural networks are
really good for unstructured data like
the ones we're having here. But if you
have structured data like the heart
disease problem, sometimes it actually
works really well. Sometimes
things like gradient boosting, XGBoost,
work really well. So, if I am actually
working on a structured data problem,
I'll try both.
I'm not going to axiomatically assume
that the DNN is going to be the best
thing. But if you have structured data,
it's the best game in town.
All right. Um
I'm just going to
By the way, I have a whole section here
on once you build a model, how do you
actually improve it?
Right? Check it out. It's an optional
thing.
All right, I'm going to stop this here.
All right. So, the next thing I want to
do is
So, we went from 88 to 90 plus percent,
right? Using convolutional networks.
Now, let's work with color images. Let's
kick it up a notch.
So, um
I actually
web scraped
all these pictures for you folks, for
your enjoyment. I web scraped about 100
color images of handbags and shoes.
Each 100 roughly 100 handbags, 100
shoes. So, the question is with these
essentially 200 images,
can we build a really good neural
network to classify handbags and shoes?
Right? It seems kind of absurd, right?
Because 200 examples, I mean, it's not
that much, right? It doesn't feel like a
lot. The MNIST data fashion has 60,000
images.
Right? So, there's no, you know, even
with that we are overfitting in like 5,
6, 7, 8 epochs.
With 200 images, maybe, you know, is
there any hope? Obviously, there is
hope, otherwise it won't be in the
lecture. So, yeah. So, we're going to
take this data set and let's see what we
can do with it. So, we'll first actually
build a convolutional network from
scratch to solve this problem. Okay?
All right.
I'm actually going to run through the
code because at the end of it we'll have
a live demo. So, I would like one
volunteer to give me a handbag and one
volunteer to give me their footwear.
Boy, in class.
Okay. So, all right. Unlike the previous
data set, this one actually I just web
scraped it. So, I just, you know, it's
it's it's I've stuck it in this Dropbox
folder.
Let's just download it and unzip it. And
once we do that, we have to now organize
it with these 200 images. So,
I have to do some sort of
sort of boring-ish Python stuff here.
So, here what we're doing is that we
have 100 handbags, roughly 100 shoes.
And what this code is doing is it's
actually creating a directory of saying
it's it's splitting stuff into train and
validation and test. And then for each
of the splits it's doing the handbags
and the shoes folder. Okay? So, once we
do that, basically this directory
structure is created.
Okay? Training, validation folder, test
folder, handbags and shoes. In fact,
actually you can I think you can see it
here.
See here, handbags and shoes. And within
that, there is, you know, train, test,
validation. And within each of these,
there's handbags and shoes. So, the idea
is that when you're working with images,
right? What you can do is you can just
create folders for each kind of image,
right? Let's say dogs, cats,
two folders with cat images and dog
images and then just point Keras at it.
It'll automatically figure out those are
the labels.
It makes it easy for you. So, it's very
convenient when you're working with
images.
And the book explains this thing in
great detail.
All right. So, when working with these
images, color images, we'll follow this
process. We'll read in the JPEGs. We'll
convert them to tensors. And then since
I'm web scraping it, they all come in
different shapes and sizes. So, I need
to like bring it all to the same size.
Okay? I resize it and then I'm going to
batch it into whatever. I'm going to
batch it using a batch size of 32 here.
So, and this utility from Keras will do
all that for you, right? Very quickly.
So, basically what it says is that I
found 98 files in the 98 images in the
training data belonging to two classes,
49 in the validation and 38 in the test.
So, less than 100 examples in the
training set. That's what we have here.
All right. What's the time? 9:30. Okay.
So, all right. Now, let us check the
dimensions to make sure Good. So, 224
224 by 3. And the reason why did I pick
224 224? As you will see later, we're
going to use something called ResNet
and the ResNet expects it to be 224 by
224 by 3. That's why I resized it to 224
224. Let's look at a few examples of my
wonderful web scraping in action.
It's pretty wild, right?
Okay. So, we have a Now, let's do a
simple convolutional network. Um
And before we would take all the X
values in Fashion MNIST and divide them
manually by 255 to normalize it to 0 1.
Well, you know what? We are actually
graduating to the higher levels of Keras
now. So, let's not do that, right?
Manual stuff is bad. So, we'll do it
within Keras by using something called
the rescaling layer where we just tell
it how much to rescale and boom, it'll
do it for you. The first convolution
block, just like the Fashion MNIST 32,
second block, again 32, max pool,
flatten. And then here we only have
handbags which are shoes, just a sigmoid
is enough, right? It's just a binary
classification problem. So, I'm just
using one output layer with a sigmoid,
and that's our model. So, let's do the
model.
All right, model summary.
103 101,000 parameters in this little
model. Okay, let's compile it and run
it. Uh, and note here because it's a
binary
classification problem, I'm using binary
cross entropy.
Same Adam.
And accuracy, compile, and then boom,
let's run it. We'll run it for 20
epochs.
Hopefully.
Okay, while it's doing this business,
I'm going to shift to the PowerPoint.
So, we'll go back to see how well it
did, but the question is, uh, whatever
it did, we built it from scratch. So,
the question is, can we do better than
that? Okay? Because we only have 100
examples of each class, and which brings
us to something very cool and very
powerful called transfer learning. And
the idea, so the key thing is there are
two research trends that are going on
that we take advantage of. The first one
is that researchers have defined, you
know, designed architectures which
exploit the kind of input you have. So,
Olivia asked the question, if you have a
particular kind of input images, do you
actually change the input, or do you
actually change the network? As it turns
out, here, for example, if it's images,
we know that we should use convolutional
layers because convolutional layers were
designed to exploit the image-ness of
the input.
Okay? Similarly, if you have sequences
of information, like obviously natural
language, audio, video, gene sequences,
and so on, so forth, these things called
transformers were invented
to exploit them, and we're going to
spend a lot of time on transformers
starting next week. So, that's the first
trend. The second trend is that
researchers have used these innovations
to actually create and train models on
vast data sets, and thankfully, they've
made them publicly available for us to
use. So, transfer learning is the idea
that if you have a particular problem,
let's just take a pre-trained network
work somebody may have already created,
and then let's just customize it to our
problem, rather than actually build
anything from scratch.
Okay, that's the basic idea. So,
so here we have this basically we have
to build a classifier which takes in an
arbitrary image and figures out if it's
a handbag or a shoe, right? That's our
goal.
And so, now handbags and shoes are
everyday objects, and so what you can do
is, hmm, you you can look around and see
if there are any networks that have been
trained by other people which actually
have been trained on everyday images.
Right? As opposed to like MRI or X-rays,
right? Specialized images, everyday
images. Of course, the first thing you
should probably do is to see if anybody
has built the specific thing you want,
handbag shoes classifier on GitHub.
Assuming it's not, then you do transfer
learning. Okay? So, now it turns out
that there's this thing called ImageNet,
which is a database of millions of
images of everyday objects in a thousand
different categories, furniture,
animals, automobiles, you get the idea.
Okay? And so, we can look for the
networks that have been trained on
ImageNet.
Okay, let me just go back to the collab
just to make sure it doesn't time out.
All right, so it has finished doing it.
Um, let's just plot these things.
Okay, so
uh, there is some overfitting that
happens around here
on the training on the 10th epoch. Let's
look at the
So, the the training accuracy is
actually getting to almost to 100%. But
we're not interested in training
accuracy, right? We care about
validation and test accuracy, and that
seems to be kind of hovering around in
the 80s. Um, so let's just evaluate it
anyway to see what happens.
Okay, so it gets to 80 87% accuracy
on this data set.
It's actually pretty good given that we
only have 100 examples. So, 87%
accuracy, but we pre-trained the whole
thing. I'm sorry, we did everything from
scratch. Okay? Now, then
I'm going to there's this whole section
about data augmentation, which, um, you
know what? Do we have time?
So,
so the idea of augmentation is that when
you have an image,
let's say you take this image, and you
just rotate it slightly by 10°.
If it's a handbag before you rotated it,
it sure as hell is a handbag after you
rotated it.
Right?
It doesn't change The meaning of the
image doesn't change just because you
rotated it slightly. Or maybe you zoom
in slightly, you zoom out slightly, you
crop it slightly, nothing happens.
So, what you can do is you can take any
image you have, and you just perturb it
slightly,
like right there, and then add it as a
new example to your training data.
This is an unbelievable free lunch,
frankly.
And the same thing actually, same kinds
of techniques actually work for text
also, which we'll cover later on.
Right? This broad area is called data
augmentation.
It's a great way when you don't have a
lot of data to artificially bolster the
amount of data you have.
Okay?
Um, and so, and of course, Keras makes
it very easy for you to do all these
things. It has already predefined a
whole bunch of data augmentation layers
for you. So, here's a little example
where I basically take a picture and
then I randomly flip it. So, if it looks
like this, I flip it this way,
horizontal. Okay? Uh, and then I
randomly rotate it by 0.1. I forget if
it's 0.1° or radians, you can look up
the documentation. And then random zoom,
right? Zoom in and out a little bit. Uh,
but it won't do this for every picture.
It will only do it randomly. Okay? So,
that only some pictures will get
perturbed in some ways. And that's how
you make sure there's enough diversity
of pictures that you have.
So, once you do that,
you can actually take a picture and see
what it does.
I just randomly grab a picture, so it
keeps changing every time.
Yeah, look at this handbag.
Handbag slightly rotated this way,
rotated that way.
Some more. Maybe a little bit of zooming
going on, and so on. You get the idea,
right? And there's a whole list of these
things you can do. But when you do those
things, make sure
that what you're doing doesn't actually
change the underlying meaning of the
picture.
It's really important.
Okay? So, for example, if you're working
with satellite data,
yes, be very careful not to do flips of
crazy flips.
Right? Or even if you're working with
everyday images, horizontal flips are
okay. Don't do vertical flips.
Right? How many times will you have an
upside-down dog picture that you need to
classify?
Make sure your augmentation doesn't go
nuts.
All right.
Once you do that, you can actually just
insert the data augmentation layers in
your model right there, right after the
input. The rest of it can stay
unchanged.
So, this is a great way to increase the
size of your training data, and here is
a model, and then I invite you to
actually just play with it and uh, and
train it. We won't try In the interest
of time, we won't actually train this
model, but it's in the collab, you can
just try it. It also figures prominently
in homework one, by the way, data
augmentation. So, you'll get more
experience with this. Okay. So, uh, back
to the PPT.
So, this is what we have. Um, and so,
any network that has been trained on
this ImageNet thing, uh, turns out
learns all kinds of interesting features
in every one of its layers. So, here
this is the first layer, and you can see
it's picking up sort of gradations of
color, sort of line-ish kind of
behavior. Layer two, um, it's actually
picking up Hey, look, it's picking up an
edge. Can you see that edge?
Right? Like like that.
And then layer three is picking up these
interesting honeycomb shapes, uh, and so
on. Oh, it's actually this thing is
already already picking up like the
shape of a human torso.
Yeah, this layer is actually picking up
what looks like a Labrador retriever.
Okay.
Isn't that cute?
Come on, even if you're not a dog
person.
All right. So, the the this this is the
visualization I was referring to
earlier,
um, to figure out what are these
networks actually learning.
This paper was one of the first ones to
actually visualize what's going on
inside. So, if you folks are curious how
these pictures are actually produced, I
would encourage you to check this out.
Okay, yep.
So, we spoke about images and you
referred to classes, but sorry, we spoke
about images and you referred to classes
and
text next week on transformers, but
what about say an email which has both
text and image, and that may be white
space depending on who has written it
out. Does that get put in as an input
for an image or
So, we'll revisit this great question a
bit later on in the course.
So, the answer is a bit complicated, so
I don't want to I want to do it justice,
so we'll come back to it.
All right, so
so it turns out this thing called ResNet
is a family of networks that are which
were trained on this ImageNet data set,
and they did really well in this
competition that's associated with the
ImageNet data set called ImageNet. And
so, this is an example of such a
network. So, you we would expect the the
weights and the parameters of ResNet,
given that it's been trained on
ImageNet, to sort of have some knowledge
about lines and shapes and curves and
things like that. So, maybe we can just
use that, right? So, so the idea is we
But the thing is we can't use ResNet as
is because remember, it was trained to
classify an incoming image into a
thousand possibilities.
Here we only have two possibilities,
handbags and shoes. So, what we do is
very simple and elegant. We do just a
little bit of surgery.
We take ResNet and stop just before the
final layer. So, take my word for it,
this thing here, what it says is fully
connected thousand.
Because it's got thousand way, right?
Thousand objects. So, what we do is we
just take everything except and we stop
just before that last layer.
And then what comes out of that layer,
hopefully, will be like a very smart
representation of the images that it has
been trained on.
And so, what we do is we can think of
sort of headless ResNet
as our model.
And we can take that we can take all our
data and run it through ResNet up to but
not including the last layer.
Okay, you get some tensor and that
tensor is probably like a very has a
very rich understanding of what's going
on in that image, all the objects and
features and things like that. And then
we can just simply connect that we can
think of it as like a smart
representation of an input. We can
connect it to just a little hidden layer
and then we have a little sigmoid which
then tells you handbag or shoe. We can
just run this network.
Okay? Um and so since the outputs to the
hidden layer now are not raw images
anymore, but this much higher level of
abstraction that ResNet has learned,
hopefully it can get the job done with
hardly any examples.
Okay? And now you can get fancier.
That's the basic idea, but you can get
much fancier. You can connect up
headless ResNet directly with our little
network with a hidden layer and the
final thing and the whole thing can be
trained.
End to end. Uh but when you do that you
must start the training with the weights
that you downloaded with ResNet because
that is the crown jewel that's been
learned so you want to start from there.
Uh and you will do this in homework one.
Okay? All right. Uh by the way, these
pre-trained models are available all
over the internet. There is the
TensorFlow hub, the PyTorch hub and then
there's the Hugging Face hub. When I
checked it on the 13th yesterday, it had
over half a million models available
for download. Half a million.
I think last year it was like 50,000
when I taught the course. Uh so yes.
I was just wondering, doesn't this make
your neural network susceptible to
adversarial attacks because the weights
have been
pre-trained on a Yes. Uh it there is
some adversarial risk. I'm happy to talk
about it offline.
All right. So that's what we have. So
back to Colab. Okay. So that's what we
have. This is ResNet. So what we do is
and ResNet is all packaged up. It's
available for download. So we download
it here.
And you see here that I'm saying use
include top equals false.
So basically you are telling Keras
uh the top the very final layer of the
thing, don't give it to me. Just give me
everything up to but not including that.
And of course I think of it as left to
right. People think of it as bottom to
top. So they could the very very top
layer, don't give it to me. You're
telling it so that you don't have to
manually go and remove it.
Okay? And then I'm not going to
summarize uh well, I'll just summarize
some of it. Just show you how big it is.
Okay?
23 million parameters.
ResNet. Okay? And I won't plot it
because then I'll be scrolling for 5
minutes. Uh
so let's just do this now. So what we're
now going to do is we're going to run
all the data through this thing and
whatever comes out in that penultimate
thing, I'm going to just grab it and
store it. So that's what this thing
does.
All right. And now we create this a
little handy function to do all these
things.
And once I do that,
uh every image has been sent through
ResNet up to but not the final layer and
then whatever comes into the final
layer, we're storing it. And then we're
going to create a network where we'll
only feed that layer that information to
a simple network.
Okay?
So what is coming out of ResNet, you can
see here 98 examples in the training
data and each example is now a 7 by 7 by
2048 tensor.
That's what came out of ResNet and you
saw that's what I did there.
Okay?
All right. So that's what it looks like.
Now let's just create our actual model
now. Right? We have our input which is
just a 7 by 7 by 2048.
We flatten it immediately.
Then we run it through a dense layer
with 256 ReLU neurons and then we use
dropout which I haven't talked about yet
which I will talk about early next week.
Uh but I will come back to it. Don't
worry about this detail for the moment.
Uh and then we just run through a
sigmoid.
Okay? And that that's our model.
Finished. Plot the model. This is what
we have. Okay? Model summary.
It's one so far. All right, good. Now
let's actually train this thing.
I'm just going to run it for 10 epochs
because I tried running it uh previously
and it seems to do a fine job in just an
epoch. Okay, it's already done. It's so
fast because we ran everything through
this monster ResNet thing and basically
took all the output values and use them
as a starting point. Right? We don't
have to run it every single time. So you
can see here the accuracy is
quite high.
Wow, interesting. So the 10th epoch
something bad happened.
So maybe I should have stopped at the
ninth epoch. I didn't see this yesterday
when I was running. So much for random
reproducibility. Uh
So let's just run this. Oh wow, look. On
the test set it's achieving 100%
accuracy.
It's unbelievable. Okay folks, now for
the moment of truth. Um all right, I
have a little code snippet here to
capture stuff from the webcam.
Because that last epoch it went down,
I'm a little worried that the demo is
going to flunk.
But you know what? We all have to live
dangerously. So
So here's a little function to predict
what's going to happen.
Okay. Now I tried it at home yesterday
by the way.
I act and it's like, "Yay, it's a
handbag."
So okay. Now let's just do something
else.
Okay. Any volunteers?
I want a a piece of footwear
or a handbag.
It's like a backpack, right?
I don't know. It feels like an
adversarial example, but yeah, let's
just try it.
Okay.
No disrespect. I'll let me let me go
with the shoe first. I have a better
chance of it working.
So
it's a pretty big shoe. If it can't get
this shoe, I'm worried about this model.
All right. So
Okay. Hold on. Hold on. Hold on.
All right.
Please don't get distracted by my hand.
Capture.
It's a shoe! LOOK AT THAT.
PHEW. ALL RIGHT. THANKS.
OKAY. Now let's try that. I'm feeling
kind of brave now.
Thank you. All right. Let's do this.
All right.
Camera capture.
Okay.
Put its better side.
It's a handbag! Look at that.
I swear every time I do the demo I age a
few years. So
All right folks, I'm done. Thank you.