All right. So, today's lecture,
introduction to neural networks and deep
learning.
Um so, we'll start with a very quick
intro to these things,
uh and then we'll switch and dive deep
into neural networks. All right. So, the
field of AI originated in 1956. Sadly,
it didn't originate at MIT, it
originated at Dartmouth.
Because all these people got together at
Dartmouth. I guess it's it's got a nice
quad or whatever. They got together,
they defined the field. But, fortunately
for us, MIT was very well represented.
So, we have Marvin Minsky who founded
the MIT AI Lab, John McCarthy who
invented Lisp, and then later defected
to the West Coast, and then Claude
Shannon who invented information theory,
right? Who was a professor at MIT. So,
MIT was well represented. These folks,
you know, founded the field, and they
were so bright, they thought that AI was
going to be substantially solved, quote
unquote,
by that fall.
Okay?
Now, obviously, it turned out a bit
differently than what they expected.
Um so, it's been, whatever, 67, 68 years
since its founding. So, it's gone
through, essentially, in my opinion,
three seminal breakthroughs,
um starting with the traditional
approach, then machine learning, deep
learning, and generative AI. So, let's
take a very quick look at each of these
breakthroughs and what motivated them.
So,
let's start with the traditional
approach to AI. And so, what is AI? AI,
informally, is the ability to imbue
computers with the
the the the ability to do things that
only humans can typically do. Cognitive
tasks, thinking tasks, and things like
that. And so, the most sort of common
sensical way to do that is to say,
"Well, if I want the computer to do
something complicated like play chess,
I'm just going to sit down with a few
chess grandmasters,
show them a whole bunch of board moves,
and ask them how they figure out how to
respond, how to play the next move." I'm
going to sort of sit down, talk to all
these people, and then I'm going to
write down a whole bunch of rules. If
this is the board position, move this.
If this is the board position, move
this, and so on and so forth. Or I might
sit down with a cardiologist and tell
them, "Okay, how do you actually
interpret an ECG?" They will give me all
the similarly a bunch of if-then rules.
I will take all these rules, I'll put
them into the computer, and boom, I have
a system that can do what a human can
do. Right? Now, this approach, even
though it's common sensical and kind of
makes sense, it had success in only a
few areas.
Um and so, the interesting question is,
why was it not pervasively successful?
Why was it not pervasively successful?
It seems like a pretty good idea to me,
right? And the people who came up with
these things are smart people, they're
not dumb people. They know what they're
doing. So, why did it not work?
Because
because it's time-intensive,
so in case that you have to run through
all these scenarios that can ever exist,
and still some new scenarios can come up
that you didn't cater for initially.
Right. So, there are two aspects to what
you said, which is the first aspect is
it's time-intensive. That, as it turns
out, is not a big deal, because
computers are getting faster and faster.
>> [clears throat]
>> Right? The second thing is actually the
key thing, which is that it doesn't
generalize to new situations very well.
Right? The problem is
there are an infinite number of things
that you're going to see when you deploy
these systems in the real world. By
definition, what you're training it on
is a small sample of rules. So, these
rules are very brittle. But, there's
actually even more interesting reason.
And that reason is that we know more
than we can tell.
This is called Polanyi's paradox. So,
the idea is that if I come to you and
say, "Hey, uh here's a picture. Is it a
dog or a cat?" you will tell me within,
I believe, they've measured it, like 20
milliseconds or something, you know if
it's a dog if it's a dog or a cat. And
then if I ask you to explain to me
exactly how you figured that out, you'll
come up with a bunch of sort of reasons,
right? Alleged reasons. Oh, you know, if
it has whiskers, I think it's a cat or
whatever.
But, the problem is that you actually,
first of all, can't really articulate
what's going on in your head, how you do
these things. And number two, even if
you articulate it, often times, your
articulation has no correspondence with
how your brain actually does it.
So, you're incomplete and a liar.
So, this is Polanyi's paradox. So, if
you can't even
tell me how you do something, how the
heck am I supposed to take it and put it
into a computer? Doesn't work. And
second is the fact that we can't write
down these rules for all possible
situations. Edge cases, corner cases,
etc. And the world is full of edge
cases.
So, for these reasons, this approach
didn't work.
And so, a different approach was
developed, and this approach was, well,
basically said, "Hey, instead of
explicitly telling the computer what to
do, why don't we simply give it lots of
examples of inputs and outputs, chess
positions, next move, right? ECG,
diagnosis, right? Inputs and outputs.
And then, why don't we just use some
statistical techniques to learn a
mapping, a function, that can go from
the input to the output? Okay? That was
the idea.
And this idea is machine learning.
Okay? So, machine learning is basically
just a fancy way of saying, "Learn from
input-output examples using statistical
techniques."
Good. All right. So, um
Now, there are numerous ways to create
machine learning models, and if you've
ever done linear regression,
congratulations, you've been doing
machine learning.
Okay? And only one of those methods
happens to be something called neural
networks.
There are many other methods, and in
fact, you probably have done these other
methods if you have done the a course
like the Analytics Edge or something
similar.
Okay. So, machine learning has got
tremendous impact around the world,
right? It's like, at this point, um it's
widely accepted, it's a very, very
successful technology.
And in fact, whenever people are
actually talking about AI,
chances are they're actually talking
about machine learning.
It's just that AI sounds cooler.
The only problem is, for machine
learning to work really well, the input
data has to be structured.
Okay? And what I mean by that is data
that can essentially be sort of
numericalized and stuck into the columns
and rows of a spreadsheet.
Right? So, for example, here, let's say
I want to put together a data data set
of, you know, uh patients, their
symptoms, and their characteristics, and
then in the following year after they
showed up at the doctor's office whether
they had a cardiac event or not.
I might create a data set like this with
age, smoking status, yes, no, exercise,
blah blah blah blah blah. Right? And so,
either these numbers are numbers,
they're numerical, or if they're not
numerical, they're categorical.
Right? Yes, no, uh smoking, yes, no,
things like that. Which means that if
you have categorical variables, you can
just numericalize them pretty easily.
You folks have done the some machine
learning before, so you know, things
like one-hot encoding and stuff like
that can be done to make them all
numerical. So, the point is, you can
just render the data into the columns
and rows of a spreadsheet pretty easily,
right? That's what I mean by structured
data. So, when you but the situation is
very different if you have unstructured
data. So, if you have an image of, you
know, a cute puppy, this is my puppy, by
the way, um
from many years ago. Sadly, he's no
more. Um
but, his name was Google.
So, yeah, anyway, uh
my DMD alums know Google well. So, this
is Google, right? If you want to take
Google,
uh this picture, and figure out how to
sort of numericalize it, the first thing
you want to need to understand is that
if you actually look at how this picture
is represented inside, uh digitally, in
the computer, basically, every picture
like this is represented using three
tables of numbers.
Okay? And these and we'll get to what
these numbers mean later on, but the
point I'm making is that each number
basically represents the amount of
light,
right? On a scale of 0 to 255, the
amount of light in that location, in
that pixel. That's all the amount of
light. So, basically, the this table is
the amount of um sorry.
This table is amount of red light,
amount of green light, amount of blue
light. Okay? Now, you will agree with me
that if you, for example, look at
something like this and say, "Okay, 251
at this location, there is a lot of blue
light because it's 251 out of a possible
255, right? Maybe a lot of blue light
somewhere here. There's a lot of blue
here."
Whether that area is blue because of a
piece of sky,
some water, or a bunch of blue paint,
could be anything, it's going to say
251.
So, the underlying reality, the
underlying object that's being
described, has nothing to do with the
251.
Right? So, that's the whole problem. The
raw form of the data has no intrinsic
meaning with the underlying thing.
So, given that there's no connection
between the number and what it's
describing, how the heck can any
algorithm do anything with it?
It can't.
Right? So, what you have to do is
something called feature engineering or
feature extraction, right? Where you
have to manually take all these things
and create essentially a spreadsheet
from them. So, basically, let's say that
you have a bunch of birds, right? And
you're trying to build a a bird
classifier to figure out what kind of
bird species it is, you might actually
have to take this picture, and then you
have to measure the beak length, the
wingspan, the primary color, and so on
and so forth.
So, you're basically structuring the
unstructured data manually, right?
And this process of structuring
unstructured data is basically called
we use the word representation. We take
the raw data and we represent the data
in a different form. And the the reason
why I'm sort of
focusing on the use of the word
representation is because it becomes
really, really important a bit later on
when we get to deep learning. Okay? So,
we have to represent the data in a
different way for it to work. That's the
basic idea.
All right. So, what that means is that,
uh historically, researchers would
manually develop these representations.
And once you develop them, once you have
representations, you can just use
traditional linear regression or
logistic regression to get the job done.
So, the whole name of the game is the
representations. So, in fact, people
doing PhDs, for example, in computer
vision, would spend like 4 years
developing amazing representations for
solving one particular little problem.
Right? We have a bunch of, say, CAT
scans, and we need to take the CAT scan
and figure out whether a particular kind
of stroke that is evidence for it in the
cat scan, right? They might actually sit
and develop all kinds of representations
and test it and so on. And then they'll
finally declare victory and say, "Yay,
I'm done with my PhD. Here is this
amazing representation, and you can
build a classifier with it to predict a
particular kind of stroke with a high
accuracy." Okay? So, that was what that
that's where the world was.
Uh now, as you can imagine, developing
representations, because it's so manual,
is this massive human bottleneck, and
this sharply limited limited the reach
and applicability of machine learning.
As you would expect.
To address this problem,
a different approach came about, and
that's deep learning. So, deep learning
sits inside machine learning. Okay?
And deep learning
can handle unstructured input data
without upfront manual processing.
Meaning,
it will automatically learn the right
representations from the raw input.
Automatically is the keyword.
Automatically learn representations,
which means that you could give it
structured data, you can give it
pictures, you can give it text, you can
give it anything you want, it just learn
it.
Okay?
Um it can automatically extract these
representations, and since it's being
automatically extracted, you can imagine
sort of a pipeline where the raw data
comes in, you have a bunch of stuff in
the middle that's learning these
representations automatically without
your help, and then boom, you just
attach a little linear regression or
logistic regression at the end, problem
solved.
That in a nutshell is deep learning.
Input, a whole bunch of representations
being learned, and then piped into a
linear or logistic regression model.
Okay?
You would So, the amazing thing is this
simple idea
this simple idea
is just incredibly powerful. Right? That
idea has led to ChatGPT, has led to
AlphaGo, AlphaFold, and so on and so
forth.
And
I I kid you not, I'm sort of
I've I've I've been doing deep learning
for about 10 years now, and every time I
look at it, I literally get goosebumps
every so often.
That that something so simple could be
so powerful, right? It's really like
boggles the mind.
I'm like I'm just so lucky to be alive
and working during this period.
Okay?
And you know, coming from people who
have been in the industry a long time,
this sort of breathless exclamation is
not very rare, particularly because I'm
not in marketing.
Okay? I actually mean it.
With all your apologies to various
marketing folks. So,
just realized it's being taped, so uh
okay. So, so this has demolished the
human bottleneck for using machine
learning with unstructured data, uh and
so it comes from the confluence of three
forces,
uh new algorithmic ideas, a whole a lot
of data, and then very importantly, the
fact that we have access to parallel
computing hardware in the in the form of
these things called GPUs, graphics
processing units. Um and these three
forces came together, and they were
applied to an old idea called neural
networks, and that's basically deep
learning. And I'll go through it very
quickly, because obviously we going to
spend half the semester looking into
this thing in detail.
Uh so, what's the immediate immediate
application of the ability to
automatically handle unstructured data?
What is like the no-brainer application?
It's okay if it's obvious, tell me.
Uh sorry.
Um image classification. Right. So,
image classification, yes. So, you can
take an image, a good example of
unstructured data, you can do some
classification on it. But more
generally, more generally, what I'm
getting at is that every sensor in the
world
can be given the ability to detect,
recognize, and classify what it's
sensing. Every sensor. Because remember,
what is a What does a sensor do?
A sensor is just a receptacle for
unstructured data.
A camera is a receptacle for
unstructured video
or unstructured, you know, still images.
Microphone, unstructured audio, right?
So, every sensor, you can you can
imagine taking a sensor and sticking a
little deep learning system behind it.
And now suddenly, the
what comes out of this sensor the deep
learning system, you can count, you can
classify, you can detect, you can do all
kinds of stuff. In short, you can
analyze.
And you can predict, right? And this is
the way I'm describing it right now,
you'll be like, "Yeah, duh, obviously."
But you know what, this obviously thing
is actually not at all obvious
in terms of whether it'll help you find
interesting applications or not. Okay?
So,
here's something I literally saw
last week. Okay? Actually, I have
another slide before that, but we are
coming to that. So, for instance, every
time you use Face ID to unlock your
phone, this is the basic principle at
work, right? The the camera in the
iPhone is the sensor, and they stuck a
deep learning system behind it to do
image classification, right? Drama,
non-drama, right? That's what it's
classifying.
Um and so here, right, you have a breast
cancer is it's a breast cancer detection
system from a mammogram.
Uh by the way, this picture
it's a very interesting picture. So, uh
there's a professor in WCS, uh Regina
Barzilay, who's a very well-known expert
in this field, and uh she actually has
built a breast cancer detection system,
which is which has been deployed at Mass
General Hospital.
And turns out she's actually a breast
cancer survivor. And uh she was
you know, she's she's she's good now,
all good. But when um after she built
her system, I heard that she actually
ran that system against the mammograms
from many years prior when she went for
a mammogram and was told that everything
is fine.
She ran the system on that mammogram,
and it came back and said, "Here is a
problem."
So, a very interesting example where a
deep learning system picked up something
that a radiologist could not, right? So,
these things can be quite powerful.
Um obviously, any self-driving system
has numerous deep learning algorithms
running under the hood, you know,
pedestrian detection, you know,
stoplight detection, zebra crossing
detection, and so on and so forth. Um
you know, it's being very heavily used
in visual inspection manufacturing.
Uh you have various cameras now instead
of people looking at saying, "Okay,
there is a dent or there's a scratch."
They have a little system, which is a
dent detector, scratch detector, and so
on. That's that's going on right now.
And now I come to the example I saw last
week,
which is um So, this is an example of
you can create dramatically better
products if you really internalize this
idea of, "Okay, it's almost like you're
looking the the world and saying, 'Oh,
there's a sensor. Can I attach a DL
thing behind it?'" That's the way you
should be looking at the world, okay,
for startup ideas. So, here's an
example, okay, these apparently are the
world's first smart binoculars.
Okay?
This is the binocular.
Two weeks ago,
where you look at the bird you look at
the bird,
and now it tells you what kind of bird
it is, right there.
It's a simple idea, but imagine, right?
Imagine you are the first out of the
gate with this feature, you'll have a
little bit of an edge till everybody
catches up like 3 months later.
Let's be very clear, there are no
long-term monopoly windows in the world.
There are only short-term windows, so
the hunt is always on for a little
monopoly window.
So, here's an example of that.
Right? So, I encourage you to always
think about the world as, you know,
where are the sensors here?
And can I attach something behind the
sensor to do something useful with it?
Okay?
All right.
Now, let's uh turn our attention to the
output.
We've been talking about in structured
data, unstructured data, and how deep
learning has sort of unlocked the
ability to work with unstructured data,
but you've sort of been neglecting the
output side of the equation. So,
traditionally, uh we could predict
single numbers or a few numbers pretty
easily, right? So, you've all done the
canonical, you know, uh should this
person be given a loan application in
machine learning, right? So, you just
predicts a probability that a borrower
will repay a loan on a whole based on a
whole bunch of data, or supply chain,
you predict the demand for the product
next week, or you could predict a bunch
of numbers. So, given a
um given a picture, you can say, "Okay,
does it Which which one of the 10 kinds
of furniture is it?" Right? You can
predict 10 numbers, 10 probabilities
that add up to one. You can predict a
whole bunch of numbers that don't have
to add up to one, such as the GPS
coordinates of a of an Uber ride. So,
these are all simple unstructured Sorry,
simple structured output, just a few
numbers, right? What we could not do
very easily was to actually generate
pictures like this.
We could not generate unstructured data.
We could only consume unstructured data,
right?
Um you can generate text, you can
generate pictures, and so on, and audio,
and so on, and so forth.
So, with generative AI, that problem is
gone.
So, generative AI is the ability to
actually create unstructured data, all
right? And therefore, it sits within
deep learning. It still runs on deep
learning, but it's just one kind of deep
learning.
Okay? There's plenty of stuff going on
in deep learning that's got nothing to
do with generative AI.
Nowadays, of course, you know, if you're
a self-respecting entrepreneur who wants
to ride this craze, you'll probably
declare whatever you're doing as
generative AI.
Right? Um and some VCs may actually be
ready to fund you, who knows?
But the point is, there's plenty of
stuff going on in deep learning that's
got nothing to do with generative AI. Uh
but this is the overall picture. Now,
here, uh we can produce unstructured
outputs, like pictures. You can take
this thing, and then you can actually,
you know, come up with a nice picture
description of it. This actually is a
very famous picture, by the way, in in
the world of computer vision. So, we are
actually going to be analyzing this
picture a little later on
in the semester.
Uh you can obviously go from a very
complicated caption to an image.
Uh you can go from text to music.
Can people hear it? Okay. Yeah. Yeah.
All right. So, and of course, we can go
from text to text, i.e., ChatGPT. Uh and
then uh as of a few months ago, things
have gotten even more interesting, where
you can actually go you can send text
and an image in, and you can get text
out.
Right? And in fact, as of a few weeks
ago, you can send text, image, text,
image, text, image in in an arbitrary
sequence
into into the system, and it'll actually
come back to you with text and image.
Right? So, things are becoming
multimodal. I just want to share with
you like a really fun example I saw
uh recently. So, this person
sends this picture. Can folks see this?
It's this very complicated parking sign.
Apparently in San Francisco.
And they're like, it's Wednesday at 4:00
p.m. Can I park here?
Tell me in one line. Because you really
didn't want GPT-4 to be giving you a big
essay about this.
Like, you literally want to park.
So, GPT-4 comes back and says, "Yes, you
can park here for up to 1 hour starting
at 4:00 p.m."
And folks, I double-checked this thing,
it's correct.
We all know these things hallucinate,
right? Can you imagine getting a parking
ticket and telling the judge, "I'm
sorry, I didn't realize it was
hallucinating."
So,
so you have to double-check it.
So, yeah. So, things are getting
multimodal very quickly.
Uh and so, the picture here is that
within gen AI, we used to have these
separate circles, text to text, text to
image, text to music, text to this, text
to that, so on and so forth. Those are
all beginning to merge now inside gen AI
because multimodal models are going to
become the norm this year, right? We
already have really good closed models.
We really have We actually already have
very good open-source multimodal models.
And so, my feeling is that by the end of
the year, the idea of using a text-only
model is going to be like, "Really, you
do that still?"
Right? It's going to become like a
quaint, old-fashioned thing. I think
multimodal modality is going to become
the norm. So, that's where the world is,
and this is the landscape. So, any
questions on the landscape?
Before we actually start doing some
math.
Okay.
Yeah.
You mean the the the evidence of that
being a problem would have been smaller.
Yeah.
Yeah. So, I think the So, the question
is that in general, how do you train
your models so that it gives you the
right answers given that over the
passage of time, the amount of evidence
in this data could be very highly
variable. So, in this particular case of
you know, the professor I talked about,
uh yeah, everything at that point was
going through a an expert radiologist.
So, 5 years ago, this mammogram was seen
by a radiologist, and that person
concluded there is no problem. So, that
was the training label, right? The wrong
training label. Uh so, in typically what
happens is that training labels could be
wrong some small fraction of the time.
So, you need to have systems that are
robust. So, your data needs to be
complete, it needs to be comprehensive,
it needs to be have correct labels. If
these ideas are not met, your systems
are not going to be that good. But as it
turns out, with neural networks, even
with some amount of noise in the labels,
they still do a pretty good job.
Right? So, it's that's sort of the
general idea.
The veri- The verification comes from
the human. So, every Remember when we
look at radiology data,
the the data we're working with is the
input is let's say an image, like a
radio mammogram or something, and then a
human radiologist or a set of
radiologists have said this has a
problem or does not have a problem. So,
that is called the ground truth.
So, it is this ground truth image and
label, this combination that's being
used to train these models.
Yeah.
Embodiment? So, So, are we are we going
to cover embodiment? So, the
the embodiment here refers to the fact
that
if you have robot robots, right?
They need to actually operate in the
real world, and so robots are an example
of what's called embodied intelligence.
So, unfortunately, due to the
constraints of time, we're not going to
get into robotics at all. But I will say
that a lot of the deep learning stuff
you're going to talk about, those are
all fundamental building blocks in
modern robotic systems.
All right. So, um so, in summary,
X and Y
can be anything, and it can be
multimodal.
Okay? I literally could not have put up
this slide maybe 2 years ago.
Right? So, it's very simple in how it
looks, but it's very profound. You can
You can learn a mapping from anything to
anything at this point very easily as
long as you have enough data.
Okay? So, um now, note that all this
excitement that we see around us
is everything stems from stems from deep
learning.
Okay?
Everything Everything depends on deep
learning. And so, if you understand deep
learning, a lot of interesting things
become possible. So, let's get going.
All right. So, we'll start with the very
basics. Uh what's a neural network?
Uh now, recall logistic regression
from back in the day.
So, what is logistic regression?
You send in a bunch of numbers, a vector
of numbers, and you get usually get a
probability out, right? Between 0 and 1.
What is the probability of something or
the other? Okay? Um and so, this
logistic regression model is also
represented in this form,
if you will recall. So, basically what
we do is we take all these numbers, we
run it through a linear function, right?
We run it through a linear function, you
get a number, and then we take that
thing and run it through 1 / 1 + e
raised to minus that,
and that's guaranteed to give you a
number between 0 and 1, which can be
interpreted as a probability, and that's
logistic regression. Okay? And the
canonical, you know,
uh loan approvals, things like that, all
fall into this sort of convenient
bucket.
Okay? So, this should be super familiar.
All right. Now, we're going to actually
look at this, you know, simple, modest,
humble little operation
using the lens of a network of
mathematical operations, and the reason
why we do it will become clear a bit
later.
So, we'll take this very simple example
where we have uh let's say two
variables, GPA and experience, right?
This is the GPA of some graduates, uh
number of years of work experience, and
then this is the dependent variable,
which is either 0 or 1, and 0 if they
don't get called for an interview, 1 if
they get called for an interview. Okay?
It's a two-input variable, one-output
variable problem. Okay? And it's a
classification problem because we're
classifying people into will they get
called for an interview, yes or no.
Okay?
And so, that's the setup for this
problem.
And let's say that we actually run it
through any you know, we actually try to
fit a logistic regression model to it.
So, if you're familiar with R, for
example, you would use something like
GLM to fit this model.
Um if you use something like statsmodels
in Python, there's a similar function
for it. Scikit-learn, there's another
function for it. You get the idea,
right? This
You can use whatever favorite methods
you have for logistic regression
modeling to get this job done. And if
you do that with this little data set,
you're going to get these coefficients.
Right? The 0.4 is the intercept, 0.2 is
the coefficient for GPA, 0.5 for
experience. And that is the resulting
sigmoid function.
Okay?
All right. Cool. So, now let's actually
rewrite this formula as a network in the
following way. So, first, what we'll do
is we'll take GPA and experience and
stick it here on the left side, and
we'll put little circles next to them,
and we'll call them the input nodes.
Okay? And so, imagine that somebody puts
the writes a GPA into the circle, 3.5 or
you know, years experience, 2.0, and
then it flows through this arrow,
and as it flows through, it gets
multiplied by its coefficient, 0.2. The
0.2 is coming from here.
Similarly, experience gets multiplied by
0.5, it comes in here, and this node, as
the plus indicates, is adding everything
that's coming into it.
So, it's adding 0.2 * GPA, 0.5 *
experience, plus the intercept, which is
the green arrow coming from on its own.
It comes through here, and what comes
out of this is just a single number,
and that number goes into this little
circle,
and then out pops a probability.
Okay?
So, I've sort of
done this ridiculously long long
long-winded way of writing a simple
function.
Okay? And the reason we why I'm doing it
will become clear in a second.
Okay? So, this is a little network of
operations for the simple function.
And so, for instance, how you would use
it is you to make a prediction, you'll
let's say someone has a 3.8 GPA and 1.2
years experience, you just plug it in
here,
do the math, you get 0.76, same thing
here, comes in here, add them all up,
you get 1.76, you run 1.76 through the
sigmoid, you get 0.85, and that is the
probability that that particular
individual may get called for an
interview.
Okay? At this point, we're just doing
logistic regression, nothing more
complicated.
Okay? So, um now, if you have many
variables, not two variables like X1
through XK, you can the same sort of
logic applies. Each one has some
coefficient, and then there's an
intercept, they all get added up here,
run through a sigmoid, and out pops this
number. Okay? Notice how the data flows
from left to right.
Okay?
All right. Any questions on this?
All right. Good.
So, now terminology.
Uh so, you will actually you'll discover
that the world of neural networks and
deep learning has its own terminology.
They have their own ways of referring to
things that we the rest of the world has
been referring using something else for
the longest time.
Right? It's kind of annoying sometimes,
but it's the way it is. So, um
Remember in regression, we used to call
those numbers next to each variable as
coefficients,
and the constant thing as an intercept?
Well, guess what? In this world, these
multi- those coefficients are actually
called weights,
and the intercepts are called biases.
So, in in the neural network world,
these are called weights and biases.
And sometimes, if you're a little lazy,
you may just call the whole thing as
weights.
Okay? So, when you see in the newspaper
that, you know, "Oh my god, this amazing
model's weights have been leaked
on the internet or on BitTorrent or
something." That's what's going on,
right? All these coefficients have been
leaked. Because once you know what the
coefficients are and what the
architecture is, you can just
reconstruct the model.
All right. So, that's what's going on
here.
Now, why did we do this network
business? Why did we write it as a
network?
Yeah, what is the advantage? Any
guesses?
When you have multiple functions for
So,
it's just easier to see it that way.
Right. If you have lots of things going
on, it's easier to see it if you
actually write it in graphical form.
Yes, correct.
But, so is it only like a usability
advantage?
I mean, the thing is you want different
functions for different layers of that.
Uh-huh.
Okay.
So, maybe we want to use different
functions in different layers. But, I
think there's actually even a larger
sort of a more basic point, which is
that
then when you the moment you write it
down, you suddenly realize
that I could have lots of things in the
middle.
I don't have to go from the input to the
output directly. I can do lots of things
in the middle, right? That's sort of the
key idea. So, what you do is
So, remember the notion of learning
representations of unstructured data,
right? Where you take a picture and say
beak length and things like that, right?
And remember, I said deep learning
actually automatically learns these
things. Where is that automatic learning
coming from?
Well, this is where it's coming from.
So, what we do is we take this thing,
right? There's just a logistic
regression model. Inputs
get multiple added up as a linear
function, run through a sigmoid.
And then
we are like, "Hmm, if we want to learn
representations of the raw input, we
better be doing something in the middle
here."
Because the output is the output.
That is That's not going to change.
You know, it's it's either a dog or a
cat. You don't have any choice
as to what it is. Okay? The only agency
you have at this point is you can take
the raw input and do things in the
middle with it.
You can do a lot of stuff in the middle
and then run it through something to get
the output. Okay? So, in any in in in
any mathematical discipline,
if someone comes to you and says,
"Here's a bunch of data.
I want you to do something with it."
What should the What is like the big the
most basic first thing you should do?
Run it through a linear function.
The most basic thing in math is a linear
function. So, given anything, just run
it through a linear function. See what
happens.
So, that's exactly what we can do. So,
the simplest thing we can do here, we
can insert a bunch of linear functions.
So, we do is we take all this input and
we just run it we we do a linear
function on it. So, think of it this as
X1 * 2 + X3 * 4 and all the way to XK *
9 plus some intercept and boom, it goes
out the other end. So, this little
circle here with a plus in it is just
Thank you.
Uh
that is This is just a linear It's a
shorthand for a linear function.
So, whenever you see a circle with a
plus, it's just a shorthand for a linear
function. Okay? So, you can take this
whole thing and run through a linear
function and when you do it, you'll get
some number right there. You'll get some
number. So, you've taken these K numbers
and you've sort of dis- compressed them
in some way into one number.
Okay?
But, you don't have to stop at one
number. You can do more.
So, we can have a stack of linear
functions in the middle.
Right? There's a linear function here,
another one here, another one here. At
this point, the K numbers you have
K could be, for example, 1,000.
Right? It's just the size of your input
data.
You've taken these K things and you've
compressed them into three numbers at
this point.
Okay?
So, okay, maybe three is the right
number, maybe 10 is the right number. We
don't know.
And we'll get to know how do we know
what the right number is later on.
So, we can stack as many linear
functions we want.
So, we have transformed this K thing
into a three-dimensional vector, right?
K numbers become three numbers.
Um
and now we can flow this three these
three numbers through some other little
function.
Okay?
And as you will see in a few minutes,
that function is called an activation
function
and it's chosen to be a non-linear
function
because if you don't choose it to be a
non-linear function, all the effort we
are doing is going to be a total waste
of time.
Okay? For now, just
take it on faith that you need to have
non-linear functions here.
But, note that the three numbers here
are still three numbers. They are three
different numbers, but they're still
three numbers.
And once we do this, we'll be like, "You
know what? This was fun. Let's do it
again."
Okay? So, you can do it again.
And you can keep on doing it. You can
keep it 100 times if you want.
And the key thing is that every time you
do it, you're giving this network some
ability, some capacity to learn
something interesting from the data.
To learn an interesting representation.
Now, of course, you're thinking, "Well,
how do we know it's interesting? How do
you know it's a useful thing?" And we'll
come to all that later on.
Right? We're just giving it the
capacity, the potential to learn
interesting things from the data.
Whether it actually lives up to its
potential, we don't know yet.
Okay? We'll give it the potential.
Because the more transformations of the
input data you make, the more
opportunity you have to do interesting
things with it.
If I don't even give you the opportunity
to transform it once, you don't have any
opportunity, right?
If I give you 10 chances to transform
things, you have 10 shots at doing
something useful.
So, you can you can do this repeatedly
and once we are done doing these
transformations, we just pipe it through
to our good old logistic regression
sigmoid here and we are done.
Okay?
So, this is the basic idea.
And so, just to contrast it, this was
good old logistic regression where we
take the input,
we run it through a linear function and
pop out a number,
a probability number. But, after we do
all this stuff, the input stays the
same, the output stays the same, but in
the middle you just run through a whole
bunch of these functions, you know,
these layers, boop boop boop boop, and
then we get the output.
Okay?
That's all we have done.
And this is a neural network.
A neural network is nothing more than
repeatedly transformed inputs which are
finally fed to a linear or logistic
regression model.
Any questions?
I have two questions. Could you use the
thing so that everyone can hear? Yeah.
I have two questions. Firstly, so when
we say that there isn't chance of
explainability, is it that we don't know
which arrow it went through? That's one.
Second,
who's controlling the number of
iterations or the number of functions?
That's up to us or how does that work?
Right. So, yeah, so the the first
question, um explainability, we actually
know exactly for any given input input
uh data data point, we know exactly how
it flows through the network. So, there
is no problem there.
The problem is in ascribing, "Okay, this
we we think this person is going to be
uh repay the loan because
of this particular attribute." We don't
know that because those attributes all
get enmeshed together and goes through
this complicated thing. So, we know
exactly what happens. We just can't give
credit to anyone thing very easily.
I'm again, I'm just standing on the
brink of this vast ocean of something
called explainability and
interpretability, uh which I'll get to a
bit later on in the semester. But,
that's sort of the quick
kind of right-ish kind of wrong answer.
Okay? Number two, um
uh
we decide the number of layers. We
decide a whole bunch of things and as
we'll see in a few minutes, uh there is
something that's given to us and
something we get to design and I'll make
it very clear which is which.
Yeah.
Did I say your name right? Yeah.
So, which functions have to be linear
and also like why does it have to be
linear? Yeah. So, these functions uh the
f of x here, they have to be non-linear.
As to why they have to be non-linear,
we'll get to that in a few minutes.
Okay. So, these are called neurons.
Okay?
These things where you basically there's
a linear function followed by uh a
little non-linear function,
right? This is a Each one of these
things is called a neuron.
Um
By the way, you know, this is loosely
inspired by the way how, you know, uh
neurons work in a human in mammalian
brains.
But, the connections between
neuroscience and deep learning
are very heavily argued.
So, I'm going to like stay away from it.
Okay? Uh suffice it to say it's I I just
think of For for building practical deep
learning systems in industry, you don't
you don't worry about this. Okay?
All right, let's move on.
Terminology. Uh this vertical stack of
linear functions or neurons,
right? This vertical stack is called a
layer.
Right? This is a layer, that's a layer.
Uh and these little non-linear
functions, which we haven't gotten to
yet, are called activation functions.
Uh and we'll get to why they are called
that in just a second.
And
the input
is called an input layer and I have the
word layer in double quotes because like
it's not really doing anything, right?
It's just the input.
So, but we call it an input layer.
And what the very final thing that
produces outputs is called the output
layer, right? Obviously. And everything
in the middle is called a hidden layer.
Okay?
So, the final piece of terminology is
that when you have a layer like this in
which say three numbers are coming out
and there's another another layer,
right? If every neuron in this layer is
connected to every neuron in this layer,
it's called a fully connected or dense
layer. So, for instance, here
this arrow that's
whatever the whatever number is coming
out. Let's say the number three is
coming out of this thing here. That
number three goes flows on this arrow to
this thing, flows on this arrow to this
neuron, and flows on this third arrow to
this neuron. That's what I mean. So,
every neuron, its output is being sent
to every neuron in the following layer.
Okay? That's we call it fully connected
or dense.
And then
if you look at logistic regression,
right? This is logistic regression. You
can see basically logistic regression is
a neural network with no hidden layers.
So, in some sense, logistic regression
is like almost the simplest possible
network you can think of.
Like barely a neural network.
Right? It's got no no hidden layers.
That's what makes it logistic
regression.
And so, as you might have guessed by
now, deep learning is just neural
networks with lots and lots of
of what?
Yes, layers.
So, here are a few.
Uh and by the way, these are not even
considered all that, you know,
impressive these days.
Okay? Uh but I put them up because this
this thing here is called ResNet.
And it's famous because the ResNet
neural network was I think the first
network
to surpass human-level performance in
image classification.
Sort of it it's sort of like the Skynet
of image classification. Okay? It
surpassed human-level performance. And
I'm putting it up here because we'll
actually work with ResNet on next next
Wednesday. And we'll actually take
ResNet, we'll fine-tune it, and solve a
real problem in class.
All right. So, it's got lots and lots of
layers. Uh now, let's turn to these
activation functions. We've been
ignoring these little guys, right? So
far.
So, the activation function at a node is
a first of all, it's a function that
receives a single number and outputs a
single number, right? It's not very
complicated, right? It receives
basically this this is a linear function
which receives all these inputs. It
could be 10 inputs, 1,000 inputs,
runs it through a linear function,
outputs a number, and that single
number, a scalar, goes in here, and it
comes out as another single number.
Just just just remember that.
And so, these are some of the most
common activation functions. In fact,
the sigmoid we saw, which is actually we
use for the output, is actually a kind
of activation function where a single
number comes in and it gets mapped into
this curve because of this thing. So,
the single number that comes in is A,
and it and it gets transformed as 1 / 1
+ e ^ -A, and you get a shape like this,
and it's called the sigmoid activation
function. And And And as you can see
here,
for very small values, for very negative
values,
it's going to be pretty close to zero,
meaning it won't get activated.
And for very very large values, it's
going to be
pretty close to one.
All the action happens in the middle.
When your When your When your values are
somewhere in this range, there's a
dramatic increases in what comes out.
Okay? So, that little thing in the
middle is a sweet spot for these
functions.
Uh
and this
I you know, I'm also almost embarrassed
to call it an activation function
because it's literally not doing
anything. It's sort of getting a nice
label for free.
Um right? You basically it says you just
get a number, just pass it straight
along.
It's a linear activation function, but
just for completeness, I want to put it
here.
And then we come to the hero of deep
learning, which is the rectified linear
unit,
right? Rectified linear unit. It's
called ReLU. Uh and ReLU is going to
become part of your vocabulary very very
quickly. Uh and so, ReLU is actually a
very interesting function. So, you write
it as maximum of whatever number and
zero,
which is another way of saying if the
number is positive, just send it along
unchanged. If the number is negative,
send a zero instead. Squish it to zero.
So, which means if the number is
negative, nothing happens. If the number
is positive, it wakes up.
So, what happens is that you could have
a very complicated linear function with
millions of variables, and then it puts
a single number, and that number
unfortunately happens to be negative.
The ReLU is not impressed. It's going to
send a zero out.
Okay? It's a very simple function.
And many many folks who've been in deep
learning for a long long time believe
that
the use of the ReLUs is one of the key
factors
that led to the amazing success of deep
learning because it's got some very
interesting properties,
uh which we'll get to hopefully on
Wednesday.
Okay. So, the shorthand here is that um
whenever you see this thing, it's just a
linear activation, linear function
followed by just sending it straight
out. If I If you do this this If I put a
ReLU in here, I'm going to denote it
like that, which mimics the graph
uh how it looks. And if I'm going If I
put a sigmoid, I'm just going to use
this thing here.
Okay?
Just a visual shorthand.
>> [clears throat]
>> There are many other functions
activation functions, by the way.
There's something called the tan h
function, the leaky ReLU, the GELU, the
Swish. I mean, it's like a menagerie of
activation functions because very often
researchers will be like, "Well, I don't
like this activation function. Here's a
little modified version of the function
which is going to be better for certain
things." So, you know, people's research
creativity is sort of on this point has
gone unhinged. Um so, there's lots of
options. But if you just stick to the
ReLU
for your hidden layers, you can
basically get anything done practically,
right? You don't have to worry about
anything else. So, we'll only focus on
ReLUs for all the intermediate stuff. Uh
yeah.
Yeah, how do you gauge which activation
function is more suited for your use
case?
Yeah. So, the rule of thumb here is that
for your hidden layers, use ReLUs,
right? Because empirically we have seen
that they they do an amazing job.
For your output layer, your very final
thing, you actually don't have a choice
because what you have to use depends on
what kind of output you have to work
with. If it's an output which is a
probability number between zero and one,
you have to use a sigmoid.
Um if it is
say 10 numbers, all of which have to be
probabilities, and they have to add up
to one,
you got to use something called the
softmax, which we'll get to on
Wednesday. So, it really depends on the
output, and the nature of the output
dictates what you use in the output
layer.
Okay.
So, coming back to this. So, if you want
to design a deep neural network,
uh the input is the input.
The output is the output. And so, you
get to choose everything else. You get
to choose the number of hidden layers,
the number of neurons in each layer, the
activation functions you're going to use
and uh for the hidden layers, and then
you have to make sure that the what you
choose for the output layer matches the
kind of output you want to generate.
Okay? So, this is this sort of This is
all in your hands. You decide what
happens. But
you will there there's a lot of guidance
for how to do these things, which we'll
which we'll cover as we go along.
Did you have a question?
Kind of, but I guess I'll do it.
Is Is there also exploration in kind of
dynamic uh
setting up layers so that your users
determine the number of layers
Yeah. So, there's a whole field called
neural architecture search, NAS,
where we can actually try a whole bunch
of different architectures,
uh and then use some optimization and in
fact reinforcement learning, which we
won't get to in this class,
as a way to figure out really good
architectures for any particular
problem. Uh but the
the question of okay,
when I'm training a model with a
particular kind of data,
the first pass through the training
data, I'm going to use two layers. The
second pass, I'm going to do seven
layers. That is not done.
Uh and the reason it's not done is
because of certain other constraints we
have in how we can do the the
optimization and the gradient descent
and stuff like that. But what you can
do, and we will we'll look at this thing
called dropout,
for certain layers, you can actually for
each time you run it through the
network, you can decide in this layer
I'm not going to use all the nodes. I'm
going to drop out a few of the nodes
randomly. And it's a very effective
technique to prevent overfitting, and
we'll come to that a little later on.
Uh yeah.
So, one question regarding like
neural networks is about the
coefficients. Is this something we
decide
or we
have to use as a defined coefficient for
the weights? No, the whole trick here
the whole name of the game is we use the
data, the training data, and something
called a loss function, which I'll get
to on Wednesday,
along with an optimization algorithm, so
that the network figures out by itself
what the weights need to be, what the
coefficients need to be, so as to
minimize prediction error.
And that's the whole thing. The magic
here is that we don't have to do
anything. We only have to set it up, sit
back, often for many hours, and watch it
do its thing.
Yeah.
Just one quick question. Um you
mentioned nodes just now when you were
answering Roland's question. Can you
just confirm exactly what a node is? I
have an idea that it's basically any
circle, but
>> Yeah, yeah. you just added a lot more
detail. Sure. No, when when I'm
referring to a node, I'm literally
referring to something like this, which
think of it as a linear function
followed by a non-linear activation.
So, it it reads a bunch of inputs, runs
it through a linear function, and pass
it through like a ReLU or a sigmoid or
something, and out pops a number.
So, in general, a node will have
many numbers potentially coming in, but
only one number going out.
Uh now, that one number may get copied
to every node in the next layer,
but what comes out of that particular
node is just a single number.
All right. So,
uh
So, let's use a DNN for our interview
example. So, in this problem we had two
inputs, right? GPA and experience. The
output variable has to be between zero
and one because you're trying to predict
the probability that someone will get
called for an interview. So, the output
size is fixed the
sorry, the input size is fixed the
output is fixed. Uh
and we so, since it's really the only
the very first network we're actually
playing with uh
let's just start simple, right? We'll
just have one hidden layer and we'll
have three neurons, right? And and as I
mentioned to Tommaso's question from
before if you are choosing activation
functions in the hidden layers, just go
with the ReLU as a default. It usually
works really well out of the box. So,
we'll just use a ReLU and since the
output has to be between zero and one,
we don't have a choice. We have to use a
sigmoid for the output layer.
Okay? That's it. So, we have the those
are the design choices and when we do
that, this is how it's looked like,
right? We have two inputs X1 and X2, GPA
and experience and then it goes through
these three
ReLUs and then out comes these three
numbers and they pass through a sigmoid
and we get a probability Y at the end.
All right, quick question. Concept
check.
How many weights
how many parameters, both weights and
biases does this network have?
Let's take a moment to count.
All right, any guesses?
Yeah.
12.
I think you're almost there.
Um
our folks going to be doing a binary
search on this now? Okay.
Uh no.
Yes? 30. Yes, very good.
So, that's 30
and my guess is that the reason you came
up with 12 and I made the same mistake,
that's why I know it is you probably
forgot this green thing here.
Um so, so the what folks often forget is
the bias.
Right? We all count the things, right?
Okay. And the easy way to do it is okay,
two things here,
three things here, two times six three
is six,
three times one is three another nine
and then you have to add up all the
intercepts.
Right? So, you get 30.
And so, when we get to very complicated
networks the the first two or three
times you work with very complex
networks
and we'll do it, you know, starting very
soon, just get into the habit of hand
calculating the number of parameters
just to make sure you understand what's
going on. Once you get it right a couple
of times, you can you don't have to do
it anymore. Okay? The first couple of
times hand calculate to make sure you
get it.
Okay. So, yeah. So, let's say that we
have trained this network using, you
know, using techniques which we'll cover
on Wednesday and it is it comes back to
you after training and says, "Okay,
these are the optimal the best values
for the weights and the biases that I
have found." So, now your network is
ready for action.
It's ready to be used
and so, so what you can do is let's say
that you want to predict with this
network,
you know,
if you have X1 and X2, what comes out of
what So, what comes out of this top
neuron, right? Let's call it A1. It's
basically this.
Okay? That's what's coming out of this
thing. For any X1 and X2, this is what's
coming out. Similarly for A2 and A3
Okay?
And then what comes out at the very end
is
basically A1 times that plus A2 times
that plus A3 times that plus 0.05 and
the whole thing gets run through the
sigmoid and this is what you get.
Okay? So, this slide and the one before,
just make sure you look at it afterwards
and to make sure you totally understand
the mechanics of it because
this is really important. If you don't
If you don't fully understand like
internalize the mechanics, when we get
to things like transformers, it's going
to get hard. Okay? So, just make sure
it's like automatic at this point. It
should be reflexive.
Um
Okay. So, yeah. And so, when we when you
want to predict anything, you just run
some numbers through it, you get all
these things
and boom, you calculate it. It turns out
to be 22.6. That's the answer.
All right. So,
I just want to say that let's say that
you built this network
and now we are like, "Hey,
given any X1 and X2, I can come up with
a Y."
But I'm feeling a little mathy. Can we
actually write down the function? Yeah,
you can write down the function. This is
what it looks like.
Super interpretable, right?
So, this goes to the comment that Itai
you made earlier on where the act of
depicting something using this sort of
graphical layout makes it so much easier
to reason with
and to think about compared to trying to
figure out what this function is doing.
Right? The other point I want to make is
that um
just contrast what we just saw with the
logistic regression thing we saw
earlier, which was this little function
and so, here
even this simple network with just three
hidden layers the sorry, three nodes in
that single hidden layer
right? It's so much more complicated
than the logistic regression model. So
much more complicated, right?
And it is from this complexity
springs the ability of these networks to
do basically magical things.
Right? That's where the complexity comes
from. That's where the magic comes from.
So, and here in this case, the number of
variables hasn't even changed. It's
still only two.
But we can go from the two inputs to the
one output in very complicated ways as
long as we know how to train these
networks the right way. That's sort of
the
the secret sauce which we'll spend a lot
of time on.
So, yeah. To summarize, this is what we
have. It's a deep neural network.
By the way, this kind of network where
things just flow from left to right is
called a feedforward
neural network
in contrast to some other kinds of
networks called recurrent networks which
you won't get to
in this class because
transformers have actually proven to be
much more capable than recurrent
networks and those have become the norm,
so we'll just focus on those instead. Um
and so, this arrangement of neurons into
layers and activation functions and all
that stuff, this called the architecture
of the neural network. And as you will
see later on, the transformer, the
famous transformer network
[clears throat] is just an example of a
particular neural network architecture
much like convolutional neural networks
which will get to next week for computer
vision or another example of a
particular network of of architecture.
So, we will focus on transformers. They
are a particular kind of architecture.
All right. So, in summary, this is what
we have.
You know, you get to choose the hidden
layers, the neurons, activation
functions, stuff like that.
The inputs and outputs are what you have
to work with and so, we will actually
take this idea and then use it
to
to actually solve a problem from start
to finish on Wednesday. So, I think I'm
done. I give you three minutes back of
your day. Thank you.
>> [applause]