This is CS231n.
And I'm Professor Fei-Fei Li from computer science
department.
I will be co-teaching this quarter
with Professor Ehsan Adeli and my graduate student Zane.
So you'll meet them as well as our wonderful TA team
that you will meet later.
So I just want to get started.
So this is what excites me, that AI
has become such an interdisciplinary field,
that what you're going to learn in this class,
of course, is very technical.
It's about computer vision and deep learning.
But I really do hope that you take
it to whichever discipline you work in and are passionate about
and apply it.
So we hear a lot about the field of AI.
So how do we position computer vision
and the scope of this class?
If you consider AI as this big bubble,
computer vision is very much an integral part of AI.
Some of you have heard me saying that not only vision is
part of intelligence, it's a cornerstone to intelligence.
Unlocking the mystery of visual intelligence
is unlocking the mystery of intelligence.
But one of the most important tools, mathematical tools,
to solving AI is machine learning or some people
call statistical machine learning.
And this is exactly what we will be talking about.
Within the field of machine learning,
in the past 10 plus years, we have seen a major revolution
called deep learning.
And I'll explain a little bit of what deep learning is.
Deep learning is a set of algorithmic techniques
that is built around a family of algorithms
called neural networks.
And so if you ask me to pinpoint the scope of this class,
we'll not be able to cover the entirety of computer vision.
We'll not be able to cover the entirety of machine
learning or deep learning.
But we're going to cover the core intersection of these two
fields.
And of course, just like the entirety of AI,
computer vision is becoming more and more
an interdisciplinary field.
A lot of the techniques we use as well as
the problems we work with intersect
with many different other fields, like natural language
processing, speech recognition, robotics, and AI as a whole
is a field that intersects with mathematics, neuroscience,
computer science, psychology, physics, biology,
and many application areas from medicine
to law to education and business and so on.
So what you will get for this lecture, our first lecture,
is I'll give a very brief history of computer
vision and deep learning.
And then Professor Adeli will go over the overview of this course
and lay the groundwork of how this course is set up
and what our expectations are.
So the history of vision did not begin when you were born
or humanity was born.
The history of vision began 540 million years ago.
You might ask, what happened 540 million years ago?
Why are we pinpointing a relatively specific date or year
in evolution.
Well, it's because a lot of fossil studies
have shown us that there is a mystery period called Cambrian
explosion.
The fossil studies showed about 10 million years in evolution
during that time, which is a very short period of time
for evolution.
We see the explosion of animal species
in the fossil study, which means before the Cambrian explosion,
life on Earth was pretty chill.
It was actually in the water.
There's no animals on the land yet.
And animals just float around.
So what caused this explosion in animal speciation?
There were many theories, from climate to chemical composition
of the ocean water.
But one of the most compelling theories was the onset of ice.
The first animal, a trilobite, they
gained photosensitive cells.
So the eyes we were talking about
were not sophisticated lenses and retinas and nerve cells.
It was literally a very simple pinhole.
And that pinhole collected light.
Once you collected light, life is completely different.
Without sensors, life is metabolism.
It's very passive.
It is just metabolism.
And you come and go.
With sensors, you become an integral part of the environment
that you might want to change.
You might want to actually survive in it.
Some animals or plants become your dinner.
And you become someone else's dinner.
So evolutionary forces drives intelligence
to evolve because of the onset of sensors,
because of the onset of vision, along with haptics
or tactile sensing.
Those are the oldest sensors for animals.
So that entire course of 540 million years
of evolution of vision is the evolution of intelligence.
Vision as one of the primary senses of animals
drove the development of nervous system, the development
of intelligence.
Almost all animals on Earth today we know of
have vision or use vision as one of the primary senses.
Humans are especially visual animals.
More than half of our cortical cells
are involved in visual processing.
And we have a very complex and convoluted visual system.
So this is what excites me to enter the field of vision.
And I hope it excites you.
So now, let's just fast forward from Cambrian explosion
to actually human civilization.
Humans do innovate.
And not only we see.
We want to build machines that see.
So here's a couple of drawings by, of course,
Leonardo da Vinci, who was just forever curious
about everything.
He studied camera obscura for how to make steam machines.
In fact, even way before him, in ancient Greece
and in ancient China, we have seen documents
about thinkers, philosophers thinking
about how to project objects through pinholes
and to create images of objects.
And of course, in our modern life,
cameras have truly exploded.
But cameras are not enough for seeing, just like eyes are not
enough for seeing.
These are apparatus.
We need to understand how visual intelligence happens.
And that's really the crux of this course.
So let's just talk a little bit of the history that brought us
to this intersection of deep learning and computer vision.
So let me go back to the 1950s.
The 1950s-- a set of very critically important experiments
happened in neuroscience.
And that was the study of the visual pathways
of mammals, especially the seminal work
by Hubel and Wiesel.
They used electrodes to put into live cats anesthetized.
And then they studied the receptive field
of neurons that are in the primary visual cortex.
What they have learned, to their surprise,
are two very important things.
One is that neurons that are responsible for seeing
in the primary visual cortex have
their own individual receptive fields.
Receptive fields means that for every neuron,
there is a part of space it actually sees.
It's not all the space.
It's not very big.
It tends to be a very confined patch of the space.
And within that space, it sees specialized patterns,
simple patterns, when you're measuring from the early part
of the visual pathway.
And by and large, in the primary visual cortex,
which is around here in the back of the head, not near your eyes,
it's oriented edges or moving oriented edges.
So every neuron, some neuron will
be seeing an edge like this.
Some will be seeing an edge like this or this.
And that's how the computation in the brain begins.
The second thing they learned is that visual pathway
is hierarchical.
As you move beyond the visual pathway,
the neurons feed into other neurons.
And the neurons in the higher layers
or deeper layers of the visual hierarchy
have more complex receptive fields.
So if you begin with oriented edges,
you might feed into a corner receptor.
You might feed into an object receptor.
I'm overly simplifying.
But that's the concept, is that neurons feed into each other.
And then they create this big network of computation.
Of course, most of you sitting here
are already thinking the way I've
been describing this will have a profound impact
on the neural network modeling of visual algorithms.
Let's keep going.
That's year 1959.
It's very early studies of seeing.
By the way, about 30 years later--
maybe not quite-- 20 something years later,
Hubel and Wiesel won the Nobel Prize in medicine
for studying this, uncovering the principles
of visual processing.
Another milestone in the early history of computer vision
was the first PhD thesis of computer vision.
Most people attribute Larry Roberts in 1963
writing the first PhD thesis just studying shape.
And this is a very, very character representation
of the world.
And the idea is that, can we take a shape like this
and understand that the surfaces and the corners and features
of this shape?
It's intuitive that humans do.
So an entire PhD thesis is devoted to this.
And that's the beginning of computer vision.
And around that time, in 1966, an MIT professor
created a summer project in MIT and asked
to hire a few undergrads, very smart ones, to study vision.
And the goal was pretty much solve computer vision
or solve vision for one summer.
Of course, just like the rest of the history of AI,
we tend to be overoptimistic of what we can
do in a short period of time.
So vision did not get solved in that summer.
In fact, it has blossomed into an incredible computer science
field.
If you go to our annual conferences every year now,
it has more than 10,000 people attending.
But 1960s is where, between Larry Roberts PhD thesis as well
as this kind of project, we in our field considered that
the beginning of the field of computer vision.
A seminal book was written in the 1970s by David Marr,
who unfortunately died too early.
He wanted to study vision systematically and start
to consider how visual processing happens.
Even though this is not explicitly
stated, but there is a lot of inspiration
from neuroscience and cognitive science.
He was thinking about, if you take an input image,
how do we visually process and understand the image?
Maybe the first layer is more like edges, just like we saw.
He calls it primal sketch.
And then there is a 2 and 1/2 D sketch which separates different
depth of the objects in the image.
So the ball is the foreground object.
And then the grass here--
oh, no, not grass.
The ground here is the background.
So he does these 2 and 1/2 D sketch.
And then, finally, David Marr believes the grand holy grail
victory of solving vision is to know the entire full 3D
representation.
And that is actually the hardest thing of vision.
Let me digress for 20 seconds.
Because if you think about vision for all animals,
it's an ill posed problem.
Since the early trilobites who collected light
from underwater, light--
the world through photons is projected on something
on a surface more or less 2D.
At that time, it was just, I don't know, some patch
in the animal.
But right now, for us, it's a retina.
But the actual world is 3D.
So recovering 3D information, the entire 3D world,
from 2D images is the fundamental problem nature had
to solve and computer vision has to solve.
And mathematically, that's an ill-posed problem.
So what did we later do?
Anybody have a wild guess?
[INAUDIBLE]
Yes.
The trick that nature did is develop multiple eyes, mostly
two.
Some animals have more than two.
And then you triangulate information.
But two eyes are not enough.
You actually have to understand correspondences and all that.
We'll touch on some of these topics.
But there are other computer vision classes taht Stanford
offers that also specifically talk about 3D vision.
But the point is it's a very hard problem.
And we have to solve it.
Nature has solved it.
Humans have solved it but not to extreme precision.
In fact, humans are not that precise.
I roughly know the 3D shapes.
But I don't have geometric precision of all the shapes.
So that's one thing to consider and appreciate
how hard this problem is.
Another thing that is very different for computer vision
and language is actually something
philosophically subtle.
Language doesn't exist in nature.
You cannot point to something and say there is language.
Language is a purely generated thing.
I don't even know what word to use.
It comes through our brain.
It's generated.
It's 1D.
It's sequential.
So this actually has profound implications in the latest
wave of GenAI algorithms.
This is why these LLMs, which is outside
of the scope of this class, is so powerful because we
can model language that way.
But vision is not generated.
There is actually a physical world
out there respecting the laws of physics and materials and all
that.
So vision has very different tasks.
So I just want you to appreciate the difference between language
and vision and actually, frankly, appreciate nature,
how it solved this problem.
Let's keep going.
1970s, the early pioneers of computer vision, without data,
without really much of powerful computers,
without the mathematical advances we have seen today,
are already beginning to solve some of the harder problems
of computer vision-- for example, recognition of objects.
Here in Stanford, one of the pioneering work
is called generalized cylinders by Rodney Brooks and Tom
Binford.
And ironically, Rodney Brooks today is on campus, actually,
over there giving a talk at the robotics conference.
And he went on to become one of the greatest
roboticists of our time and was founder of Roomba
and many other robots.
And then not very far from us in another part of Palo Alto,
researchers have worked on these also compositional models
of human body and objects.
And then in the 1980s, digital photos start to appear.
At least photos start to appear.
And people can digitize that a little bit.
And then there are some great work in edge detection.
You look at all this and probably feel
a sense of disappointment.
I mean, it's kind of trivial to get some sketches and edges.
And it's not really going anywhere.
That's how computer vision, works at that time.
And in fact, you're not so wrong.
That was around the time before many of you
were born that we entered AI winter.
The field entered AI winter because the enthusiasm
and, hence, funding for AI research has really dwindled.
A lot of things didn't deliver.
Computer vision didn't deliver.
Expert systems didn't deliver.
Robotics didn't deliver.
But under the hood of this winter, a lot of research
start to grow from different fields,
like computer vision, NLP, robotics.
So let's also look at another strand of research
that had a profound implication in computer vision,
is that cognitive and neuroscience
continue to blossom.
And what is really important, especially
for the field of computer vision, is cognitive
and neuroscience is starting to point to as the North Star
problems we should work on.
For example, psychologists have told us
there's something special about seeing nature,
seeing real world.
This is a study by Irv Biederman, who
shows that the detection of bicycles on two images
differ depending on if the images are scrambled or not.
Think about it.
From a phton point of view, these two bicycles
land in the same location on your retina.
But somehow the rest of the image
impacts the viewer, seeing the target objects.
So there is something telling us that seeing
the entire forest or the entire world
impacts the way we see objects.
It also tells us visual processing is very fast.
Here's another direct measure of how fast we detect objects.
This is an early 1970s experiment showing people
a video.
And the test for the subject is to detect the human
in one of the frames.
I suppose every one of you have seen that human in one
of the frames.
But think about how remarkable your eyes are
or your brain is because you've never seen this video.
I didn't tell you which frame that the target object would
appear.
I did not tell you what the target
object will look like, where it is, its gestures, and all that.
Yet, you have no problem detecting the humans.
And on top of that, these frames are
played at 10 Hertz, which means you're
seeing every frame for only 100 milliseconds.
And this is how remarkable our visual system is.
In fact, Simon Thorpe, another cognitive neuroscientist,
have measured the speed.
If you hook people up in EEG caps
and show them complex natural scenes
and ask human subjects to categorize things
from animals without--
versus things without animals--
hundreds of them.
And then you measure the brain wave.
It turned out, after 150 milliseconds of seeing a photo,
your brain already has a differential signal
that categorizes.
You might not be so impressed.
Because compared to today's GPUs and modern chips,
150 milliseconds is really orders of magnitude slower.
But you got to admire.
Our wetware, our brain, our neurons
don't work as fast as transistors.
150 milliseconds is actually really fast.
It's only a few hops in the brain
in terms of neural processing.
So yet, again, this is telling us
humans are really good at seeing objects and categorizing them.
In fact, not only we're so good at seeing objects
and categorizing them, we even develop specialized brain
areas that have expert ability in recognizing
faces or places or body parts.
And these are discoveries by MIT neurophysiologist in the 1990s
and early 21st century.
So all these studies tell us, well, we should not just
be studying these kind of character shapes
or the sketches of images.
We really should go after important fundamental problems
that drives visual intelligence.
And one of those problems that everything
has been telling us is object recognition--
is object recognition in natural setting.
There is a lot of objects out there in the world.
And studying this is going to be part
of the unlocking of visual intelligence.
And that's what we did.
As a field, we started by looking
at how we can separate foreground objects
from background objects.
This is called recognition by grouping in the 1990s.
Keep in mind, we're still in AI winter.
But research is actually happening and progressing.
And then there is studies of features.
And some of you might still remember
sift features and matching.
And when I enter grad school, the most exciting thing
was face detection.
I remembered that first year in my grad school,
this paper was published.
And five years later, the first digital camera
used this paper's algorithm and delivered automatic face focus
because of face detection.
So things started to work and taken into industry.
And then around the early 21st century,
a very important thing started to happen,
is internet started to happen.
When internet started to happen, data started to proliferate.
And the combination of digital cameras and internet
started to give the field of computer vision
some data to work with.
So in that early days, we're working with thousands of images
or tens of thousands of images to study the visual recognition
problem or the object recognition problem.
So you've got data sets like Pascal Visual Object
Challenge or Caltech 101.
I'm going to pause here.
And this is where the first thread of computer vision
start to progress.
And you might be wondering, why is she pausing?
Because I'm going to come back and talk about deep learning.
So while this field of vision was progressing
through neurophysiology to computer vision,
to cognitive neuroscience, to computer vision again,
a separate effort is going on in parallel.
And that eventually became deep learning.
It started from these early studies of neural network,
things like perceptron.
And people like Rumelhart started to work.
And of course, Jeff Hinton in his early days,
started to work with a small number of artificial neurons
and look at how that can process information and learn.
And you've heard people like the great minds like Marvin Minsky
and his colleagues working on different aspects
of this perception.
But Marvin Minsky did say that perceptrons cannot learn these
XOR logic functions.
And that caused a little bit of a setback in neural network.
Well, things continued to progress despite the setback.
And one of the most important work before the first inflection
point is this neocognitron work by Fukushima in Japan.
Fukushima hand-designed a neural network that looks like this.
So it has about five or six layers.
And then he kind of designed the different functions
across the layers, which you will
learn more, that more or less was
inspired by the visual pathway that I was describing.
Remember the cat experiment from simple receptive field
to more complicated receptive field.
And he was doing that here.
The early layers have simple functions.
And then the later lighter layers
have more complex functions.
And the simple ones can call it convolution.
Or he uses the convolution function.
And the more complex one, he was pulling the information
from the convolution layers.
So neocognitron was really an engineering feat
because every parameter was hand-designed.
There are hundreds of parameters.
He has to just meticulously put them together
so that this small neural network can
recognize digits or letters.
So the real breakthrough came around that time in 1986
is a learning rule.
That learning rule is called backpropagation.
It's going to be one of our first classes
to show you that Rumelhart, Jeff Hinton--
they took neural network architecture
and introduced an error correcting objective function
so that if you put in some input and know
what the correct output is, how do you
take the difference between what the neural network outputs
versus the actual correct answer and then
propagate the information back so that you
can improve the parameters along the neural network?
And that propagation from the output
back to the entire neural network
is called backpropagation.
It follows some of these basic calculus chain rules.
And that was a watershed moment for neural network algorithm.
And of course, we're still smack in the middle of AI winter.
All these work was happening without public fanfare.
But of course, in the world of research,
these are very important milestones.
One of the most earliest applications of this neural
network with backpropagation is Yann LeCun's convolutional
neural network, made in the 1990s when he was working
in the Bell Labs.
And what he did is just created a slightly bigger network,
about seven layers-ish, and made it good enough
with great engineering capability to recognize letters.
And it was actually shipped to some part of the US Postal
Offices and banks to read digits and letters.
So that was an application of early neural network.
And then Jeff Hinton and Yann LeCun
continued to work on neural network.
It didn't go very far.
Because despite these improvements and tweaks
of these neural network, things more or less just stalled.
They collected a big data set of digits and letters.
And digits and letters kind of was quasi soft
in terms of recognition.
But if you put the system through the kind
of digital photos that the neuroscientists were using
to recognize cats and dogs and microwaves and chairs
and flowers, it just didn't work.
And a huge part of this problem is the lack of data.
And lack of data is not just an inconvenience.
It's actually a mathematical problem
because these algorithms are high capacity algorithms that
actually needs to be driven by lots of data
in order to learn to generalize.
And there is some deep mathematical principles
behind these rules of generalization and model
overfitting.
And data was underappreciated, was
underlooked because most people are just
looking at these architectures.
They did not realize that data is
part of the first class citizen for machine
learning and deep learning.
So this is part of the work that my students and I did
in the early 2000s, that we recognize this importance
of data.
We hypothesized that the whole field was actually
missing this-- underappreciating the importance of data.
So we went about and collected a huge data
set called ImageNet that has 50 million images
after cleaning a billion images.
And these 15 million images were sorted across 22,000 categories
of objects.
We actually studied a lot of the cognitive and psychology
literature to appreciate that 22,000 images were--
sorry, 22,000 categories were roughly in the order
of the number of categories that humans learned to recognize
in the early years of their life.
And then we open sourced this data
set and created an ImageNet challenge called the Large Scale
Visual Recognition Challenge.
We curated a subset of ImageNet of a million images or a million
plus images and 1,000 object classes and then ran
an international object recognition challenge for many
years.
And the goal is that we ask researchers to participate.
And their goal is to create algorithms.
It doesn't matter which kind of algorithms.
And they will test you on your algorithm's ability to recognize
photos and see if you can call out these 1,000 object classes
as correctly as possible.
And here are the errors.
First year we run this competition,
the best performing algorithms error was nearly 30%.
And it's really pretty abysmal because humans can perform
under like, say, 3% error.
And then 2011, it wasn't that exciting.
But something happened in 2012.
That was the most exciting year.
That year, Jeff Hinton and his students
participated in this challenge using
convolutional neural network.
And they reduced the error almost by half.
And it truly showed the power of deep learning algorithms.
And so the participating algorithm in 2012 ImageNet
challenge was called AlexNet.
And the funny thing is, if you look at AlexNet,
it's not that different from Fukushima's neocognitron
32 years ago.
But two major things happened between these two.
One is that backpropagation happened.
It's a principled, mathematically rigorous learning
rule so that you don't have to ever use hand
to tune parameters.
And that was a major breakthrough theoretically.
Another breakthrough was data.
The recognition of data and the understanding of data driving
these high capacity models, which eventually will have
trillion parameters-- but at that time was millions
of parameters-- was critical for setting off the deep learning
for this to work.
And really, many people consider the year of 2012
and the AlexNet algorithm that won the ImageNet
the challenge the historical moment of the birth
or rebirth of modern AI or the birth of deep learning
revolution.
And of course, the reason many of you are here
is since then, we are in the era of deep learning explosion.
If you look at computer vision, some main annual research
conference, called CVPR--
the number of papers have exploded.
And our arXiv paper has exploded.
And many new algorithms since then
have been invented to participate in the ImageNet
challenge.
In the following years, we're going
to study some of these algorithms.
But the point is that some of these
algorithms beyond Alex that have had a profound impact
in the progress of the field of computer vision
and into the applications of computer vision.
So a lot of things have happened.
We're going to cover some of these.
Not only the field of computer vision
made a major progress in creating algorithms
to recognize everyday m like cats and dogs and chairs--
we also quickly, right after ImageNet challenge,
the 2012 moment, we've got algorithms
that can recognize much more complicated images,
can retrieve images, or can do multiple object detections,
can do image segmentation.
These are all different tasks in visual recognition
that you'll find yourself getting
familiar with throughout this course
because vision is not just calling out cats and dogs.
There is so much in the nuanced ability of visual recognition.
And of course, vision is not just static images.
So there are work in video classification, human activity
recognition.
I'm showing you this overview.
You will learn some of these.
You don't have to understand exactly what's going on here.
But I want you to appreciate the variety of vision tasks.
Medical imaging, those of you who come from a medical field,
whether it's radiology or pathology or even
other aspects of medicine, is deeply visual.
And this has a profound impact.
Scientific discovery-- even the seminal picture
you probably remember of the first photography of black hole
uses a lot of computer vision and computational photography
techniques.
Of course, applications in sustainability and environment
is $also computer vision contributed a lot of that.
And we also have made a lot of progress
in image captioning right after the image-- that 2012 moment.
This is actually work by Andrej Karpathy, where he was
my student, his thesis work.
Then we also worked on relationship understanding.
So not only visual intelligence is
about seeing what's on the pixel,
you can also see what's beyond pixels,
including relationships of objects and also style transfer.
A Lot of this work, you will-- actually,
Justin Johnson, who will come to guest lecture this course,
will tell you all about his seminal work in style transfer.
And of course, in generative AI eras,
we get these really incredible results like face generation.
And this is the very early days of image generation of Dall-E. I
think this is the early Dall-E. Of course, now, Midjourney
and everything has gone beyond these avocado and peach chairs.
But really, we are squarely in the most exciting modern era
of AI explosion.
The three converging forces of computation, algorithms,
and data have taken this field just
to a whole different level, where we're now
totally out of AI winter.
I would say we're in an AI global warming period.
And I don't see any of this slowing down
for both good and bad reasons.
And also, just a word, because we are in the Silicon Valley,
we're in the very building of Huang building and NVIDIA
lecture hall-- so we cannot ignore also the progress
of hardware and what that played.
So here is just the FLOP per dollar graph for NVIDIA's GPUs.
And before 2020, the progress was steady.
But as soon as deep learning started
to drive these GPUs and chips, you
can just see the GFLOPS have just completely taken off.
And by any measure, we are in this accelerated curve
of lots of compute as well as lots of AI.
And these are just different graphs
showing you conference attendees, startups,
and enterprise applications in AI all across
not just computer vision.
But also, NLP and others have just exploded.
So quickly, last but not the least, it's been exciting.
There has been a lot of successes.
But there is still a lot to be done in computer vision.
So this problem is still not totally solved.
And with great tools comes with great consequences as well.
So computer vision can do a lot of good.
But it also can do harm.
For example, human bias--
every single AI algorithm today, the large ones,
are driven by data.
And data is an artifact of human activities
on Earth and in history.
And a lot of the data carry our bias.
And this gets carried in AI systems.
We have seen a lot of face recognition algorithms having
the same kind of bias that humans have.
And we do have to really recognize that.
We can also use AI to impact human lives, some for the good.
Think about medical imaging.
But some are questionable.
What if AI is solely behind deciding your job
or deciding your financial loans?
So again, is it totally bad?
Is it totally good?
These are very complicated issues.
This is also why I always get so excited when students from HMS
or law school or education school or business school
attend my class because not all AI
issues are engineering issues.
We have a lot of human factors and societal issues to solve.
I'm also particularly excited by AI's medicine and health care
use.
This is something really dear to my heart.
Professor Adeli and Zane, who are
also co-instructors of this course, we three of us
work on AI for aging population as well as
patients and to try to use computer vision to deliver care
to people.
So this is a good use.
And also, even in terms of technology,
human vision is remarkable.
I want you to come out of not only today's class
but also this entire course to appreciate,
despite how much computer vision can do,
there's just so much more nuance, subtlety, richness,
complexity, and also emotion in human vision.
Look at these kids studying whatever
that their curiosity lead them or the humor in this image.
There's still a lot more that computer vision cannot do.
So I hope that continue to entice
you to study computer vision.
At this point, I'm going to give the podium to Professor Adeli
to go over the rest of the class.
Thank you.
[APPLAUSE]
Awesome.
Thank you, Fei-Fei.
Great to start of the quarter.
And I hope my microphone is working right now.
OK, good.
I'm seeing some nodding of heads.
So very excited to be here with you all.
And I'm hoping that you will have a fun
and challenging course with an amazing list of core instructors
that we have and great TAs.
So in this class, we are going to cover
a wide variety of topics around computer vision and use
of deep learning in this space, categorized
into four different topics.
We will start with deep learning basics.
And let's start actually with a simple question of,
what is computer vision really?
So at its core, it's about enabling machines
to see and understand images.
And basically, this is the most fundamental task in this space--
in this space is image classification.
You give the model an image, say, of a cat.
And the model should output a label cat.
And that's it.
But this deceptively simple task is the foundation
for much of more complex applications,
from self-driving to medical diagnosis and so on.
So how do we teach a machine to do this?
One of the simplest approaches is to use linear classification,
as you can see in this slide.
So imagine each of the images in our data set
is shown with a dot in that space.
And each axis shows some sort of feature
which was driven from the image itself.
Here, we are showing a 2D space for simplicity.
But the task of a linear classifier
is to find the hyperplane or the linear function
that separates these two, say, cats from dogs.
But we all know that these linear models often
go just only so far.
They struggle when the data isn't cleanly separable
with a straight line.
So the question is, what's next?
We'll get into the topics of how to model more complex patterns.
And if we do so, we often face challenges
of overfitting and underfitting, which
are the topics we will cover in the early lectures of the class.
And to strike the right balance, we
use techniques like regularization
to control model complexity and optimization to find the best
fit parameters.
So these are the nuts and bolts of deep learning and creating
these models, training models, that not only fits the data
but also generalizes to unseen and new data as well.
And now comes the fun part--
neural networks.
We've been talking about them quite a lot.
And what neural networks do, unlike the linear classifiers,
they stack multiple layers of operations
to model non-linear functions to be
able to either classify, to solve the same problem of image
classification, and so on.
These are the models powering everything from Google Photos.
And now, everybody's familiar with ChatGPT, ChatGPT's vision
models, and so on.
In this course, we will go deep into the details of how they
work, how they are trained.
And we will be looking into debugging and improving them.
After looking at the deep learning basics,
we will cover the topics of perceiving and understanding
the visual world, which is a complex process that
involves interpreting a vast array of visual information.
And to do so, we often first define
tasks that refer to specific challenges or problems.
We aim to solve--
some of the examples are object detection, scene understanding,
motion detection, and so on.
And to solve these tasks, we use different models, which
are computational and theoretical
frameworks we develop to mimic or explain
how our visual system accomplishes these tasks.
One of the examples of these types of models
is neural networks.
So by aligning models with tasks,
we can create systems that can see and interpret
the world around us.
Speaking of tasks, let's go back to the topic
of image classification, predicting a single label
for an entire image.
But we know that real world computer vision
is much richer than this.
And let's walk through some of the tasks that
go beyond classification.
First, semantic segmentation, where we are not just
labeling the object or the entire image
as cat or tree or whatever.
Here, we are looking for labels for every single pixel
in the image.
So every pixel is a grass, cat, tree, or sky.
But we don't distinguish between individual objects.
And next, we have object detection,
where we now want to not only say what is in the image
but also pinpoint the location.
And that's why we create bounding boxes
around the objects and associate them with specific labels.
And finally, we have instance segmentation.
We'll go into instance segmentation, which is
the most granular of them all.
It combines the ideas of detection and segmentation
together.
And every object instance gets its own mask.
So these tasks require much deeper special understanding
and images.
And they push the models to do more than just
recognizing categories.
The complexity doesn't stop with static images.
Let's look at some temporal dimensions.
So there's the task of video classification,
as Fei-Fei talked about, where we want to understand
what's happening in video.
Is there someone running, jumping, or dancing?
There is the topic of multimodal video understanding,
which is combining vision and sound and other modalities.
For example, in this example, the person
is playing a vibraphone to really understand
what's happening here.
We have to create a blend of visual features
and audio features to be able to understand what's happening.
And finally, there is the topic of visualization
and understanding that we will be covering in this class, where
we want to interpret what's being learned by the models
and see an attention frame or attention map of what
the model is attending to to do a correct classification
and so on.
And then we have models beyond tasks.
We look into models.
And the very first topic-- let me introduce to you--
that we'll be covering is Convolutional Neural Networks
or CNNs.
There are a number of operations.
We will be going over the details
in the class, starting from an image, a number of convolutions,
sampling and fully connected operations,
and, finally, creating the output.
And beyond convolutional neural networks,
we will study recurrent neural networks for sequential data
and even neural architectures, such as transformers
and attention-based frameworks.
So next, we will be covering some large-scale distributed
training topics, which is kind of new this quarter.
I'm sure you've all heard about large language models,
large vision models, and so on.
And we will be briefly discussing
how these models are actually trained.
We know that data and data sets are expanding models.
And models are becoming larger and larger.
And in order to train such models,
there are some strategies--
for example, data parallelization,
model parallelization-- that we will cover in this class.
But beyond that, there will be so many challenges,
such as synchronization between these models and workers
and so on, as well as several other aspects
that we'll be covering in one of the lectures this quarter.
And we will go also over some of the trends for training
these large models.
After completing this topic, what we will do
next is looking into generative and interactive visual
intelligence, where we will first start
with self-supervised learning.
Self-supervised learning is a branch of machine learning
in which models learn to understand and represent data
by getting some training signals from the data itself.
We will cover this topic.
It's one of the approaches that has enabled training
of large scale models using vast amounts of data that do not
require labels, unlabeled data.
And they have played a key role in recent breakthroughs
in computer vision in general.
And we will talk a little bit about generative models.
They go beyond recognition.
They actually generate.
This is an example of the content of a Stanford campus
photo, which is reimagined in the style of Van Gogh's Starry
Night.
This is known as style transfer, a classic application
of neural generative techniques.
Generative models can now translate language
into images given a prompt.
A model like Dall-E, Dall-E 2 generates an entirely novel
image.
This showcases how generative vision models
blend understanding, creativity, and control
in their generations.
And you've probably heard recently
about the topic of diffusion models in general.
That's another thing that we'll be covering in this quarter.
They basically learn to reverse a gradual noising
process to generate images.
And interestingly, in assignment 3,
you will actually be implementing a generative model
that generates emojis from text inputs,
from prompts-- for example, a face with a cowboy hat, which
is denoised from pure noise.
Vision language models are the next topic of interest
we will be covering.
They connect text and images in a shared representation space.
And given a caption or image, the model
retrieves or generates its corresponding pair,
as you can see.
So there are a lot of advances in this area.
We'll be covering some of the key examples.
Again, this is a key task for cross-modal retrieval
or understanding and visual question answering and so on.
So we'll get to that in the class 2.
Moving beyond 2D, models can now reconstruct and generate 3D
representations from images.
And here, you can see some voxel-based reconstructions,
shape completion, and even 3D object detection from single
view images.
So 3D vision enables more especially grounded
understanding, which is crucial for robotics and AI VR
applications.
And finally, vision empowers embodied agents
that act in the physical world.
So these models often must perceive, plan,
and execute whether it's cleaning up a messy room
or generalizing from human demonstrations.
So with all of these, we will be covering different topics
around generative and interactive visual intelligence.
And finally, we will cover some human-centered applications
and implications, as Fei-Fei very nicely explained.
So there is a computer vision.
And generally, AI have been having a lot
of impact in the past years.
And it's very important to understand
the human-centered aspects and applications.
And some of these impacts are reflected
by these awards that are going to researchers in this space.
It was first recognized by the Turing Award 2018, which
is the most prestigious technical award given
to major contributions of lasting importance
for computing.
Geoffrey Hinton, Yoshua Bengio, and Yann LeCun
received the award for conceptual and engineering
during breakthroughs that have made
deep neural networks a critical component of computing.
Beyond that, last year, in 2024, Geoffrey Hinton
was jointly awarded the Nobel Prize in physics
alongside John Hopfield for their foundational contributions
to neural networks.
And finally, I want to very briefly mention the learning
objectives for this class will be formalizing computer vision
applications into tasks.
As you can see some of the details here,
we want to develop and train vision models, models
that operate on images and visual data--
images, videos, and so on--
gain an understanding of where the field is
and where it is headed.
That's why we have some new topics also covered specifically
in this year.
So the four topics that I mentioned earlier,
we will be going over the basics in the very first few weeks.
Bear with us because these are important topics.
And you need to understand the details first,
how to build the models from scratch.
And then we'll get to more interesting, exciting topics
of the day--
computer vision.
And finally, we'll have one big lecture on human-centered AI
and computer vision.
I want to just leave you with what we
will be covering next session.
That's going to be image classification
and linear classifiers, which will get us started
with the world of CS231n.
Thank you.