So all right so today we actually come
to the last lecture of the class because
Wednesday it's going to be project
presentations and um so I want to talk
to you about diffusion models today
which is a incredibly exciting area
which I don't think gets It's the same
amount of attention in some ways
compared to large language models. Uh
but it's got enormous potential. Um so
I'm very excited to talk to you about
it. So you know just for kicks last
night I said I asked Chad GPT create a
photorealistic image of graduate
students in class in a class on deep
learning and this is what it came back
with.
There is a noticeable absence of an
instructor
plus various students are facing in
various directions
but apart from that it's not bad. Um and
uh here is an example of a midjourney
text to image abusion model uh which
produces the amazing picture from this
prompt. a quaint Italian seaside village
with colorful buildings blah blah blah
blah blah uh rendered in the style of
Claude Monet and so on so forth and
that's what you get. It's pretty
unbelievable.
Uh and I'm sure you folks have played
around with these things and you have
your favorite pictures and prompts and
whatnot.
Um now
uh February 15th um OpenAI released a
texttovideo model called Sora which your
folks may have seen uh which I find
frankly just stunning what it can do. It
can produce a one minute uh video from a
text prompt. And so,
so if you actually give it this prompt,
in an ornate historical hall, a massive
tidal wave peaks and begins to crash and
two surfers seizing the moment
skillfully navigate the fa the wave.
Okay. Uh I think we can all agree that
such a thing has never happened in
history and therefore there it was not
in the training data, right? So and then
you get this picture, this video
and then some random person is coming
back in a completely dry [laughter]
hall. So anyway, but it's pretty
amazing. I think you would agree. So
if you actually look at the open sora
technical report, you actually find this
uh opening paragraph where they say that
we train text conditional diffusion
models blah blah blah using a
transformer architecture. Okay, so now
we know what a transformer architecture
is. You've been working with it. You're
quite familiar with it at this point. So
today's class is really about text
conditional diffusion models. Okay, so
the other building block. Okay, so let's
get to it. Uh what I'm going to do is
I'm going to sort of uh divide this into
two parts. The first part is I'm just
going to talk about how do you get a
model to just generate an image for you?
Right? If you wanted to generate an
image from a class of potential images,
how can it just generate an image? And
then next we talk about okay, great. Now
that you can do that, how do you
actually control or steer the model to
do an image based on whatever prompting
you give it? Okay, how do you condition
it? How do you control it? Those are all
the words. How do you steer it? You'll
find all these synonyms being used
heavily in the literature. That's
basically what they mean. How do you
give it a prompt and then steer what
gets produced? All right, so let's say
we want to build a model that can be
used to generate images of stately
college buildings.
Okay, obviously our very own Killian
Court is the finest example of such a
thing. Um, and uh, but let's say you
want to do that. So what you do is you
as as we always do with machine
learning, we collect a bunch of data. In
this particular case, we collect a whole
bunch of images of stately college
buildings. Uh, and what you see here is
literally me just doing a Google image
search with the query stately college
buildings. Okay, so this is the kind of
stuff you get. Uh, so you have your
training data at your disposal. It's
ready to go. Now the question is if you
have such a model, let's say, and
obviously we'll talk about how to build
such a model very soon. But let's say
you have such a model and every time you
sort of sample this model, every time
you ask the model, hey, give me an
image, you obviously wanted to give a
different image, right? Otherwise, it's
kind of boring. All right? Some you know
maybe you want the Killian Court, maybe
you want the rotunda from the University
of Virginia. Anybody any UVA alums here?
Nobody. Okay. Um, so and right. So the
question is how can we actually get it
to randomly give us different images?
But but they all have to be stately
college buildings. It can't be just some
random stuff, right? So, how do you do
that? And the way we do that, and I
still find it really astonishing that
this approach actually works. The way we
do that is that we actually give it
noise.
And I will define very precisely what I
mean by noise in just a just a bit.
Okay, basically assume
an image in which all the pixel values
are randomly picked.
Right? So every time you generate a
random image and you give it to the
model, you'll it'll use that random
starting point and then create an image
for you. And because by definition, if
you choose noise randomly, they are, you
know, obviously going to be different
each time. It's hopefully going to
generate a different image. But if the
model is trained on stately college
buildings, it will produce images of
stately college buildings. It's not
going to produce a picture of a Labrador
retriever.
Okay, so that's basically what we're
going to do. Now, if you look at
something like this, the first question
of course is that how can we train a
model to generate an image from pure
noise? This just sounds ridiculous,
right? You basically give it a bunch of
random numbers and say, give me code.
It feels really ridiculous. And at that
point, you know, folks can sort of come
to a stop and say, "All right, this
approach is probably not going to take
me anywhere. It's a bit of a dead end.
But then some clever people had this
very interesting idea.
They said
um it's not clear how to do this you
know um just a quick aside there's this
really amazing book which is published
maybe 50 years ago maybe earlier than
that called how to solve it by George
Polia. George Poliov was a eminent
mathematician
um and he wrote this small book called
how to solve it and it lists a whole
bunch of huristics that mathematicians
use when they solve problems and perhaps
the most commonly used heristic is just
reverse the question
just reverse the question and see if
anything comes out of it most of the
time nothing will come out of it but
maybe some other time something amazing
comes out right this is a great example
of that heristic at work we don't know
how to do this so the question is can we
do the reverse
If I give you Killian code, can you
produce noise out of it for me?
And the answer is yeah, of course we can
do that.
Right? Given an image, we can easily
create a noisy version of it. So you can
take the original image, you can add
some noise to it to get this and you
keep on adding a lot of noise and
finally you'll get something that's
basically you can't tell that there is
clean clear clean code anymore. Right?
This process, the reverse process is
actually very easy to do. Okay? So the
question bec by the way for folks of who
may not be very familiar with this
notion of adding noise to an image or
making an image noisy. Let me just show
you in a collab just a minute how easy
it is.
All right. So um we let's say we import
a bunch of these things. As usual we
have numpy and so there is this thing
called the python imaging library pil
which is very handy for image
manipulations. So we import that and
then I just literally read read this
image in. I uploaded it before class.
Let's just make sure it's here. Okay,
good. Kalian.png.
So I I read this image. Okay. Uh and
then once I read it, I convert it into a
numpy array. And then remember in a in
any color image, you have three tables
of numbers. The the number there's a
number for each pixel for red, blue, and
green. And then each number is between 0
and 255. U and so here what we do is we
divide everything by 255 just to
normalize it so it's all between zero
and one and we have done this in the
past right I do that here uh all right
so let me just read this back in convert
it and then if you look at the shape
it's basically a 411 x 583 * 3 um three
channels as we have seen before and then
I'll just show it all right that's the
picture so now what we want to do is we
want to add noise to this picture all we
have to do Okay, for each pixel,
we basically randomly pick a normal
variable, a normal distribution,
normally distributed random variable
with a mean of zero and a small standard
deviation. So it's like a small number
and then we just literally add that
number to every pixel. But for every
pixel, we sample. Every pixel we sample.
It's not like we sample once and add it
to all the pixels. We sample for every
pixel. And so the way you do that is
basically literally np.random.normal.
normal and then this 3 here is a
standard deviation and we tell it
generate as many of these things as the
as the shape of the image that I gave
you. Okay. And then add each one of
these numbers to the original image you
get this noisy image. Okay. So if you
this is the original image these are all
the values between 0 and one. And then
you add do this noisy image. You can see
the numbers have become different. The
23 has become.18.15
has become minus.17 and so on and so
forth. Right? You just added a small
number random to everything. But as you
can see here now you have some negative
numbers. You may have some numbers
that's greater than one. And we do want
everything to be between 0 and one. So
all we do is we do this thing called
clip it where essentially values smaller
than zero are set to zero. Values
greater than one are set to one. And so
we'll just do that. That's it.
Everything over one squashed to one.
Everything under zero set to zero.
Others leave it unchanged. Now it's
again well behaved between 0 and one and
we can just plot it and you get this.
That's it. That's all it takes to
actually add noise to an image. One line
of numpy. Okay. Uh obviously you can
just put this whole thing in a loop and
keep increasing that standard deviation
number from 3 point 4.5 so on and so
forth. And when you do that you get this
nice sequence of clean code and all the
way to some very very noisy version of
Ken code. That's it. So that's the basic
idea of adding noise.
Any questions on the the mechanics?
Okay, good. Um so so we can add random
numbers, right? And we can by increasing
the magnitude of the standard deviation
of these of these normal random
variables, we can make the image
noisier. Okay, so that suggests a really
interesting idea.
What idea would that be?
Yeah, doing the opposite. Could you
please uh microphone please?
>> Uh doing the opposite like recreating
the image from the noise.
>> So we are trying to create the image
from the noise. But
that feels a little hard. So what
exactly can we do? Be a little more
specific.
So here we have the ability to take any
image and add any amount of noise to it.
Right? That's the data we have. There is
Kian code and there's various noisy
versions of Kian code like that for the
return the unit Virginia and so on and
so forth.
>> I would assume you would do some kind of
loss function for the the final image
that you get and compare it with the the
original image that you train it on and
then find uh yeah fine as you go. Okay,
you're on the right track. Uh, any other
proposals?
>> I think we could try to train a neural
network to reconstruct the image going
from the noise to the noise noisy one.
Like we could have a whole data set with
images, find their noise counterpart and
train to do the oppos
network to do the opposite task.
Yeah, that's definitely on the right
track. That's definitely on the right
track. Yep, good ideas. So, what we do
more concretely is
we we can take each image in the
training data and create noisy versions
of it as we have seen before. And then
what we do is that we say uh we can
create XY training data pairs input
output pairs from all these images. So
specifically what we do is we take
the noisy slightly noisy version of
Killian code and call it the input and
we take the the the nice version of
clean code and call it the output.
Okay, that's the y1 x1 pair
and then we get y2 x2 we get y3 x3 and
all the way. So at any point in time,
the relationship between X and Y, what's
the relationship between X and Y? If you
set it up like this as the input and the
output,
>> it's the set of uh standard deviations
and uh the values which you change for
each pixels. Those are like weights to
which you transform,
>> right? Or maybe I was looking for
something simpler which was that that's
correct. So what he's looking for is
really the the relationship between X
and Y. X is an image, any image, and Y
happens to be a slightly less noisy
version of the image.
The slightly less noisy is really,
really important.
You're not going from Killian code,
right? You're not going from the image
to full noise. That's an impossible
leap. You're going from the image to a
slightly noisy version of the image.
Okay, it is that slightly that allows
all the magic to happen.
So that's what we have.
And so here what we can do with these XY
pairs when you have an So here's the
thing, right? This is like a larger
comment about machine learning and deep
learning. Um
whenever you have basically what machine
learning deep learning are or really
it's like this this black box where if
you can find interesting input output
pairs you can learn a function to go
from the input to the output that's it
but this sounds kind of simple when I
describe it like that but there are like
some incredibly non-obvious ways of
applying this idea right so for example
a few years ago Google had this uh thing
which may actually be in production in
Google Sheets now where whenever you um
sort of choose a bunch of numbers, a
range of numbers in a spreadsheet and
and then go into another cell, it'll
immediately suggest a formula for you.
Where is that coming from?
It's because all the Google Sheets users
all over the world, they have been
creating all these numbers with
formulas, right? So, someone says,
"Look, wait a second. We have all this
data on people choosing a range of
numbers and then entering a formula. So
let's imagine the range is the input and
the formula as the output
and let's just give a million examples
of this pair and see if anything comes
out of it and boom you get that feature.
Okay. So similarly here
X is an image less noisy version of the
image. What that means is that we can
build a dnoising network.
Okay, we can take an image and we can
build a network using all these XY pairs
to slightly dn noiseise it.
Okay. Um and so all how do we do it? We
just run stocastic gradient to sit on
the data. We have a network. It has X
and Y and then Y is a slightly less
noisy version and then B.
Okay, you're just a network. It has a
bunch of weights. we have the we have
the right answer in terms of what the
images need to be u we can do stocastic
gradient descent or atom or something
and before you know it if you have
enough data you have a network which can
d noiseise anything you give it okay um
you had a question
>> why slightly
>> why slightly um we'll come back to that
question the the reason is that u in
general you you have to do what you can
to help the model and this is sort of
the proverbial there is an old adage you
can't cross a ditch in two jumps.
It's too big. So, right. So, you can't
do it. So, what you do is you create a
bridge to go from here to there. And so,
what you do is if you can slightly d
noiseise something really well. Well, I
can actually den noiseise anything you
want really well using that fundamental
capability as you will see in a second.
>> Just to follow up. So, if you go back
the last slide, I could have created the
same thing as that is my x1 and that is
my y. Then the second one is x2 and
still this is the y. So there is
effectively there is a learning there
that it could have taken from those
pairs and come back with okay this is
also a possibility this is also a
possibility and it found out that noise
matrix and it can subtract.
>> Yeah. So the thing is you want to make
sure that each time the amount of
learning it has to do is as bounded and
small as possible. If you give it some
starting point and an ending point and
keep moving this ending point, the gap
is still really high for the first
several of those starting points. That's
the problem.
Okay. So to come back to this, so we can
build a dinoising model. We can do this.
And now when you have once you have
built such a thing, you give it some
noisy thing and then it'll you know give
you a slightly less noisy version of it.
Okay, the resolution is going to go up
slightly if you do that. This of course
suggests the obvious way in which you
would use it which is that once you
train it we can solve this problem.
Okay. And how can we solve this problem?
So what you do is you start with pure
noise and then repeatedly dn noiseise
it.
Okay. You get that, you get that, and
then before you know it, Killian Kurt
has emerged from the fog,
right? It's pretty insane that it
actually works this idea.
So, so the model will generate a
sequence of less noisy images and the
final one you have is the answer. Okay.
Now there's a whole bunch of detail here
which I'm glossing over about okay how
many times must we run this loop to get
to a really good picture. The short
answer is you it initially it was like
you have to run it like a thousand
times. Each each each doising step was
like a baby step. You have to do it a
thousand times to get a really good
answer. Again research has been very
active in the area continues to be very
active. Now you can I think do it like
50 steps or 100 steps. Right? But
diffusion models like this uh they tend
to take more time than a large language
model which is why if you give a prompt
to one of these models like midjourney
it will take some time for it to come
back with an image and and that the
reason for the delay is because it's
going through this you know incremental
dnoising loop. Yeah.
>> Uh from this we understand that each uh
the final noise output sample would be
very particular to each image in the
matrix. So I mean like say two if you
take two images the final we are getting
is the image in the after when we start
voicing it and the final output we get
is the noise sample will be too distinct
for each of them right
>> correct
>> so but when we are picking up image to
generate a diffusion model and we work
backwards we may not have the exact
thing available to us what was there
initially
>> no no the thing is we don't want to
necessarily regenerate images that were
in the training data right that's kind
of pointless we want to geneneral new
images
and for new images we just use start use
noise as a starting point
you know the fact that Killian code was
here and then the fully noised version
of Kian code is here that is used for
training and once you use it for
training you don't need it anymore
because you're not trying to recreate
Killian code again you want to create
new images which belong to the category
of stately college buildings and for
that all you you just grab noise send it
in it gives you a stately college
building end of
And because noise by definition is
different each time you pick it, it's
going to come up with a different
stately college building.
So the way I think about it is that uh
all right so you can think of it as this
right this is
so when you sample think of this as like
the noise distribution
each time you sample right there's a
little point you pick from here another
time you sample maybe you get a point
here right each is just you know nice
distribution that's it what actually
these things are doing is they are
mapping mapping it
to the distribution of stately college
buildings which might be in a you know
strange crazy distribution.
So each time you sample you just go from
here and you land at a point here
and when you go from here you know you
land at a point there.
That's what so what you have done is
when you when you take the training data
you basically created points here and
then found the matching noise here and
then flipped it for training as we have
seen before and once you're done with it
you basically have a mechanism for
transforming any entry in this
distribution of images to an entry in
this distribution of images. So it's a
way to transform one distribution to
another distribution. That's what's
going on. Um all right. Um so there was
a question. Yeah. And then we'll go.
>> I understand the going from noise to to
the image and back how you how the
training works. So my question is you
know in some of these models today you
have you know when you give it the noise
now to generate with an image for
example it could generate a human with
four fingers or you know stuff like
that. So is it that the that the model
that the training data is not just quite
enough to or more as robust enough to uh
generate that kind of detail? [cough]
Can you kind of talk through like what's
more?
>> Yeah. So so fundamentally what it's
doing is it actually does not understand
the notion of fingers and things like
that. Right? Because there is like we
haven't injected any domain knowledge
into this whole process by saying that
hey we need to generate you need to
generate a human body and here are the
semantics of what the human body is
right it's got uh five fingers and all
the anatomical stuff we're not giving
anything we literally giving it pixel
values bunch of pictures so everything
you're seeing is basically just coming
out of that very blind statistical
transformation process so it's so you
would expect that macrolevel details
will probably get it Right? Because
there are so many right answers. So
imagine it's actually, you know, it's
creating um the roof of a house. There
could be all kinds of variations in the
roof of the house and you would still
think it's a roof of a house, right?
Because there are many possible right
answers. But when it comes to five
fingers, there are not many possible
right answers, which is why you notice
the error very quickly. As far as the
model is concerned, it doesn't know,
right? It's just producing a
statistically plausible sample from that
distribution. And since we haven't
forced it to obey constraints like five
fingers and so on and so forth, it's not
going to do any of that stuff. It's an
unconstrained process. Now over time,
these things have gotten better and
better and that's because the data has
gotten better to your point. But I think
our approach to doing these things is
also getting better, right? There are
lots of ways to now steer it and control
it so it behaves the right way. And that
is actually part of what's happening as
well. So when we talk about how do you
actually give a text prompt and have it
build the image for that particular
prompt, we would we'll revisit this
question. Um okay, there was there were
more questions. Yeah.
>> Is there some randomness in the model
itself? Right. So if you gave it the
same noise image twice, will it actually
produce the same final image or will it
>> Yeah, there is randomness in the process
as well.
>> In the process process, exactly.
Um, so to actually that's a really good
point, but now I'm afraid to open my
laptop. I'm an iPad. One second.
All right.
Okay. So, what's going on here is that
if you um go to this thing
so I talked about we are transforming
from here to some crazy distribution
here, right? So, what happens that let's
say that this is the starting point for
the the noise input. This is your noise
input and then what it does what you
actually do is you go here
and then you take this point and then
you do a small sample next to it. So you
use this as like the mean value and then
sample around it and that's actually
what gets published in the user
interface. That's where the randomness
comes in.
Okay. So um
so back to this was there another
question somewhere.
>> Yeah.
>> Um it's okay.
>> Uh I was just wondering about the when
going when training on a on a clear
picture to go to a noisy image uh to
pull from a random sample like random
this sample probably pseudo random. I
was just wondering if it's like learning
relationships that are dependent on
pseudo randomness and so when it goes
from a noisy image back to pure image
it's dependent on that or it matters at
all.
>> Oh I see. So if I understand your
question what you're saying is that it's
pseudo random not actually random
>> and so therefore there is some signal in
the supposedly random generation is it
actually glomming onto that signal right
is the question. Theoretically, it's
probably possible, but in practice, it
really doesn't matter because we
basically say random is good enough for
our purposes. And in fact, in practice,
you will see it's not an issue.
Um,
okay. So, oh yeah, go ahead.
>> There's a quick question. when you're
doing uh like text to text, let's say
you're uh tokenizing the input, but here
you somehow have to identify that this
is Killian Cord and like a stately home
and this is just going from pixel image
to or like decoding a pixel image. Um
where does the the tag or tokenization
of like columns or fingernails or like
>> does nothing. It's learning everything
from the pixel values.
>> Everything.
>> Yeah. And this is sort of what I was,
you know, when I when Ike asked the
question about the four fingers, five
fingers thing, it has no idea of
fingers. It has zero knowledge about any
of these things. All it's seeing is a
bunch of photographs.
>> Okay. So when you when you type in say I
want a hand with green.
>> Oh, I see. So we haven't yet come to the
stage of okay, how do you actually steer
this image using your text prompt? It's
coming
>> right now. All we're saying is that
look, I'm going to give you a bunch of
uh photographs of a particular kind of
thing, stately college buildings and I
want to have a model which at the end of
the day I just poke it. Every time I
poke it, it gives me a stately college
building. That's it. Now I'm going to
actually start giving it text and saying
okay build the you know create the thing
I'm just telling you about that's coming
and that's sort of some additional magic
is going on to get that done. U okay so
this is what we have u and this is
called a diffusion model. Okay. And this
is the original paper that figured this
out. Um, and
the the process of actually creating
taking an image and creating noisy
versions of it to create a training data
is called the forward process. And then
what we did in reverse is called the
reverse process. Uh, check out the
paper. It's actually really well
written. Uh, and I recommend it. Now, in
practice, uh, some other researchers
came along shortly after this and made a
small improvement. turns out to be
actually a big improvement in practice
in terms of improving the quality of
what's being produced. And so what they
said is hey instead of training the
model to predict the less noisy version
of the image we actually ask it to
predict just the noise
in the input and then we will just
simply subtract the noise from the input
to get the image. So instead of saying
here is an X X is an image Y is the
noisy image we actually tell it here is
an image here is the noise that we added
to X to get the the noisy version and
then just predict the noise for me and
then once I get it I just do X minus
noise and I get the less noisy version
of the image. Okay, this feels
arithmetically equivalent but in
practice it ends up generating much
higher quality images and there's some
very interesting theory as to why that
works and so on and so forth and you can
read this paper if you're interested.
Okay, so if you actually look at what's
going on in most diffusion models today,
they're basically using an approach like
this. They're actually predicting each
time they predict noise and take it
away, subtract it. So iterative
subtracting of predicted noise.
That's what's going on. So all right, so
that's what we have. U now at this point
you may be wondering, okay, so far in
the semester, uh we have actually
learned how to take an image and then
classify it into one of you know 20
things, 10 things, whatever. We also
taken text and figured out what to do
things with it. We haven't yet talked
about how do you actually take an image
and the how can we get the output also
to be another image. We haven't done
that yet. Okay. So we have actually not
done image to image. How do you actually
build a neural network to do image to
images? And in the interest of time
you're not going to get into it
massively but I want to give you a quick
idea of how it works. So the the most
sort of I would say the dominant
architecture
to take an in image as an input and
produce an image as an output is called
the unit. Okay. And that's the
architecture we see here. So
so fundamentally if you look at the left
half so there's a left half to the
network and a right half to the network
hence the U. If you look at the left
half of the network it's it's a good old
convolutional neural network like the
kind we know and love. Okay. And the the
kind that we are very familiar with. So
you take an input image and then you run
it through a bunch of convolutional
uh convolutional blocks and then we do
some max pooling and then we keep on
doing it and at some point it becomes
smaller and smaller and we get something
you know like this which we are very
familiar with right the the big image
with three channels gets smaller and
smaller smaller but the number of
channels gets wider and wider. it
becomes sort of much smaller but much
deeper right it becomes like a 3D volume
and we have seen that again and again
right the left part is just a good old
convolutional with pooling layers and
then you come to the middle and then
from this point on what we do is we take
whatever this thing here and then we
essentially reverse the process we go
from the small things which are really
deep to slightly bigger things that are
a little less steep and so on and so
forth till we get the original size back
again Okay. And we do that using the
some an inverse of the convolution layer
called an upconvolution or deconvolution
layer. Okay. And you can check out 9.2
in the textbook to to to understand how
it's done. It's it's also called con 2D
transpose.
Okay. It's a very similar idea and I'm
not going to get into the details here
but you essentially do an inverse of a
convolutional operation to get the size
to come back to the bigger size and you
do it gradually till the output you have
matches the size of the input that came
in.
Okay, so image gets smaller and smaller
into a thing and then you just blow it
back up again to get an image back. So
that is the unit. Now there's very one
very important thing that happens in the
unit, right? which is
you see these connections, right?
Basically, what they do is at every step
when you're coming back up in the right
half, you actually attach whatever was
in sort of the mirror image of the
original input as we processed on the
left side, we attach it to this side as
well. Remember I talked about this whole
notion of a residual connection back,
you know, many classes ago where I said
when uh when an input goes through each
layer of a neural network at one point,
let's say you're in the 10th layer,
you're only seeing what is the ninth
layer is produced for you. That's all
you're working with. But would it be
nice if the the the 10th layer actually
had access to the eighth layer, the
seventh layer, the sixth layer, the
fifth layer? Heck, why not the input,
right? Because the more information it
has, the more able it's probably to do
whatever it can with the input it's
giving. Why restrict it to only the
input of the the output of the previous
layer? Why can't we give it everything
that has came before it? Now giving
everything is too much. But we can be
selective in what we give it. Right? So
what these folks decided I'm sure after
much experimentation is that if they
actually attach whatever was coming out
of this layer to this layer before it
goes through the output, it really
helped. Similarly, this thing gets
attached and so on and so forth. And it
kind of makes sense. You know, why force
it to figure out everything it has to
figure out just from this thing that
came in, right? Let's give this that
that. Let's also give a little here, a
little here. So, these residual
connections are a huge building block
for why these things work as well as
they do. Okay? And in general, giving a
layer as much information as you can
give it is always a good idea, but you
can't go nuts, right? Because then you
have much more parameters and all kinds
of stuff happens. So there is a bit of a
balance you have to strike and this was
the balance struck by these researchers.
And so this thing was originally
invented for some medical segmentation
use cases but it's just heavily used for
everything now. It's a really powerful
architecture. Uh questions
>> uh can we have example of like in what
scenarios we use this kind of
>> anytime you have an image to image
>> like what kind of conversion do you get
image to image? or like what kind of
examples of use cases. Let's say that
for example you want to take an image
and like a black and white image and you
want to colorize it
for instance boom you unit you want to
take an image and make it a higher
resolution image unit you want to take
an image and for every pixel in the
image you want to classify it into you
know one of 10 things. So anytime when
you want the output shape the shape of
the output to be basically the same
shape as the input but with other data
you need to use this.
Yeah.
>> But this logic of having access to all
the previous iterations
>> not iterations
>> all the previous layers
>> right the outputs of the previous layers
>> layers. Uh but this would also help uh
clean up and give better categorization
like does it always have to be an image
to image?
>> No. No. In fact, if you look at restnet,
restnet is the one in fact that
pioneered the idea of the residual
connection. So we use it for restnet. We
actually use the the transformer stack
if you remember it goes through the self
attention layer. It comes out the other
end and then we add the input back to it
and then we send it through layer.
So you will see that this residual
connection is sitting in two different
places in a single transformer block. So
it's extremely heavily used. There is
something called deep and wide network
if I remember or denset which uses the
same trick. In fact if you when you're
working with structured data right good
old say linear regression and you've
looked at your data and you come up with
all kinds of very clever features you
know I'm going to look at price per
square foot right you do a bunch of
feature engineering and you have a bunch
of new features. Well, you should take
your old features and your new features
and send both in.
Why send only the new stuff that you
have concocted? Why can't you send
everything in? That's the idea.
All right. Um, so let's come back here.
Now we have seen how to generate a good
image. Okay. Now let's figure out how to
steer it or condition it with a text
prompt, right? Because that's sort of
the holy grail.
So we want to take
so here's some intuition. We want to
take the text prompt into account and
obviously generate the image. Now
imagine if we had like a rough image
that corresponds to the text prompt.
Just imagine. So the text prompt is you
know cute laborator retriever and you
have like a very noisy image of a
laboratory retriever. This just happens
to be handy. You have it. Well now
you're in good shape because you just
feed that in and your system will denise
it for you. Right? Right? You can get a
better image. That's pretty easy. So,
but obviously in reality, you don't have
a rough image. In fact, you're trying to
create one of those things in the first
place. We don't. So, but what if we had
an embedding for the prompt that's close
to the embeddings of all the images that
correspond to the prompt. So, let's take
a prompt and let's imagine all the
images in the in the universe that
correspond to that prompt. Okay?
And now further imagine because
everything is a vector. Everything is
embedding in our world that that image
has an embedding.
All sorry the text prompt has an
embedding. Every image has an embedding
and we have somehow calculated these
embeddings so that the text prompts
embedding is smack where all the image
embeddings are.
We will get to how we actually do it in
a in just a moment. But conceptually
imagine if we had an embedding if you
could calculate embeddings for text and
embeddings for images. So they all live
in the same space.
Okay. So if we feed this embedding to a
dinoising model because that text
embedding is sitting in the same space
as all the image embeddings that it
corresponds to. Maybe our model can just
d noiseise that embedding and give you
what you want.
Okay, so since this embedding is already
close to the embeddings of the things we
want to generate, maybe you'll just get
it done.
So ultimately we want to generate an
image and if we had an embedding for
that image, we could generate the image
from the embedding and we use the text.
So we go from text to embedding which
happens to live in the same space as all
the embeddings of the images we care
about. And then from that image
embedding, we go to the final image.
Okay, this is a bunch of me talking and
handwaving. it'll all become very clear
but that's sort of the rough intuition.
Okay. So, so what we'll know is we'll
describe an approach to calculate an
embedding for any text any piece of text
that is close to the embeddings of the
images that correspond to that piece of
text. So this is the problem we're going
to solve. There's a bunch of text
conceptually there are a whole bunch of
images that are describe that text and
we're going to now create embeddings so
that that is close to all the embeddings
of those images. Right? It feels kind of
like almost impossible that you can
actually do something like this, but
there's a very clever idea uh that
OpenAI came up with that tells you how
to do it. So, here's what we're going to
do. So, let's say we have an image and a
caption. So, here's an image. Uh here's
a caption, right? And we need some way
to take that piece of text and run it
through some network and create a nice
embedding from it. Okay? Similarly, we
want to take this image, run it through
some network and create an embedding
from it. Okay. Now, first first
question, how can we compute embeddings
from a piece of text? First question,
how can we comput an embedding from a
piece of text? You know the answer.
Run through a transformer. Piece of
cake. We know how to do that, right? U
right in particular, you can do
something like BERT. And for an image
encoder, you just run it through
something like restnet like the the
penultimate layer, right? one of the
final layer is going to be a very good
representation of that image. You get
another embedding. So using the building
blocks we already know, we can create
embeddings very quickly from these
things. Okay, but if you just take a
piece of text and run it through a bird
and you take an image and run it through
SNET, you're going to get some
embeddings. But why the heck should they
be related?
They were not trained together. So
there's no basis for them to be related.
They would just be some two embeddings.
Maybe they are kind of similar. Maybe
they're not. We don't know. There's no
reason to expect that they're going to
be similar. Okay, they're just two
embeddings.
Now, what we want to do is but once we
have these, we need to make sure the
embeddings that comes out of these two
things satisfy two very important
requirements.
We want to make sure that if you give it
an image
and a caption that describes that image.
So you have an image and a caption that
describes that image, we want to make
sure that the embeddings that come out
of these two boxes, they are as close to
each other as possible.
Okay? Given an em given an image and a
caption that describes it, that's the
connection. They have to be close to
each other. And conversely, if you have
an image and a caption that's totally
irrelevant,
right? A train rounding a bend with a
beautiful fall foliage all around,
right? Clearly irrelevant. Those
embedings should be far apart.
that it to really make sense,
right? Pairs of related things should be
together, irrelevant things should be
far apart. So if you can find embeddings
that satisfy these two criteria, maybe
we will be in the game. Okay. So now
this ensures that the text embedding and
the image embedding are referring to the
same underlying concept. Right? This
these requirements will enforce that. Uh
and so the embedding for any text prompt
is close to the embedding for all the
images that correspond to that prompt.
So the question is how do we do this? Uh
how can first of all how can we tell how
close two embeddings are? You know the
answer to this what's the answer
>> correct cosine similarity right? We use
the cosine similarity of the embeddings.
U so we know how to measure closeness.
So the question is how can we compute
embeddings that satisfy the two
requirements and openai uh built a model
called clip which is very famous uh to
solve this problem right it stands for
contrastive language image pre-training
uh and this forms the basis for a whole
bunch of models that have sprung up
after this called blip and blip 2 and so
on and so forth but this is the
fundamental idea
okay so
this is how clip works we uh what they
did is they took a a 12 block 12 layer 8
head transformer cosal encoder stack as
a text encoder
uh okay now you understand this right
that's what it is eight layer I mean
sorry 8 head 12 layer transformer causal
encoder TC stack um and and that's a
text encoder so we send any piece of
text through it right you get the next
word prediction embedding and that's the
embedding you're going to use uh and
they took restnet 50 and made it the
image encoder they took rest 50 chopped
off the top and whatever was left is the
the image encoder. Okay,
then they initialized with random
weights these things and then they
grabbed they grab a batch of image
caption pairs. So in this example, let's
say that we have these three images u
and I have captions to go with these
images. Okay, we have these three things
and this is the key step. They run the
images through the image encoder and the
captions through the text encoder and
get these embeddings. Okay, it's a
forward pass. You send it through this
network, you get two embeddings. Um, and
then this is what they do. With these
embeddings, they calculate the cosine
similarity for every image caption pair.
Okay? And so imagine something like
this. So you have these three captions,
you have these three images, and those
are the embeddings.
uh and then they calculate the cosine
similarity for every one of those
things.
It took me like 5 minutes or 10 minutes
to do this PowerPoint. You're welcome.
Particularly trying to get this comma to
line up is a real pain in the neck. So,
so all right. So, we have this here.
Okay. And now what we want to do is uh
we want these scores to be as high as
possible, right? Because the scores in
the diagonal are the ones where for the
matching picture and caption,
right?
Those are the those are the those are
the the scores for the matching pairs of
embeddings. We want them to be as high
as possible.
Okay. Um
so so we want to maximize the sum of the
green cells, right? These are the green
cells the diagonal. So, so if I if you
want to write it as a loss function
because the loss function is always
minimization, we basically say minimize
the negative sum of the green cells.
Okay, so the question is would this loss
function do the trick?
Seems reasonable. You want to make sure
the related things are really close
together. So you want to maximize
uh if that was the only part of the loss
function, wouldn't it just kind of
squish everything to the same spot in
the space?
>> Correct.
What it's going to do is it's going to
basically ignore the input.
The optimizer can simply ignore the
input, make all the embeddings the same.
For example, it can just make all the
embedding zero.
That's it. And then now we have a
perfect cosine similarity for
everything. For a any pair of image and
captions, the cosine similarity is going
to be one. It's perfect, right? So
clearly that's not enough. This is by
the way is called model collapse, right?
So to prevent it from doing that, we
need to do one more thing to the loss
function. Any guesses?
>> Yeah.
>> Uh make the images that aren't related
not have a cosine similarity.
>> Exactly. Right. Exactly right. So what
we want to do is we want the scores of
the red stuff to be as small as
possible.
We want the green stuff to be as much as
possible and the red stuff to be as
small as possible.
Together it'll get the job done.
Okay. And so um so we want to maximize
the sum of the green cells and minimize
the sum of the red cells. So the
equivalent loss function is minimize the
sum of the red cells and the negative
sum of the green cells. That's it. So
all clip does is that it just grabs a
batch of image caption pairs, runs it
through the networks, calculates the
embeddings and calculates this sum of
the stuff here and that is your loss and
then back propagates through the
network. Boom. Batch batch batch. Do it
a whole bunch of times. And OpenAI did
this with uh oh this is the official
picture from the open from the paper
which is worth reading by the way right
it comes in text encoder you get these
uh embedding vectors image encoder and
then boom the diagonal is maximized and
the off diagonals are minimized
and they did it with 400 million image
caption pairs scraped from the internet.
400 million.
By the way, you folks who work in the
space may know this really well, but uh
one very easy way to get a caption for
an image, right? You we see images, but
where do you think the captions come
from? Where did they get those captions?
They didn't obviously they didn't ask
people to manually label each image of
the caption. Where do you think they got
it from?
>> Google search.
>> Uh Google search can help but why does
Google search actually find the caption?
How does it because Google search is not
creating the caption? um
>> take it from the alt text on the images.
>> Correct. Alt text. So a lot of folks for
accessibility reasons they have alt text
right on all the images they create. A
lot of people have alt text in their
images they publish on the web and
that's what we use. And the alt text
actually ends up being a a more verbose
description of the image than a typical
caption which tends to be much briefer.
And for us more verbose longer the
better because there's more stuff for
the bottle to learn from.
Um, so that's how they built clip.
And so now what we do is we use we can
use clip's text encoder by itself,
right? We can send in any text and get
an embedding that is close to the
embedding of any image that described by
the text.
Okay. Now, by the way, clip can be used
for zeros image classification.
And what I mean by zeroshot image
classification, I'll I'll walk through
the picture in just a second, is that
typically when you want to build an
image classifier, right, you can get a
whole bunch of training data of images
and their labels and then we train them,
right? Maybe you take something like
restnet, chop off the top, attach our
own output head and train, train, train.
Boom, you have a classifier. But the
only problem with that is let's say that
tomorrow so today for example you had
five classes in your problem and
tomorrow somebody comes along and says
oh actually we have a sixth category
right what do you do then well you have
to go back to the drawing board and
retrain the whole thing with six labels
now not five because your problem has
changed would it be great if you had a
classifier where you just come to it and
say here's an image and here are the six
possible labels I want you to pick from
pick one from me and you want to be able
to give it a different set of labels
those each time and it'll just use the
labels you're giving it and the image
and figures out which which label
corresponds to the image you just fed it
that would be an insanely flexible image
classification system right and that's
what I mean by zeroshot image
classification and you can use clip to
do zero short image classification
the now how you do it is actually in the
picture though not very clearly done
anyone wants to
How can you use clip to build a like a
infinitely flexible image classifier?
>> Um I mean the text input was like was
trained vert right? So in the same way
vert can handle words never seen before
does it essentially do that? Sorry, say
that again. The second part
>> you're saying you're saying it sees a
text input with something it's never
seen before, right? Yeah.
>> Okay. So, in the BERT model, which is
where where it came from, in the text
encoding in the BERT model, I think we
talked about when it sees a word it
doesn't know that it's never seen
before, it can use the the context words
around it to try to
>> Right. Right. So, but but here, just to
be clear, I I want you to use clip that
we just built, right? And assume clip
can see any knows all the words because
it's been trained on a big vocabulary.
You can give it any text you want. It'll
create an embedding from it. That's the
key capability.
>> So it creates a text embedding for
>> Yeah.
>> because like and then for your image.
So comparing similarity scores between
the two the image is complete but the
text is not complete. there'll be
missing pieces and then make some
prediction using this.
Why is there a missing piece in the
text?
>> Because um the image the the text
the text does not contain the class. Um
and then but for the image the way it
was trained it was trained like with
pairs with class including
>> right but we actually know the class now
because so the use case is that I come
to you with an image and I say here are
the seven possible labels for this image
and each label is a piece of text.
So you can you actually have seven
pieces of text and an image and all I
want clip to do is to tell me okay the
seventh the fourth label is the right
one for this image
but you're on the right track
once you see how it's done you'll be
like yeah of course
might not be understanding something but
wouldn't you just pick the embedding
that's the closest to the like the the
text embedding that's the closest to the
image embedding Correct. You're not
missing anything. That's the right
answer. Well done.
Come on people. Can you applaud our
fellow here? [applause]
You folks are hard to impress.
That's exactly what we do. So here
the the key thing to remember the key
thing to keep in your head is that when
you a label is just text,
dog, cat, right? It's just text. So you
can just imagine taking each label with
which in this case is plane car dog
whatever for each one of them you create
an embedding you get t1 through whatever
if you have n labels for the image you
just have one embedding i and then you
just create the cosine calculate the
cosine similarity and whichever is the
highest number you say okay it's a dog
that's it
it's super just imagine the level of
flexibility here
so that's a a side use of clip unrelated
to diffusion models but that's just
thought it's really clever so I wanted
to share that okay good u now let's see
how we can actually use this entire
capability to go to solve the original
problem we set out to solve which is can
we steer the diffusion model to create
an image based on a particular prompt we
give it um so now remember if you go
back to how we did it we created all
these training pairs of x and y based on
you know the the noising the image x is
the image y is the less noisy version of
image. So what we can simply do is we
can actually change the input so it
becomes the image and then the clip text
embedding of the caption for that image.
So you have an image and you have a
caption. You take the caption run it
through clip you get an embedding. By
definition that embedding is in the
lives in the same space as all the
images that correspond to that caption.
Right? So you just attach you
concatenate the embedding of the clip
output of a caption along with the
image. You say make that the new input.
Now Y continues to be the less noisy
version of the image or as we saw
earlier it could be just the noise
component of the image. Okay, this is
the new XY pair that we have. And so now
the model is you send the clip X
embedding the image X send it through
noisy version of the image and you keep
on training it for a while. Once your
model is trained for when you want to
use it for inference for a new uh
prompt, you just give it you know
Killian quoted MIT during the springtime
along with a bunch of noise goes in it
starts dinoising it. But because this
embedding of this thing thanks to clip
lives in the same image space as all Ken
code embeddings clean code images at
some keep on doing it for a while at
some point you'll get Kian code.
That's how they do it. That's how they
steer the image. It's a two-step
process. You create all these clip
embeddings uh which clip was a
breakthrough in my opinion because they
it was one of the maybe the first
example. I don't know if it's the very
first but one of the early examples of
saying we have different kinds of data.
We have images, we have captions, we
have text. How do we create embeddings
for every one of these very different
data types that all happen to live in
the same space, the same concept space?
That was the key idea. And if you look
at the modern multimodal large language
models, they are all based on the same
exact idea.
So it's very powerful this approach.
Yeah. Now I understand this for images,
but for video generation models like
Sora, do they have some sort of
underlying physics structure or do they
learn the physical representations?
>> There's a lot of debate on the internet
about this stuff. Um they haven't
published the results, the full
technical report yet. So we don't know
for sure but the consensus seems to be
no it's not they are not using a physics
engine what they have done uh and again
this may be wrong once the report comes
out we'll know for sure but uh people
what people are saying computer vision
experts is that it was has been trained
on a lot of video game data
uh along with actual videos and so on
and if you and the corpus of training is
so massive that it has basically learned
to mimic certain physics aspects to it
just as a side effect much like LLM you
train them on a large amount of text
data they begin to start to do things
which you didn't anticipate that they'll
do right so for example I read this I
thought it's a really great example of
what is surprising about large language
models is not that you know you train
them on a b bunch of high school math
problems and then you give it a new high
school math problem it can actually
solve it that's not surprising you give
it a whole bunch of high school math
problems in English then you ask it to
read a bunch of French literature and
then you give French high school math
will solve it. That is that is the new
news, right? So similarly here I think
the expectation is that it's not
actually using a physics engine under
the hood. It may have used a physics
engine to actually come up with the
videos and renderings but there are no
physics constraints in the model itself.
It just comes out of the training
process. That's the current view. Once
the technical report comes out, we'll
know for sure what they actually did.
U
>> so quick question about stability. It's
claiming to be a little bit more real
time in their image generation. Um, so
>> you mean stable diffusion?
>> Yeah, stable diffusion. So, are they
jumping through the noise more quickly
or are they kind of like pre-prompting
it and kind of trick?
>> Very good question and there's a very
key trick. It's coming.
>> Um,
>> so here the example of the noise is
normal distribution. However, if we have
changed the noise distribution, is it
change the result? Oh, you mean if you
change it to like a pson or some other
distribution, it'll definitely change
the results because u if you look at the
underlying math of why this works, it
heavily depends on the Gaussian
assumption.
>> Yeah. Um there was another question
somewhere here.
>> Um you may not know the answer because
the technical report out, but could it
be in terms of video generation sort of
analogous to going from like one fuzz
one noisy image to another? like you're
almost doing a series of still images
and learning how to
>> No, I think that I think people are sure
is is how it's done. So, basically you
think think of the video as just a
series of frames, right? And each frame
is an image and there is a sequentiality
to it. Um, which is where the
transformer stack will come in because
it handles sequentiality. So, in general
video stuff typically operates on frame
by frame which is just an image. So,
that is definitely there. What we don't
know is if they also used some
understanding of the fact that for
example that if an object is dropped it
has to fall to the earth in a certain
rate or if an object goes behind another
object you can't see the object anymore
right things like that which we take for
granted um the question is are they
using it and the consensus seems to be
uh in the absence of an actual technical
report that no they're not doing it
because there are lots of examples on
Twitter where people will show a Sora
video in which it's not obeying the laws
of physics. So you take like a beach
chair and then put it in the sand. You
see the sand come through the base of
the beach chair, right? Or you take an
object and put it behind an object. You
can still see the object even though the
original object is opaque. So you be
seeing some evidence that no no it's not
obeying the laws of physics. What you're
seeing is just an amaz
fingers without knowing there has to be
only five fingers.
Um
okay. All right. So we let's keep going
now. Um so this there was another paper
afterwards and this is the original
paper which took that idea of the
diffusion model and then diffusion is
very slow as Olivia you pointed out. So
the question is can we make it much
faster? Right? So what they did and I'm
not going to get into this whole thing
here. I just want to highlight a couple
of things. The first one is that um
first of all notice that you see unit
here. So it they are using a unit right
to go from image to image.
The second thing is that the clip
embedding of the text prompt is
basically is woven meaning it's
incorporated into the w the into the
into the unit through an attention
mechanism a transformer mechanism and
you can see the QKV business here which
should be familiar at this point. So it
is integrated into the transformer stack
directly that input the clip embedding
that's the second thing I want to point
out. And then thirdly
and this is where the speed up comes. So
what you do is instead of taking the
image running it through the whole
network and creating a slightly less
noisy version of the image here what you
do is you take the image you run it
through an image encoder you get an
embedding and now you only work with the
embedding you take the embedding and
create a slightly less noisy version
embedding keep on doing it and these
embeddings are much smaller than images
therefore they're much faster to process
and once you've done it like a thousand
times you get a very sort of almost pure
noless version of the embedding now you
run it through an image decoder to get
So this is the the idea here is that you
operate um
uh in the lat latent space meaning the
embedding space and hence it's called a
latent diffusion model. So that's where
the speed up comes but research
continues to be very strong to make it
even faster because for a lot of
consumer applications people are
obviously not going to wait around for I
mean who wants to wait for 10 seconds
right so uh and so there a lot of
pressure to make it even faster
um
all right so that's what we have
obviously um you know they're these
models are transforming everything and
uh by the way this site here lexicon.art
art. You can go check it out. Uh it has
a whole bunch of very interesting images
and prompts that created the images. So
if you're working in the space, it gives
you a lot of interesting ideas. But it's
not just for you know consumer fun
applications. U you know these models
are being used to actually you know
alpha fold if you'll recall if you give
it an amino acid sequence it can
actually create the 3D structure. Right?
So that's an example of they they don't
I don't think they use a diffusion
model. But you can imagine using a
diffusion model to create these
complicated objects. Meaning the objects
you create don't have to be images.
They can be arbitrarily complicated
things. As long as you have enough data
about such things to use for training
and the notion of noising the input is
meaningful, you can create some very
interesting structures. you can create
3D things and u you know protein
structures and there's a whole bunch of
very interesting applications in
biomedical uh sciences. So this is
really just the tip of the iceberg and
now there are these things um there are
ways in which you can use diffusion
models to create to do large language
modeling as well. So there's a lot of
overlap and blending and so on going on
in the space. So so I'm going to do a
quick demo. Um if you look at hugging
face there is something called the
diffusers library which is like the the
as the name suggests it's a library for
a lot of diffusion models
and let's take a quick look.
All right so we will uh the diffusers
library has a whole bunch of diffusion
models. We going to work with stable
diffusion which is one of you know like
the the better known models. So let's
install diffusers.
Uh you will recall when we when I did
the quick lightning tour of the hugging
face ecosystem for language. Uh hugging
face is a whole bunch of u capabilities
sort of built out of the box and you use
this thing called the pipeline function
to very quickly use any model you want.
The same exact philosophy applies here.
You still use the pipeline. So I'm going
to import a bunch of stuff.
All right. So, oh, I see I have to do
this thing. Okay.
Great. F.
Okay. So, uh, all right. that we have
here. So you you'll remember that we
when we worked with text we had to pre
we we would grab a pre-trained model and
then we actually run it through a
pipeline and we can do all the inference
we want on it. The same exact philosophy
applies here. So um and this very
similar to what we did in lecture 8 for
NLP. So what we're going to do is we use
this command the stable diffusion
pipeline from pre-trained and we use
this version 1.4 stable diffusion model.
Um so let's just create the pipeline and
and obviously we have used tensorflow
not pyarch here but a lot of these
models unfortunately happen to be in
pyarch so knowing a little bit of pyarch
is actually very helpful um to be able
to work with these things and what we're
doing here uh while it's downloading uh
we are using this fp16
um storage format for the the model
weights because it's going to be a
little smaller than using 32 bits so
it'll download faster. So that's what's
happening here. So all right, it's
downloaded fine. So now we just give it
a prompt and this is actually one of the
original famous uh meme prompts a
photograph of an astronaut riding a
horse. And so uh once we have the
pipeline set up uh I'll just a seat for
reproducibility. And then literally I do
pipe of prompt and then it's actually
you can see here 50. So it's going
through 50 dinoising steps. Okay. Um and
you come up with a national rating of
horse. Okay. So that's that. Um you can
actually change the seed and you can get
get a different um the seed is basically
sets the the the random starting point
for the image. So therefore you would
expect a different astronaut. Yep. This
is an astronaut riding another horse. So
um I think people came up with these
kinds of fun examples because it's
guaranteed not to be in the training
data, right? So so whatever the model is
doing, it's not remember it's not
regurgitating what it has already seen.
Uh, all right. Give me a prompt.
Prompts. Anyone?
Wow.
>> Okay,
that might be a
All right. Riding a horse.
All right,
there are two of them and clearly MIT
professors don't have really.
Yeah, moving on. [laughter]
So, so by the way, um if you you should
spend some time with the diffusers
library, they have a bunch of tutorials
which are really interesting because
this core capability of giving a prompt
and getting an image out can actually be
manipulated for all sorts of very
interesting use cases. So, for example,
there is this thing called negative
prompting. And the idea of negative
prompting is that you can give it two
prompts and say create an image which
embodies the first prompt but not the
second prompt. essentially subtract the
second prompt from the first one. That's
called negative prompting. And you might
be wondering like what use is that?
There are lots of fun uses. So here we
are going to the prompt is going to be a
labrador in the style of vermier. Okay,
that's the first prompt. 50 steps.
Uh look at that. Amazing, right? Uh but
maybe you don't care for the blue scarf.
So you basically give it a negative
prompt. And you basically the negative
prompt is blue meaning remove everything
that's blue. I don't like this otherwise
keep the Labrador thing going. So you
run it.
Look at that. The blue is gone. Negative
prompting. Okay. Yeah.
>> If you change that from five from 50 to
a th00and will it become less pixelated
or will it eventually just keep going
and iterating?
>> No. Typically, if you do more of these
things, it gets better. The quality is
much better because each step will den
noiseise it very slightly. So, errors
won't accumulate and things like that.
And the diffuses library gives you lots
of controls for fiddling around with all
these things. Um, okay. So, that's what
we had. Uh, 949.
Okay. So, check out this tutorial if
you're curious about how this stuff
works. And I'm going to do one other
thing um because I didn't get to do it
earlier on. So uh we spent some time
with the hugging face hub and I walked
you through a few use cases for text uh
where you can take a text model and use
it for you know classification uh things
like that summarization and so on and so
forth. You can do the same thing for
computer vision models. So if you have a
computer vision problem that just maps
to a standard C uh computer vision task
you can just use the hugging face hub as
well. So um let me just show you very
quickly the same kind of thing actually
works here.
All right. Okay. So,
so let's say that you want to classify
something. You just import the pipeline
as before.
And once you import it, you can just
literally give it the standard task that
you care about like image
classification.
And and then you can start using it
right from that point on.
Okay.
All right. Okay. So now I'm going to
just get this image. So it's a very
famous image. Um, right. And we're going
to ask it to classify this image. So we
just literally run it through the
pipeline.
And it says the most likely label is 94%
probability. It's an Egyptian cat. Seems
reasonable. Okay. I mean, it's it's a
tough picture, right? Because there are
lots of things going on in that picture.
It's not like one one image, one object.
Um okay so you don't have to use the
default model you can actually give it
your own model that you want. So for
example, you can go um sorry
you can go to the hub hugging face hub
and you can go in there and say all
right I want image classification these
are all the models 10,487 models let's
sort by I don't know most downloads or
maybe most likes
u and you have all these models you can
pick any one of them so for example
let's say you want to pick Microsoft
restnet as your one that's what I tried
here so I have Microsoft restnet you
just s model equals that run it and it
takes care of all the tokenization this
that and whatnot. It's really very handy
and then you run it through the pipeline
again and it says tiger cat 94%
probability according to restnet. Um so
yeah so that's how you do it. Now let's
actually try a more interesting example
where you want to detect all the objects
in the picture which we didn't talk
about in class object detection. So just
create an object detection pipeline.
Same thing as before. when you actually
run this command, an astonishing some
amount of complicated stuff is going on
under the hood. Okay, and we are all the
beneficiaries of that. So, thank you.
Um, so yeah, so we have this here and
then we run it through um the pipeline.
It's looking at all the possible things
that might be sitting in the pipeline.
The results are hard to read. So, let's
actually visualize them. Um,
and I got some nice code from this site
for how to visualize them. Let's just
reuse it. So, yeah. So if you plot the
results,
look at that.
Okay, so it has picked up the cat. 100%
probability, I guess. The remote, the
couch, the other remote, and then the
cat. Pretty good, right? Off the shelf,
ready to go. No, no heavy lifting
required. Now, in in this case, we are
actually putting these boxes called
bounding boxes around each picture. But
what if you actually don't want a
bounding box? what you want to actually
find the exact contour of that cat or
the remote. No problem. We do something
called image segmentation. So let's do
an image segmentation pipeline
uh and run it through.
It takes some time. Um all right. All
right. Let's visualize it. So you can So
each object it finds it gives you a
mask. It basically tells you for each
object what object it is and then which
pixels are on for that object and off
for everything else. It's a mask. It
tells you where it stands. And you can
see here it is the first the object has
found is this thing here. And it's
perfectly delineated, right? It's pretty
amazing. So we can overlay this on the
original image and see it has found that
and it is Let's look at the other
objects. Oh, it has found the remote.
That's the second object.
And the third remote
and the fourth. You think any other
objects are remaining?
>> Couch. Good. All right, let's find the
couch.
And look, the couch is pretty good
except that the middle part has gotten
confused.
All right, but it's still pretty good,
right? So, yeah. So, that is um so
hugging faces all all these things and
so you should definitely check it out
and if you're not already very familiar
with it. So, uh, we have one minute
left. Any questions?
No questions. Okay. All right, folks.
See you on Wednesday. Thanks.