Machine Learning for Everybody – Full Course — Full Transcript

1:46

whether it was a gamma particle, or some other head, like hadron. Down here, these are all of

1:54

the attributes of those patterns that we collect in the camera. So you can see that there's, you

2:00

know, some length, width, size, asymmetry, etc. Now we're going to use all these properties to

2:06

help us discriminate the patterns and whether or not they came from a gamma particle or hadron.

2:13

So in order to do this, we're going to come up here, go to the data folder. And you're going

2:19

to click this magic zero for data, and we're going to download that. Now over here, I have a colab

2:28

notebook open. So you go to colab dot research dot google.com, you start a new notebook. And

2:34

I'm just going to call this the magic data set. So actually, I'm going to call this for code camp

2:43

magic example. Okay. So with that, I'm going to first start with some imports. So I will import,

2:52

you know, I always import NumPy, I always import pandas. And I always import matplotlib.

3:06

And then we'll import other things as we go. So yeah,

3:14

we run that in order to run the cell, you can either click this play button here, or you can

3:19

on my computer, it's just shift enter and that that will run the cell. And here, I'm just going

3:24

to order I'm just going to, you know, let you guys know, okay, this is where I found the data set.

3:30

So I've copied and pasted this actually, but this is just where I found the data set.

3:35

And in order to import that downloaded file that we we got from the computer, we're going to go

3:40

over here to this folder thing. And I am literally just going to drag and drop that file into here.

3:50

Okay. So in order to take a look at, you know, what does this file consist of,

3:55

do we have the labels? Do we not? I mean, we could open it on our computer, but we can also just do

4:00

pandas read CSV. And we can pass in the name of this file.

4:06

And let's see what it returns. So it doesn't seem like we have the label. So let's go back to here.

4:16

I'm just going to make the columns, the column labels, all of these attribute names over here.

4:23

So I'm just going to take these values and make that the column names.

4:29

All right, how do I do that? So basically, I will come back here, and I will create a list called

4:36

calls. And I will type in all of those things. With f size, f conk. And we also have f conk one.

4:50

We have f symmetry, f m three long, f m three trans, f alpha. Let's see, we have f dist and class.

5:09

Okay, great. Now in order to label those as these columns down here in our data frame.

5:16

So basically, this command here just reads some CSV file that you pass in CSV has come about comma

5:22

separated values, and turns that into a pandas data frame object. So now if I pass in a names here,

5:31

then it basically assigns these labels to the columns of this data set. So I'm going to set

5:38

this data frame equal to DF. And then if we call the head is just like, give me the first five things,

5:44

give me the first five things. Now you'll see that we have labels for all of these. Okay.

5:52

All right, great. So one thing that you might notice is that over here, the class labels,

5:57

we have G and H. So if I actually go down here, and I do data frame class unique,

6:07

you'll see that I have either G's or H's, and these stand for gammas or hadrons.

6:11

And our computer is not so good at understanding letters, right? Our computer is really good at

6:17

understanding numbers. So what we're going to do is we're going to convert this to zero for G and

6:23

one for H. So here, I'm going to set this equal to this, whether or not that equals G. And then

6:35

I'm just going to say as type int. So what this should do is convert this entire column,

6:43

if it equals G, then this is true. So I guess that would be one. And then if it's H, it would

6:48

be false. So that would be zero, but I'm just converting G and H to one and zero, it doesn't

6:52

really matter. Like, if G is one and H is zero or vice versa. Let me just take a step back right

7:02

now and talk about this data set. So here I have some data frame, and I have all of these different

7:09

values for each entry. Now this is a you know, each of these is one sample, it's one example,

7:18

it's one item in our data set, it's one data point, all of these things are kind of the same

7:23

thing when I mentioned, oh, this is one example, or this is one sample or whatever. Now, each of

7:29

these samples, they have, you know, one quality for each or one value for each of these labels

7:36

up here, and then it has the class. Now what we're going to do in this specific example is try to

7:41

predict for future, you know, samples, whether the class is G for gamma or H for hadron. And

7:50

that is something known as classification. Now, all of these up here, these are known as our features,

8:00

and features are just things that we're going to pass into our model in order to help us predict

8:05

the label, which in this case is the class column. So for you know, sample zero, I have

8:14

10 different features. So I have 10 different values that I can pass into some model.

8:19

And I can spit out, you know, the class the label, and I know the true label here is G. So this is

8:26

this is actually supervised learning. All right. So before I move on, let me just give you a quick

8:35

little crash course on what I just said. This is machine learning for everyone. Well, the first

8:43

question is, what is machine learning? Well, machine learning is a sub domain of computer science

8:49

that focuses on certain algorithms, which might help a computer learn from data, without a

8:56

programmer being there telling the computer exactly what to do. That's what we call explicit

9:01

programming. So you might have heard of AI and ML and data science, what is the difference between

9:08

all of these. So AI is artificial intelligence. And that's an area of computer science, where the

9:14

goal is to enable computers and machines to perform human like tasks and simulate human behavior.

9:23

Now machine learning is a subset of AI that tries to solve one specific problem and make predictions

9:31

using certain data. And data science is a field that attempts to find patterns and draw insights

9:39

from data. And that might mean we're using machine learning. So all of these fields kind of overlap,

9:45

and all of them might use machine learning. So there are a few types of machine learning.

9:52

The first one is supervised learning. And in supervised learning, we're using labeled inputs.

9:58

So this means whatever input we get, we have a corresponding output label, in order to train

10:05

models and to learn outputs of different new inputs that we might feed our model. So for example,

10:12

I might have these pictures, okay, to a computer, all these pictures are are pixels, they're pixels

10:19

with a certain color. Now in supervised learning, all of these inputs have a label associated with

10:27

them, this is the output that we might want the computer to be able to predict. So for example,

10:32

over here, this picture is a cat, this picture is a dog, and this picture is a lizard.

10:41

Now there's also unsupervised learning. And in unsupervised learning, we use unlabeled data

10:47

to learn about patterns in the data. So here are here are my input data points. Again, they're just

10:57

images, they're just pixels. Well, okay, let's say I have a bunch of these different pictures.

11:05

And what I can do is I can feed all these to my computer. And I might not, you know,

11:09

my computer is not going to be able to say, Oh, this is a cat, dog and lizard in terms of,

11:14

you know, the output. But it might be able to cluster all these pictures, it might say,

11:19

Hey, all of these have something in common. All of these have something in common. And then these

11:26

down here have something in common, that's finding some sort of structure in our unlabeled data.

11:33

And finally, we have reinforcement learning. And reinforcement learning. Well, they usually

11:40

there's an agent that is learning in some sort of interactive environment, based on rewards and

11:46

penalties. So let's think of a dog, we can train our dog, but there's not necessarily, you know,

11:54

any wrong or right output at any given moment, right? Well, let's pretend that dog is a computer.

12:03

Essentially, what we're doing is we're giving rewards to our computer, and tell your computer,

12:08

Hey, this is probably something good that you want to keep doing. Well, computer agent terminology.

12:16

But in this class today, we'll be focusing on supervised learning and unsupervised learning

12:21

and learning different models for each of those. Alright, so let's talk about supervised learning

12:29

first. So this is kind of what a machine learning model looks like you have a bunch of inputs

12:35

that are going into some model. And then the model is spitting out an output, which is our prediction.

12:41

So all these inputs, this is what we call the feature vector. Now there are different types

12:48

of features that we can have, we might have qualitative features. And qualitative means

12:53

categorical data, there's either a finite number of categories or groups. So one example of a

13:01

qualitative feature might be gender. And in this case, there's only two here, it's for the sake of

13:07

the example, I know this might be a little bit outdated. Here we have a girl and a boy, there are

13:13

two genders, there are two different categories. That's a piece of qualitative data. Another

13:19

example might be okay, we have, you know, a bunch of different nationalities, maybe a nationality or

13:25

a nation or a location, that might also be an example of categorical data. Now, in both of

13:33

these, there's no inherent order. It's not like, you know, we can rate us one and France to Japan

13:43

three, etc. Right? There's not really any inherent order built into either of these categorical

13:51

data sets. That's why we call this nominal data. Now, for nominal data, the way that we want

14:00

to feed it into our computer is using something called one hot encoding. So let's say that, you

14:06

know, I have a data set, some of the items in our data, some of the inputs might be from the US,

14:13

some might be from India, then Canada, then France. Now, how do we get our computer to recognize that

14:19

we have to do something called one hot encoding. And basically, one hot encoding is saying, okay,

14:24

well, if it matches some category, make that a one. And if it doesn't just make that a zero.

14:31

So for example, if your input were from the US, you would you might have 1000. India, you know,

14:40

0100. Canada, okay, well, the item representing Canada is one and then France, the item representing

14:46

France is one. And then you can see that the rest are zeros, that's one hot encoding.

14:54

Now, there are also a different type of qualitative feature. So here on the left,

15:00

there are different age groups, there's babies, toddlers, teenagers, young adults,

15:08

adults, and so on, right. And on the right hand side, we might have different ratings. So maybe

15:15

bad, not so good, mediocre, good, and then like, great. Now, these are known as ordinal pieces of

15:26

data, because they have some sort of inherent order, right? Like, being a toddler is a lot closer to

15:33

being a baby than being an elderly person, right? Or good is closer to great than it is to really

15:41

bad. So these have some sort of inherent ordering system. And so for these types of data sets,

15:48

we can actually just mark them from, you know, one to five, or we can just say, hey, for each of these,

15:54

let's give it a number. And this makes sense. Because, like, for example, the thing that I

16:02

just said, how good is closer to great, then good is close to not good at all. Well, four is closer

16:09

to five, then four is close to one. So this actually kind of makes sense. And it'll make sense for the

16:14

computer as well. Alright, there are also quantitative pieces of data and quantitative

16:22

pieces of data are numerical valued pieces of data. So this could be discrete, which means,

16:29

you know, they might be integers, or it could be continuous, which means all real numbers.

16:34

So for example, the length of something is a quantitative piece of data, it's a quantitative

16:40

feature, the temperature of something is a quantitative feature. And then maybe how many

16:46

Easter eggs I collected in my basket, this Easter egg hunt, that is an example of discrete quantitative

16:53

feature. Okay, so these are continuous. And this over here is the screen. So those are the things

17:02

that go into our feature vector, those are our features that we're feeding this model, because

17:08

our computers are really, really good at understanding math, right at understanding numbers,

17:14

they're not so good at understanding things that humans might be able to understand.

17:21

Well, what are the types of predictions that our model can output? So in supervised learning,

17:29

there are some different tasks, there's one classification, and basically classification,

17:35

just saying, okay, predict discrete classes. And that might mean, you know, this is a hot dog,

17:42

this is a pizza, and this is ice cream. Okay, so there are three distinct classes and any other

17:48

pictures of hot dogs, pizza or ice cream, I can put under these labels. Hot dog, pizza, ice cream.

17:56

Hot dog, pizza, ice cream. This is something known as multi class classification. But there's also

18:03

binary classification. And binary classification, you might have hot dog, or not hot dog. So there's

18:10

only two categories that you're working with something that is something and something that's

18:14

isn't binary classification. Okay, so yeah, other examples. So if something has positive or negative

18:23

sentiment, that's binary classification. Maybe you're predicting your pictures of their cats or

18:28

dogs. That's binary classification. Maybe, you know, you are writing an email filter, and you're

18:35

trying to figure out if an email spam or not spam. So that's also binary classification.

18:41

Now for multi class classification, you might have, you know, cat, dog, lizard, dolphin, shark,

18:46

rabbit, etc. We might have different types of fruits like orange, apple, pear, etc. And then

18:53

maybe different plant species. But multi class classification just means more than two. Okay,

18:59

and binary means we're predicting between two things. There's also something called regression

19:06

when we talk about supervised learning. And this just means we're trying to predict continuous

19:11

values. So instead of just trying to predict different categories, we're trying to come up

19:15

with a number that you know, is on some sort of scale. So some examples. So some examples might

19:24

be the price of aetherium tomorrow, or it might be okay, what is going to be the temperature?

19:31

Or it might be what is the price of this house? Right? So these things don't really fit into

19:37

discrete classes. We're trying to predict a number that's as close to the true value as possible

19:43

using different features of our data set. So that's exactly what our model looks like in

19:51

supervised learning. Now let's talk about the model itself. How do we make this model learn?

19:59

Or how can we tell whether or not it's even learning? So before we talk about the models,

20:05

let's talk about how can we actually like evaluate these models? Or how can we tell

20:10

whether something is a good model or bad model? So let's take a look at this data set. So this data

20:19

set has this is from a diabetes, a Pima Indian diabetes data set. And here we have different

20:26

number of pregnancies, different glucose levels, blood pressure, skin thickness, insulin, BMI,

20:32

age, and then the outcome whether or not they have diabetes one for they do zero for they don't.

20:37

So here, all of these are quantitative features, right, because they're all on some scale.

20:48

So each row is a different sample in the data. So it's a different example, it's one person's data,

20:56

and each row represents one person in this data set. Now this column, each column represents a

21:04

different feature. So this one here is some measure of blood pressure levels. And this one

21:11

over here, as we mentioned is the output label. So this one is whether or not they have diabetes.

21:19

And as I mentioned, this is what we would call a feature vector, because these are all of our

21:23

features in one sample. And this is what's known as the target, or the output for that feature

21:33

vector. That's what we're trying to predict. And all of these together is our features matrix x.

21:42

And over here, this is our labels or targets vector y. So I've condensed this to a chocolate

21:51

bar to kind of talk about some of the other concepts in machine learning. So over here,

21:58

we have our x, our features matrix, and over here, this is our label y. So each row of this

22:08

will be fed into our model, right. And our model will make some sort of prediction. And what we do

22:15

is we compare that prediction to the actual value of y that we have in our label data set, because

22:21

that's the whole point of supervised learning is we can compare what our model is outputting to,

22:26

oh, what is the truth, actually, and then we can go back and we can adjust some things. So the next

22:31

iteration, we get closer to what the true value is. So that whole process here, the tinkering that,

22:41

okay, what's the difference? Where did we go wrong? That's what's known as training the model.

22:47

Alright, so take this whole, you know, chunk right here, do we want to really put our entire

22:54

chocolate bar into the model to train our model? Not really, right? Because if we did that, then

23:02

how do we know that our model can do well on new data that we haven't seen? Like, if I were to

23:10

create a model to predict whether or not someone has diabetes, let's say that I just train all my

23:18

data, and I see that all my training data does well, I go to some hospital, I'm like, here's my

23:23

model. I think you can use this to predict if somebody has diabetes. Do we think that would

23:28

be effective or not? Probably not, right? Because we haven't assessed how well our model can

23:41

generalize. Okay, it might do well after you know, our model has seen this data over and over and

23:46

over again. But what about new data? Can our model handle new data? Well, how do we how do we get our

23:54

model to assess that? So we actually break up our whole data set that we have into three different

24:02

types of data sets, we call it the training data set, the validation data set and the testing data

24:07

set. And you know, you might have 60% here 20% and 20% or 80 10 and 10. It really depends on how

24:15

many statistics you have, I think either of those would be acceptable. So what we do is then we feed

24:22

the training data set into our model, we come up with, you know, this might be a vector of predictions

24:28

corresponding with each sample that we put into our model, we figure out, okay, what's the difference

24:36

between our prediction and the true values, this is something known as loss, losses, you know,

24:42

what's the difference here, in some numerical quantity, of course. And then we make adjustments,

24:50

and that's what we call training. Okay. So then, once you know, we've made a bunch of adjustments,

24:58

we can put our validation set through this model. And the validation set is kind of used as a reality

25:06

check during or after training to ensure that the model can handle unseen data still. So every

25:14

single time after we train one iteration, we might stick the validation set in and see, hey, what's

25:19

the loss there. And then after our training is over, we can assess the validation set and ask,

25:25

hey, what's the loss there. But one key difference here is that we don't have that training step,

25:32

this loss never gets fed back into the model, right, that feedback loop is not closed.

25:38

Alright, so let's talk about loss really quickly. So here, I have four different types of models,

25:45

I have some sort of data that's being fed into the model, and then some output. Okay, so this output

25:52

here is pretty far from you know, this truth that we want. And so this loss is going to be high. In

26:02

model B, again, this is pretty far from what we want. So this loss is also going to be high,

26:07

let's give it 1.5. Now this one here, it's pretty close, I mean, maybe not almost, but pretty close

26:15

to this one. So that might have a loss of 0.5. And then this one here is maybe further than this,

26:23

but still better than these two. So that loss might be 0.9. Okay, so which of these model

26:30

performs the best? Well, model C has a smallest loss, so it's probably model C. Okay, now let's

26:40

take model C. After you know, we've come up with these, all these models, and we've seen, okay, model

26:45

C is probably the best model. We take model C, and we run our test set through this model. And this

26:52

test set is used as a final check to see how generalizable that chosen model is. So if I,

27:00

you know, finish training my diabetes data set, then I could run it through some chunk of the

27:05

data and I can say, oh, like, this is how we perform on data that it's never seen before at

27:11

any point during the training process. Okay. And that loss, that's the final reported performance

27:19

of my test set, or this would be the final reported performance of my model. Okay.

27:29

So let's talk about this thing called loss, because I think I kind of just glossed over it,

27:34

right? So loss is the difference between your prediction and the actual, like, label.

27:43

So this would give a slightly higher loss than this. And this would even give a higher loss,

27:50

because it's even more off. In computer science, we like formulas, right? We like formulaic ways

27:57

of describing things. So here are some examples of loss functions and how we can actually come

28:03

up with numbers. This here is known as L one loss. And basically, L one loss just takes the

28:10

absolute value of whatever your you know, real value is, whatever the real output label is,

28:18

subtracts the predicted value, and takes the absolute value of that. Okay. So the absolute

28:26

value is a function that looks something like this. So the further off you are, the greater your losses,

28:35

right in either direction. So if your real value is off from your predicted value by 10,

28:42

then your loss for that point would be 10. And then this sum here just means, hey,

28:47

we're taking all the points in our data set. And we're trying to figure out the sum of how far

28:53

everything is. Now, we also have something called L two loss. So this loss function is quadratic,

29:01

which means that if it's close, the penalty is very minimal. And if it's off by a lot,

29:08

then the penalty is much, much higher. Okay. And this instead of the absolute value, we just square

29:15

the the difference between the two. Now, there's also something called binary cross entropy loss.

29:26

It looks something like this. And this is for binary classification, this this might be the

29:32

loss that we use. So this loss, you know, I'm not going to really go through it too much.

29:38

But you just need to know that loss decreases as the performance gets better. So there are some

29:47

other measures of accurate or performance as well. So for example, accuracy, what is accuracy?

29:55

So let's say that these are pictures that I'm feeding my model, okay. And these predictions

30:02

might be apple, orange, orange, apple, okay, but the actual is apple, orange, apple, apple. So

30:12

three of them were correct. And one of them was incorrect. So the accuracy of this model is

30:17

three quarters or 75%. Alright, coming back to our colab notebook, I'm going to close this a little

30:25

bit. Again, we've imported stuff up here. And we've already created our data frame right here. And

30:33

this is this is all of our data. This is what we're going to use to train our models. So down here,

30:40

again, if we now take a look at our data set, you'll see that our classes are now zeros and ones.

30:49

So now this is all numerical, which is good, because our computer can now understand that.

30:53

Okay. And you know, it would probably be a good idea to maybe kind of plot, hey, do these things

31:00

have anything to do with the class. So here, I'm going to go through all the labels. So for label

31:10

in the columns of this data frame. So this just gets me the list. Actually, we have the list,

31:15

right? It's called so let's just use that might be less confusing of everything up to the last

31:20

thing, which is the class. So I'm going to take all these 10 different features. And I'm going

31:26

to plot them as a histogram. So and now I'm going to plot them as a histogram. So basically, if I

31:37

take that data frame, and I say, okay, for everything where the class is equal to one, so these are all

31:45

of our gammas, remember, now, for that portion of the data frame, if I look at this label, so now

31:55

these, okay, what this part here is saying is, inside the data frame, get me everything where

32:03

the class is equal to one. So that's all all of these would fit into that category, right?

32:09

And now let's just look at the label column. So the first label would be f length, which would

32:14

be this column. So this command here is getting me all the different values that belong to class one

32:20

for this specific label. And that's exactly what I'm going to put into the histogram. And now I'm

32:27

just going to tell you know, matplotlib make the color blue, make this label this as you know, gamma

32:37

set alpha, why do I keep doing that, alpha equal to 0.7. So that's just like the transparency.

32:43

And then I'm going to set density equal to true, so that when we compare it to

32:50

the hadrons here, we'll have a baseline for comparing them. Okay, so the density being true

32:56

just basically normalizes these distributions. So you know, if you have 200 in of one type,

33:05

and then 50 of another type, well, if you drew the histograms, it would be hard to compare because

33:12

one of them would be a lot bigger than the other, right. But by normalizing them, we kind of are

33:17

distributing them over how many samples there are. Alright, and then I'm just going to put a title

33:24

on here and make that the label, the y label. So because it's density, the y label is probability.

33:32

And the x label is just going to be the label.

33:36

What is going on. And I'm going to include a legend and PLT dot show just means okay, display

33:44

the plot. So if I run that, just be up to the last item. So we want a list, right, not just the last

33:54

item. And now we can see that we're plotting all of these. So here we have the length. Oh, and I

34:02

made this gamma. So this should be hadron. Okay, so the gammas in blue, the hadrons are in red. So

34:11

here we can already see that, you know, maybe if the length is smaller, it's probably more likely

34:16

to be gamma, right. And we can kind of you know, these all look somewhat similar. But here, okay,

34:24

clearly, if there's more asymmetry, or if you know, this asymmetry measure is larger, then it's

34:34

probably hadron. Okay, oh, this one's a good one. So f alpha seems like hadrons are pretty evenly

34:44

distributed. Whereas if this is smaller, it looks like there's more gammas in that area.

34:48

Okay, so this is kind of what the data that we're working with, we can kind of see what's going on.

34:55

Okay, so the next thing that we're going to do here is we are going to create our train,

35:03

our validation, and our test data sets. I'm going to set train valid and test to be equal to

35:12

this. So NumPy dot split, I'm just splitting up the data frame. And if I do this sample,

35:20

where I'm sampling everything, this will basically shuffle my data. Now, if I I want to pass in where

35:29

exactly I'm splitting my data set, so the first split is going to be maybe at 60%. So I'm going

35:38

to say 0.6 times the length of this data frame. So and then cast that 10 integer, that's going

35:44

to be the first place where you know, I cut it off, and that'll be my training data. Now, if I

35:50

then go to 0.8, this basically means everything between 60% and 80% of the length of the data

35:57

set will go towards validation. And then, like everything from 80 to 100, I'm going to pass

36:03

my test data. So I can run that. And now, if we go up here, and we inspect this data, we'll see that

36:12

these columns seem to have values in like the 100s, whereas this one is 0.03. Right? So the scale of

36:20

all these numbers is way off. And sometimes that will affect our results. So I'm going to run this

36:28

is way off. And sometimes that will affect our results. So one thing that we would want to do

36:35

is scale these so that they are, you know, so that it's now relative to maybe the mean and the

36:46

standard deviation of that specific column. I'm going to create a function called scale data set.

36:54

And I'm going to pass in the data frame. And that's what I'll do for now. Okay, so the x values are

37:04

going to be, you know, I take the data frame. And let's assume that the columns are going to be,

37:14

you know, that the label will always be the last thing in the data frame. So what I can do is say

37:20

data frame, dot columns all the way up to the last item, and get those values. Now for my y,

37:30

well, it's the last column. So I can just do this, I can just index into that last column,

37:34

and then get those values. Now, in, so I'm actually going to import something known as

37:46

the standard scalar from sk learn. So if I come up here, I can go to sk learn dot pre processing.

37:56

And I'm going to import standard scalar, I have to run that cell, I'm going to come back down here.

38:04

And now I'm going to create a scalar and use that skip or so standard scalar.

38:10

And with the scalar, what I can do is actually just fit and transform x. So here, I can say x

38:21

is equal to scalar dot fit, fit, transform x. So what that's doing is saying, okay, take x and

38:31

fit the standard scalar to x, and then transform all those values. And what would it be? And that's

38:36

going to be our new x. Alright. And then I'm also going to just create, you know, the whole data as

38:45

one huge 2d NumPy array. And in order to do that, I'm going to call H stack. So H stack is saying,

38:53

okay, take an array, and another array and horizontally stack them together. That's what

38:58

the H stands for. So by horizontally stacked them together, just like put them side by side,

39:03

okay, not on top of each other. So what am I stacking? Well, I have to pass in something

39:10

so that it can stack x and y. And now, okay, so NumPy is very particular about dimensions,

39:20

right? So in this specific case, our x is a two dimensional object, but y is only a one dimensional

39:27

thing, it's only a vector of values. So in order to now reshape it into a 2d item, we have to call

39:35

NumPy dot reshape. And we can pass in the dimensions of its reshape. So if I pass in negative

39:45

one comma one, that just means okay, make this a 2d array, where the negative one just means infer

39:51

what what this dimension value would be, which ends up being the length of y, this would be the

39:56

same as literally doing this. But the negative one is easier because we're making the computer

40:01

do the hard work. So if I stack that, I'm going to then return the data x and y. Okay. So one more

40:13

thing is that if we go into our training data set, okay, again, this is our training data set.

40:18

And we get the length of the training data set. But where the training data sets class is one,

40:28

so remember that this is the gammas. And then if we print that, and we do the same thing, but zero,

40:39

we'll see that, you know, there's around 7000 of the gammas, but only around 4000 of the hadrons.

40:49

So that might actually become an issue. And instead, what we want to do is we want to oversample

40:57

our our training data set. So that means that we want to increase the number of these values,

41:06

so that these kind of match better. And surprise, surprise, there is something that we can import

41:13

that will help us do that. It's so I'm going to go to from in the learn dot oversampling. And I'm

41:23

going to import this random oversampler, run that cell, and come back down here. So I will actually

41:31

add in this parameter called oversample, and set that to false for default. And if I do want to

41:43

oversample, then what I'm going to do, and by oversample, so if I do want to oversample,

41:51

then I'm going to create this ROS and set it equal to this random oversampler. And then for x and y,

41:59

I'm just going to say, okay, just fit and resample x and y. And what that's doing is saying, okay,

42:06

take more of the less class. So take take the less class and keep sampling from there to increase

42:15

the size of our data set of that smaller class so that they now match. So if I do this, and I scale

42:24

data set, and I pass in the training data set where oversample is true. So this let's say this

42:33

is train and then x train, y train. Oops, what's going on? These should be columns. So basically,

42:48

what I'm doing now is I'm just saying, okay, what is the length of y train? Okay, now it's

42:55

14,800, whatever. And now let's take a look at how many of these are type one. So actually,

43:05

we can just sum that up. And then we'll also see that if we instead switch the label and ask how

43:12

many of them are the other type, it's the same value. So now these have been evenly, you know,

43:19

rebalanced. Okay, well, okay. So here, I'm just going to make this the validation data set. And

43:31

then the next one, I'm going to make this the test data set. Alright, and we're actually going to

43:39

switch oversample here to false. Now, the reason why I'm switching that to false is because my

43:46

validation and my test sets are for the purpose of you know, if I have data that I haven't seen yet,

43:51

how does my sample perform on those? And I don't want to oversample for that right now. Like,

43:59

I don't care about balancing those I'm, I want to know if I have a random set of data that's

44:06

unlabeled, can I trust my model, right? So that's why I'm not oversampling. I run that. And again,

44:16

what is going on? Oh, it's because we already have this train. So I have to go come up here and split

44:23

that data frame again. And now let's run these. Okay. So now we have our data properly formatted.

44:32

And we're going to move on to different models now. And I'm going to tell you guys a little bit

44:37

about each of these models. And then I'm going to show you how we can do that in our code. So the

44:43

first model that we're going to learn about is KNN or K nearest neighbors. Okay, so here, I've

44:49

already drawn a plot on the y axis, I have the number of kids that a family might have. And then

44:57

on the x axis, I have their income in terms of 1000s per year. So, you know, if if someone's

45:07

making 40,000 a year, that's where this would be. And if somebody making 320, that's where that

45:12

would be somebody has zero kids, it'd be somewhere along this axis. Somebody has five, it'd be

45:18

somewhere over here. Okay. And now I have these plus signs and these minus signs on here. So what

45:28

I'm going to represent here is the plus sign means that they own a car. And the minus sign is going

45:42

to represent no car. Okay. So your initial thought should be okay, I think this is binary

45:49

classification because all of our points all of our samples have labels. So this is a sample with

46:00

the plus label. And this here is another sample with the minus label. This is an abbreviation for

46:13

width that I'll use. Alright, so we have this entire data set. And maybe around half the people

46:20

own a car and maybe around half the people don't own a car. Okay, well, what if I had some new

46:29

point, let me use choose a different color, I'll use this nice green. Well, what if I have a new

46:35

point over here? So let's say that somebody makes 40,000 a year and has two kids. What do we think

46:42

that would be? Well, just logically looking at this plot, you might think, okay, it seems like

46:52

they wouldn't have a car, right? Because that kind of matches the pattern of everybody else around

46:57

them. So that's a whole concept of this nearest neighbors is you look at, okay, what's around you.

47:06

And then you're basically like, okay, I'm going to take the label of the majority that's around me.

47:11

So the first thing that we have to do is we have to define a distance function. And a lot of times

47:17

in, you know, 2d plots like this, our distance function is something known as Euclidean distance.

47:25

And Euclidean distance is basically just this straight line distance like this. Okay. So this

47:45

would be the Euclidean distance, it seems like there's this point, there's this point, there's

47:54

that point, etc. So the length of this line, this green line that I just drew, that is what's known

48:00

as Euclidean distance. If we want to get technical with that, this exact formula is the distance here,

48:10

let me zoom in. The distance is equal to the square root of one point x minus the other points x

48:20

squared plus extend that square root, the same thing for y. So y one of one minus y two of the

48:29

other squared. Okay, so we're basically trying to find the length, the distances, the difference

48:36

between x and y, and then square each of those sum it up and take the square root. Okay, so I'm

48:43

going to erase this so it doesn't clutter my drawing. But anyways, now going back to this plot,

48:53

so here in the nearest neighbor algorithm, we see that there is a K, right? And this K is basically

49:03

telling us, okay, how many neighbors do we use in order to judge what the label is? So usually,

49:09

we use a K of maybe, you know, three or five, depends on how big our data set is. But here,

49:16

I would say, maybe a logical number would be three or five. So let's say that we take K to be equal

49:25

to three. Okay, well, of this data point that I drew over here, let me use green to highlight this.

49:34

Okay, so of this data point that I drew over here, it looks like the three closest points are definitely

49:40

this one, this one. And then this one has a length of four. And this one seems like it'd be a little

49:50

bit further than four. So actually, this would be these would be our three points. Well, all those

49:57

points are blue. So chances are, my prediction for this point is going to be blue, it's going to be

50:05

probably don't have a car. All right, now what if my point is somewhere? What if my point is

50:14

somewhere over here, let's say that a couple has four kids, and they make 240,000 a year. All right,

50:26

well, now my closest points are this one, probably a little bit over that one. And then this one,

50:34

right? Okay, still all pluses. Well, this one is more than likely to be plus. Right? Now,

50:45

let me get rid of some of these just so that it looks a little bit more clear. All right,

50:55

let's go through one more. What about a point that might be right here? Okay, let's see. Well,

51:06

definitely this is the closest, right? This one's also closest. And then it's really close between

51:16

the two of these. But if we actually do the mathematics, it seems like if we zoom in,

51:22

this one is right here. And this one is in between these two. So this one here is actually shorter

51:30

than this one. And that means that that top one is the one that we're going to take. Now,

51:37

what is the majority of the points that are close by? Well, we have one plus here, we have one plus

51:45

here, and we have one minus here, which means that the pluses are the majority. And that means

51:52

that this label is probably somebody with a car. Okay. So this is how K nearest neighbors would

52:04

work. It's that simple. And this can be extrapolated to further dimensions to higher dimensions. You

52:13

know, if you have here, we have two different features, we have the income, and then we have

52:19

the number of kids. But let's say we have 10 different features, we can expand our distance

52:25

function so that it includes all 10 of those dimensions, we take the square root of everything,

52:31

and then we figure out which one is the closest to the point that we desire to classify. Okay. So

52:39

that's K nearest neighbors. So now we've learned about K nearest neighbors. Let's see how we would

52:45

be able to do that within our code. So here, I'm going to label the section K nearest neighbors.

52:51

And we're actually going to use a package from SK learn. So the reason why we, you know, use these

52:59

packages and so that we don't have to manually code all these things ourselves, because it would

53:04

be really difficult. And chances are the way that we would code it, either would have bugs,

53:08

or it'd be really slow, or I don't know a whole bunch of issues. So what we're going to do is

53:13

hand it off to the pros. From here, I can say, okay, from SK learn, which is this package dot

53:20

neighbors, I'm going to import K neighbors classifier, because we're classifying. Okay,

53:27

so I run that. And our KNN model is going to be this K neighbors classifier. And we can pass in

53:38

a parameter of how many neighbors, you know, we want to use. So first, let's see what happens if

53:43

we just use one. So now if I do K, and then model dot fit, I can pass in my x training set and my

53:52

weight y train data. Okay. So that effectively fits this model. And let's get all the predictions. So

54:03

why can and I guess yeah, let's do y predictions. And my y predictions are going to be cannon model

54:11

dot predict. So let's use the test set x test. Okay. Alright, so if I call y predict, you'll see

54:24

that we have those. But if I get my truth values for that test set, you'll see that this is what

54:29

we actually do. So just looking at this, we got five out of six of them. Okay, great. So let's

54:33

actually take a look at something called the classification report that's offered by SK learn.

54:39

So if I go to from SK learn dot metrics, import classification report, what I can actually do is

54:49

say, hey, print out this classification report for me. And let's check, you know, I'm giving you the

54:57

y test and the y prediction. We run this and we see we get this whole entire chart. So I'm going

55:04

to tell you guys a few things on this chart. Alright, this accuracy is 82%, which is actually

55:10

pretty good. That's just saying, hey, if we just look at, you know, what each of these new points,

55:15

what it's closest to, then we actually get an 82% accuracy, which means how many do we get right

55:23

versus how many total are there. Now, precision is saying, okay, you might see that we have it

55:29

for class one, or class zero and class one. What precision is saying was, let's go to this Wikipedia

55:36

diagram over here, because I actually kind of like this diagram. So here, this is our entire data set.

55:42

And on the left over here, we have everything that we know is positive. So everything that is

55:48

actually truly positive, that we've labeled positive in our original data set. And over here,

55:54

this is everything that's truly negative. Now in the circle, we have things that are positive that

56:01

were labeled positive by our model. On the left here, we have things that are truly positive,

56:08

because you know, this side is the positive side and the side is the negative side. So these are

56:13

truly positive. Whereas all these ones out here, well, they should have been positive, but they

56:18

are labeled as negative. And in here, these are the ones that we've labeled positive, but they're

56:24

actually negative. And out here, these are truly negative. So precision is saying, okay, out of all

56:33

the ones we've labeled as positive, how many of them are true positives? And recall is saying,

56:40

okay, out of all the ones that we know are truly positive, how many do we actually get right? Okay,

56:47

so going back to this over here, our precision score, so again, precision, out of all the ones

56:55

that we've labeled as the specific class, how many of them are actually that class, it's 7784%. Now,

57:03

recall how out of all the ones that are actually this class, how many of those that we get, this

57:09

is 68% and 89%. Alright, so not too shabby, we can clearly see that this recall and precision for

57:18

like this, the class zero is worse than class one. Right? So that means for hadron, it's worked for

57:24

hadrons and for our gammas. This f1 score over here is kind of a combination of the precision and

57:30

recall score. So we're actually going to mostly look at this one because we have an unbalanced

57:35

test data set. So here we have a measure of 72 and 87 or point seven two and point eight seven,

57:43

which is not too shabby. All right. Well, what if we, you know, made this three. So we actually see

57:55

that, okay, so what was it originally with one? We see that our f1 score, you know, is now it was

58:04

point seven two and then point eight seven. And then our accuracy was 82%. So if I change that to

58:10

three. Alright, so we've kind of increased zero at the cost of one and then our overall accuracy

58:20

is 81. So let's actually just make this five. Alright, so you know, again, very similar numbers,

58:28

we have 82% accuracy, which is pretty decent for a model that's relatively simple. Okay,

58:35

the next type of model that we're going to talk about is something known as naive Bayes. Now,

58:42

in order to understand the concepts behind naive Bayes, we have to be able to understand

58:48

conditional probability and Bayes rule. So let's say I have some sort of data set that's shown in

58:55

this table right here. People who have COVID are over here in this red row. And people who do not

59:03

have COVID are down here in this green row. Now, what about the COVID test? Well, people who have

59:09

tested positive are over here in this column. And people who have tested negative are over here in

59:18

this column. Okay. Yeah, so basically, our categories are people who have COVID and test positive,

59:25

people who don't have COVID, but test positive, so a false false positive, people who have COVID

59:32

and test negative, which is a false negative, and people who don't have COVID and test negative,

59:38

which good means you don't have COVID. Okay, so let's make this slightly more legible. And here,

59:48

in the margins, I've written down the sums of whatever it's referring to. So this here is the

59:55

sum of this entire row. And this here might be the sum of this column over here. Okay. So the first

1:00:05

question that I have is, what is the probability of having COVID given that you have a positive

1:00:11

test? And in probability, we write that out like this. So the probability of COVID given, so this

1:00:21

line, that vertical line means given that, you know, some condition, so given a positive test,

1:00:29

okay, so what is the probability of having COVID given a positive test? So what this is asking is

1:00:39

saying, okay, let's go into this condition. So the condition of having a positive test, that is this

1:00:48

slice of the data, right? That means if you're in this slice of data, you have a positive test. So

1:00:53

given that we have a positive test, given in this condition, in this circumstance, we have a positive

1:00:59

test. So what's the probability that we have COVID? Well, if we're just using this data, the number

1:01:05

of people that have COVID is 531. So I'm gonna say that there's 531 people that have COVID. And then

1:01:15

now we divide that by the total number of people that have a positive test, which is 551. Okay,

1:01:24

so that's the probability and doing a quick division, we get that this is equal to around

1:01:34

96.4%. So according to this data set, which is data that I made up off the top of my head, so it's

1:01:43

not actually real COVID data. But according to this data, the probability of having COVID given

1:01:50

that you tested positive is 96.4%. Alright, now with that, let's talk about Bayes rule, which is

1:02:02

this section here. Let's ignore this bottom part for now. So Bayes rule is asking, okay, what is

1:02:10

the probability of some event A happening, given that B happened. So this, we already know has

1:02:18

happened. This is our condition, right? Well, what if we don't have data for that, right? Like, what

1:02:26

if we don't know what the probability of A given B is? Well, Bayes rule is saying, okay, well, you

1:02:31

can actually go and calculate it, as long as you have a probability of B given A, the probability

1:02:36

of A and the probability of B. Okay. And this is just a mathematical formula for that. Alright,

1:02:43

so here we have Bayes rule. And let's actually see Bayes rule in action. Let's use it on an example.

1:02:51

So here, let's say that we have some disease statistics, okay. So not COVID different disease.

1:02:58

And we know that the probability of obtaining a false positive is 0.05 probability of obtaining a

1:03:05

false negative is 0.01. And the probability of the disease is 0.1. Okay, what is the probability of

1:03:12

the disease given that we got a positive test? Hmm, how do we even go about solving this? So

1:03:20

what what do I mean by false positive? What's a different way to rewrite that? A false positive

1:03:26

is when you test positive, but you don't actually have the disease. So this here is a probability

1:03:32

that you have a positive test given no disease, right? And similarly for the false negative,

1:03:42

it's a probability that you test negative given that you actually have the disease. So if I put

1:03:47

that into a chart, for example, and this might be my positive and negative tests, and this might

1:03:58

be my diseases, disease and no disease. Well, the probability that I test positive, but actually

1:04:07

have no disease, okay, that's 0.05 over here. And then the false negatives up here for 0.01. So I'm

1:04:14

testing negative, but I don't actually have the disease. This so the probability that you test

1:04:20

positive, and you don't have the disease, plus a probability that you test negative, given that you

1:04:25

don't have the disease, that should sum up to one. Okay, because if you don't have the disease,

1:04:30

then you should have some probability that you're testing positive and some probability that you're

1:04:34

testing negative. But that probability, in total should be one. So that means that the probability

1:04:43

negative and no disease, this should be the reciprocal, this should be the opposite. So it

1:04:47

should be 0.95 because it's one minus whatever this probability is. And then similarly, oops,

1:04:59

up here, this should be 0.99 because the probability that we, you know,

1:05:06

test negative and have the disease plus the probability that we test positive and have the

1:05:10

disease should equal one. So this is our probability chart. And now, this probability of disease

1:05:16

being point 0.1 just means I have 10% probability of actually of having the disease, right? Like,

1:05:23

in the general population, the probability that I have the disease is 0.1. Okay, so what is the

1:05:30

probability that I have the disease given that I got a positive test? Well, remember that we

1:05:37

can write this out in terms of Bayes rule, right? So if I use this rule up here, this is the

1:05:43

probability of a positive test given that I have the disease times the probability of the disease

1:05:52

divided by the probability of the evidence, which is my positive test.

1:06:00

Alright, now let's plug in some numbers for that. The probability of having a positive test given

1:06:05

that I have the disease is 0.99. And then the probability that I have the disease is this value

1:06:13

over here 0.1. Okay. And then the probability that I have a positive test at all should be okay,

1:06:26

what is the probability that I have a positive test given that I actually have the disease

1:06:29

and then having having the disease. And then the other case, where the probability of me having a

1:06:37

negative test given or sorry, positive test giving no disease times the probability of not actually

1:06:45

having a disease. Okay, so I can expand that probability of having a positive test out into

1:06:52

these two different cases, I have a disease, and then I don't. And then what's the probability of

1:06:58

having positive tests in either one of those cases. So that expression would become 0.99 times 0.1

1:07:09

plus 0.05. So that's the probability that I'm testing positive, but don't have the disease.

1:07:16

And the times the probability that I don't actually have the disease. So that's one minus

1:07:20

0.1 probability that the population doesn't have the disease is 90%. So 0.9. And let's do that

1:07:29

multiplication. And I get an answer of 0.6875 or 68.75%. Okay. All right, so we can actually expand

1:07:48

that we can expand Bayes rule and apply it to classification. And this is what we call naive

1:07:56

base. So first, a little terminology. So the posterior is this over here, because it's asking,

1:08:04

Hey, what is the probability of some class CK? So by CK, I just mean, you know, the different

1:08:12

categories, so C for category or class or whatever. So category one might be cats, category two,

1:08:19

dogs, category three, lizards, all the way, we have k categories, k is just some number. Okay.

1:08:27

So what is the probability of having of this specific sample x, so this is our feature vector

1:08:36

of this one sample. What is the probability of x fitting into category 123 for whatever, right,

1:08:44

so that that's what this is asking, what is the probability that, you know, it's actually from

1:08:49

this class, given all this evidence that we see the x's. So the likelihood is this quantity over

1:08:59

here, it's saying, Okay, well, given that, you know, assume, assume we are, assume that this

1:09:07

class is class CK, okay, assume that this is a category. Well, what is the likelihood of

1:09:13

actually seeing x, all these different features from that category. And then this here is the

1:09:21

prior. So like in the entire population of things, what are the probabilities? What is the

1:09:26

probability of this class in general? Like if I have, you know, in my entire data set, what is the

1:09:32

percentage? What is the chance that this image is a cat? How many cats do I have? Right. And then this

1:09:40

down here is called the evidence because what we're trying to do is we're changing our prior,

1:09:47

we're creating this new posterior probability built upon the prior by using some sort of evidence,

1:09:54

right? And that evidence is a probability of x. So that's some vocab. And this here

1:10:05

is a rule for naive Bayes. Whoa, okay, let's digest that a little bit. Okay. So what is

1:10:15

let me use a different color. What is this side of the equation asking? It's asking,

1:10:21

what is the probability that we are in some class K, CK, given that, you know, this is my first

1:10:28

input, this is my second input, this is, you know, my third, fourth, this is my nth input. So let's

1:10:33

say that our classification is, do we play soccer today or not? Okay, and let's say our x's are,

1:10:41

okay, is it how much wind is there? How much rain is there? And what day of the week is it? So let's

1:10:49

So let's say that it's raining, it's not windy, but it's Wednesday, do we play soccer? Do we not?

1:10:56

So let's use Bayes rule on this. So this here

1:11:06

is equal to the probability of x one, x two, all these joint probabilities, given class K

1:11:13

times the probability of that class, all over the probability of this evidence.

1:11:24

Okay. So what is this fancy symbol over here, this means proportional to

1:11:33

so how our equal sign means it's equal to this like little squiggly sign means that this is

1:11:38

proportional to okay, and this denominator over here, you might notice that it has no impact on

1:11:48

the class like this, that number doesn't depend on the class, right? So this is going to be constant

1:11:53

for all of our different classes. So what I'm going to do is make things simpler. So I'm just

1:11:59

going to say that this probability x one, x two, all the way to x n, this is going to be proportional

1:12:07

to the numerator, I don't care about the denominator, because it's the same for every

1:12:10

single class. So this is proportional to x one, x two, x n given class K times the probability of

1:12:20

that class. Okay. All right. So in naive Bayes, the point of it being naive, is that we're actually

1:12:32

this joint probability, we're just assuming that all of these different things

1:12:36

are all independent. So in my soccer example, you know, the probability that we're playing soccer,

1:12:44

or the probability that, you know, it's windy, and it's rainy, and, and it's Wednesday, all these

1:12:50

things are independent, we're assuming that they're independent. So that means that I can

1:12:56

actually write this part of the equation here as this. So each term in here, I can just multiply

1:13:07

all of them together. So the probability of the first feature, given that it's class K,

1:13:14

times the probability of the second feature and given this problem, like class K all the way up

1:13:20

all the way up until, you know, the nth feature of given that it's class K. So this expands to

1:13:30

all of this. All right, which means that this here is now proportional to the thing that we just

1:13:39

expanded times this. So I'm going to write that out. So the probability of that class.

1:13:47

And I'm actually going to use this symbol. So what this means is it's a huge multiplication,

1:13:54

it means multiply everything to the right of this. So this probability x, given some class K,

1:14:04

but do it for all the i's. So I, what is I, okay, we're going to go from the first

1:14:11

the first x i all the way to the nth. So that means for every single i, we're just multiplying

1:14:19

these probabilities together. And that's where this up here comes from. So to wrap this up,

1:14:27

oops, this should be a line to wrap this up in plain English. Basically, what this is saying

1:14:31

is a probability that you know, we're in some category, given that we have all these different

1:14:37

features is proportional to the probability of that class in general, times the probability of

1:14:44

each of those features, given that we're in this one class that we're testing. So the probability

1:14:51

of it, you know, of us playing soccer today, given that it's rainy, not windy, and and it's

1:14:59

Wednesday, is proportional to Okay, well, what is what is the probability that we play soccer

1:15:04

anyways, and then times the probability that it's rainy, given that we're playing soccer,

1:15:10

times the probability that it's not windy, given that we're playing soccer. So how many times are

1:15:15

we playing soccer when it's windy, how you know, and then how many times are what's the probability

1:15:21

that's Wednesday, given that we're playing soccer. Okay. So how do we use this in order to make a

1:15:30

classification. So that's where this comes in our y hat, our predicted y is going to be equal to

1:15:39

something called the arg max. And then this expression over here, because we want to take

1:15:45

the arg max. Well, we want. So okay, if I write out this, again, this means the probability of

1:15:55

being in some class CK given all of our evidence. Well, we're going to take the K that maximizes

1:16:06

this expression on the right. That's what arc max means. So if K is in zero, oops,

1:16:14

one through K, so this is how many categories are, we're going to go through each K. And we're going

1:16:21

to solve this expression over here and find the K that makes that the largest. Okay. And remember

1:16:32

that instead of writing this, we have now a formula, thanks to Bayes rule for helping us

1:16:40

approximate that right in something that maybe we can we maybe we have like the evidence for that,

1:16:47

we have the answers for that based on our training set. So this principle of going through each of

1:16:54

these and finding whatever class whatever category maximizes this expression on the right,

1:17:00

this is something known as MAP for short, or maximum a posteriori.

1:17:12

Pick the hypothesis. So pick the K that is the most probable so that we minimize the probability

1:17:20

of misclassification. Right. So that is MAP. That is naive Bayes. Back to the notebook. So

1:17:31

just like how I imported k nearest neighbor, k neighbors classifier up here for naive Bayes,

1:17:38

I can go to SK learn naive Bayes. And I can import Gaussian naive Bayes.

1:17:46

Right. And here I'm going to say my naive Bayes model is equal. This is very similar to what we

1:17:52

had above. And I'm just going to say with this model, we are going to fit x train and y train.

1:18:06

All right, just like above. So this, I might actually, so I'm going to set that. And

1:18:19

exactly, just like above, I'm going to make my prediction. So here, I'm going to instead use my

1:18:26

naive Bayes model. And of course, I'm going to run the classification report again. So I'm actually

1:18:35

just going to put these in the same cell. But here we have the y the new y prediction and then y test

1:18:40

is still our original test data set. So if I run this, you'll see that. Okay, what's going on here,

1:18:49

we get worse scores, right? Our precision, for all of them, they look slightly worse. And our,

1:18:58

you know, for our precision, our recall, our f1 score, they look slightly worse for all the different

1:19:04

categories. And our total accuracy, I mean, it's still 72%, which is not too shabby. But it's still

1:19:11

72%. Okay. Which, you know, is not not that great. Okay, so let's move on to logistic regression.

1:19:22

Here, I've drawn a plot, I have y. So this is my label on one axis. And then this is maybe one of

1:19:29

my features. So let's just say I only have one feature in this case, text zero, right? Well,

1:19:36

we see that, you know, I have a few of one class type down here. And we know it's one class type

1:19:44

because it's zero. And then we have our other class type one up here. And then we have our

1:19:51

y. Okay. So many of you guys are familiar with regression. So let's start there. If I were to

1:19:58

draw a regression line through this, it might look something like like this. Right? Well, this

1:20:10

doesn't seem to be a very good model. Like, why would we use this specific line to predict why?

1:20:16

Right? It's, it's iffy. Okay. For example, we might say, okay, well, it seems like, you know,

1:20:27

everything from here downwards would be one class type in here, upwards would be another class type.

1:20:34

But when you look at this, you're just you, you visually can tell, okay, like, that line doesn't

1:20:41

make sense. Things are not those dots are not along that line. And the reason is because we

1:20:46

are doing classification, not regression. Okay. Well, first of all, let's start here, we know that

1:20:55

this model, if we just use this line, it equals m x. So whatever this let's just say it's x plus b,

1:21:04

which is the y intercept, right? And m is the slope. But when we use a linear regression,

1:21:10

is it actually y hat? No, it's not right. So when we're working with linear regression,

1:21:15

what we're actually estimating in our model is a probability, what's a probability between zero

1:21:20

and one, that is class zero or class one. So here, let's rewrite this as p equals m x plus b.

1:21:32

Okay, well, m x plus b, that can range, you know, from negative infinity to infinity,

1:21:39

right? For any for any value of x, it goes from negative infinity to infinity.

1:21:44

But probability, we know probably one of the rules of probability is that probability has to stay

1:21:49

between zero and one. So how do we fix this? Well, maybe instead of just setting the probability

1:21:57

equal to that, we can set the odds equal to this. So by that, I mean, okay, let's do probability

1:22:03

divided by one minus the probability. Okay, so now becomes this ratio. Now this ratio is allowed to

1:22:10

take on infinite values. But there's still one issue here. Let me move this over a bit.

1:22:18

The one issue here is that m x plus b, that can still be negative, right? Like if you know,

1:22:24

I have a negative slope, if I have a negative b, if I have some negative x's in there, I don't know,

1:22:28

but that can be that's allowed to be negative. So how do we fix that? We do that by actually taking

1:22:36

the log of the odds. Okay. So now I have the log of you know, some probability divided by one minus

1:22:47

the probability. And now that is on a range of negative infinity to infinity, which is good

1:22:54

because the range of log should be negative infinity to infinity. Now how do I solve for P

1:23:00

the probability? Well, the first thing I can do is take, you know, I can remove the log by taking

1:23:08

the not the e to the whatever is on both sides. So that gives me the probability

1:23:16

over the one minus the probability is now equal to e to the m x plus b. Okay. So let's multiply

1:23:27

that out. So the probability is equal to one minus probability e to the m x plus b. So P is equal to

1:23:39

e to the m x plus b minus P times e to the m x plus b. And now we have we can move like terms to

1:23:49

one side. So if I do P, so basically, I'm moving this over, so I'm adding P. So now P one plus e

1:23:58

to the m x plus b is equal to e to the m x plus b and let me change this parentheses make it a

1:24:11

little bigger. So now my probability can be e to the m x plus b divided by one plus e to the m x plus b.

1:24:22

Okay, well, let me just rewrite this really quickly, I want a numerator of one on top.

1:24:33

Okay, so what I'm going to do is I'm going to multiply this by negative m x plus b,

1:24:40

and then also the bottom by negative m x plus b, and I'm allowed to do that because

1:24:45

this over this is one. So now my probability is equal to one over

1:24:54

one plus e to the negative m x plus b. And now why did I rewrite it like that?

1:25:01

It's because this is actually a form of a special function, which is called the sigmoid

1:25:07

function. And for the sigmoid function, it looks something like this. So s of x sigmoid, you know,

1:25:20

that some x is equal to one over one plus e to the negative x. So essentially, what I just did up here

1:25:30

is rewrite this in some sigmoid function, where the x value is actually m x plus b.

1:25:38

So maybe I'll change this to y just to make that a bit more clear, it doesn't matter what

1:25:42

the variable name is. But this is our sigmoid function. And visually, what our sigmoid function

1:25:50

looks like is it goes from zero. So this here is zero to one. And it looks something like this

1:26:01

curved s, which I didn't draw too well. Let me try that again. It's hard to draw

1:26:10

something if I can draw this right. Like that. Okay, so it goes in between zero and one.

1:26:19

And you might notice that this form fits our shape up here.

1:26:29

Oops, let's draw it sharper. But if it's our shape up there a lot better, right?

1:26:37

Alright, so that is what we call logistic regression, we're basically trying to fit our data

1:26:44

to the sigmoid function. Okay. And when we only have, you know, one data point, so if we only have

1:26:56

one feature x, and that's what we call simple logistic regression. But then if we have, you know,

1:27:06

so that's only x zero, but then if we have x zero, x one, all the way to x n, we call this

1:27:12

multiple logistic regression, because there are multiple features that we're considering

1:27:19

when we're building our model, logistic regression. So I'm going to put that here.

1:27:26

And again, from SK learn this linear model, we can import logistic regression. All right.

1:27:36

And just like how we did above, we can repeat all of this. So here, instead of NB, I'm going to call

1:27:43

this log model, or LG logistic regression. I'm going to change this to logistic regression.

1:27:54

So I'm just going to use the default logistic regression. But actually, if you look here,

1:27:59

you see that you can use different penalties. So right now we're using

1:28:02

an L2 penalty. But L2 is our quadratic formula. Okay, so that means that for,

1:28:09

you know, outliers, it would really penalize that. For all these other things, you know,

1:28:16

you can toggle these different parameters, and you might get slightly different results.

1:28:22

If I were building a production level logistic regression model, then I would want to go and I

1:28:26

would want to figure out how to do that. So I'm going to go ahead and I'm going to go ahead and

1:28:31

I would want to figure out, you know, what are the best parameters to pass into here,

1:28:36

based on my validation data. But for now, we'll just we'll just use this out of the box.

1:28:42

So again, I'm going to fit the X train and the Y train. And I'm just going to predict again,

1:28:49

so I can just call this again. And instead of LG, NB, I'm going to use LG. So here, this is decent

1:28:57

precision 65% recall 71, f 168, or 82 total accuracy of 77. Okay, so it performs slightly

1:29:07

better than I base, but it's still not as good as K and N. Alright, so the last model for

1:29:15

classification that I wanted to talk about is something called support vector machines,

1:29:20

or SVMs for short. So what exactly is an SVM model, I have two different features x zero and

1:29:31

x one on the axes. And then I've told you if it's you know, class zero or class one based on the

1:29:39

blue and red labels, my goal is to find some sort of line between these two labels that best divides

1:29:51

the data. Alright, so this line is our SVM model. So I call it a line here because in 2d, it's a

1:30:00

line, but in 3d, it would be a plane and then you can also have more and more dimensions. So the

1:30:06

proper term is actually I want to find the hyperplane that best differentiates these two

1:30:11

classes. Let's see a few examples. Okay, so first, between these three lines, let's say A, B, and C,

1:30:30

and C, which one is the best divider of the data, which one has you know, all the data on one side

1:30:37

or the other, or at least if it doesn't, which one divides it the most, right, like which one

1:30:42

is has the most defined boundary between the two different groups. So this this question should be

1:30:53

pretty straightforward. It should be a right because a has a clear distinct line between where you

1:31:02

know, everything on this side of a is one label, it's negative and everything on this side of a

1:31:09

is the other label, it's positive. So what if I have a but then what if I had drawn my B

1:31:16

like this, and my C, maybe like this, sorry, they're kind of the labels are kind of close together.

1:31:27

But now which one is the best? So I would argue that it's still a, right? And why is it still a?

1:31:38

Right? And why is it still a? Because in these other two, look at how close this is to that,

1:31:47

to these points. Right? So if I had some new point that I wanted to estimate, okay,

1:31:57

say I didn't have A or B. So let's say we're just working with C. Let's say I have some new point

1:32:02

that's right here. Or maybe a new point that's right there. Well, it seems like just logically

1:32:10

looking at this. I mean, without the boundary, that would probably go under the positives,

1:32:19

right? I mean, it's pretty close to that other positive. So one thing that we care about in SVM

1:32:27

is something known as the margin. Okay, so not only do we want to separate the two classes really

1:32:36

well, we also care about the boundary in between where the points in those classes in our data set

1:32:43

are, and the line that we're drawing. So in a line like this, the closest values to this line

1:32:53

might be like here. And I'm trying to draw these perpendicular. Right? And so this effectively,

1:33:10

if I switch over to these dotted lines, if I can draw this right. So these effectively

1:33:22

are what's known as the margins. Okay, so these both here, these are our margins in our SVMs.

1:33:38

And our goal is to maximize those margins. So not only do we want the line that best separates the

1:33:43

two different classes, we want the line that has the largest margin. And the data points that lie

1:33:51

on the margin lines, the data. So basically, these are the data points that's helping us define our

1:33:57

divider. These are what we call support vectors. Hence the name support vector machines. Okay,

1:34:08

so the issue with SVM sometimes is that they're not so robust to outliers. Right? So for example,

1:34:16

if I had one outlier, like this up here, that would totally change where I want my support

1:34:25

vector to be, even though that might be my only outlier. Okay. So that's just something to keep

1:34:31

in mind. As you know, when you're working with SVM is, it might not be the best model if there

1:34:38

are outliers in your data set. Okay, so another example of SVMs might be, let's say that we have

1:34:45

data like this, I'm just going to use a one dimensional data set for this example. Let's

1:34:50

say we have a data set that looks like this. Well, our, you know, separators should be

1:34:56

perpendicular to this line. But it should be somewhere along this line. So it could be

1:35:02

anywhere like this. You might argue, okay, well, there's one here. And then you could also just

1:35:09

draw another one over here, right? And then maybe you can have two SVMs. But that's not really how

1:35:13

SVMs work. But one thing that we can do is we can create some sort of projection. So I realize here

1:35:21

that one thing I forgot to do was to label where zero was. So let's just say zero is here.

1:35:32

Now, what I'm going to do is I'm going to say, okay, I'm going to have x, and then I'm going to

1:35:36

have x, sorry, x zero and x one. So x zero is just going to be my original x. But I'm going to make

1:35:44

x one equal to let's say, x squared. So whatever is this squared, right? So now, my natives would be,

1:35:56

you know, maybe somewhere here, here, just pretend that it's somewhere up here.

1:36:02

Right. And now my pluses might be something like

1:36:10

that. And I'm going to run out of space over here. So I'm just going to draw these together,

1:36:16

use your imagination. But once I draw it like this, well, it's a lot easier to apply a boundary,

1:36:27

right? Now our SVM could be maybe something like this, this. And now you see that we've divided

1:36:35

our data set. Now it's separable where one class is this way. And the other class is that way.

1:36:42

Okay, so that's known as SVMs. I do highly suggest that, you know, any of these models that we just

1:36:49

mentioned, if you're interested in them, do go more in depth mathematically into them. Like how

1:36:54

do we how do we find this hyperplane? Right? I'm not going to go over that in this specific course,

1:37:00

because you're just learning what an SVM is. But it's a good idea to know, oh, okay, this is the

1:37:05

technique behind finding, you know, what exactly are the are the how do you define the hyperplane

1:37:13

that we're going to use. So anyways, this transformation that we did down here, this is known

1:37:19

as the kernel trick. So when we go from x to some coordinate x, and then x squared,

1:37:27

what we're doing is we are applying a kernel. So that's why it's called the kernel trick.

1:37:33

So SVMs are actually really powerful. And you'll see that here. So from sk learn.svm, we are going

1:37:40

to import SVC. And SVC is our support vector classifier. So with this, so with our SVM model,

1:37:49

we are going to, you know, create SVC model. And we are going to, again, fit this to X train, I

1:37:59

could have just copied and pasted this, I should be able to do that. So we're going to create SVC

1:38:06

again, fit this to X train, I could have just copied and pasted this, I should have probably

1:38:10

done that. Okay, taking a bit longer. All right. Let's predict using RSVM model. And here,

1:38:23

let's see if I can hover over this. Right. So again, you see a lot of these different

1:38:28

parameters here that you can go back and change if you were creating a production level model. Okay,

1:38:37

but in this specific case, we'll just use it out of the box again. So if I make predictions,

1:38:46

you'll note that Wow, the accuracy actually jumps to 87% with the SVM. And even with class zero,

1:38:53

there's nothing less than, you know, point eight, which is great. And for class one,

1:38:59

I mean, everything's at 0.9, which is higher than anything that we had seen to this point.

1:39:06

So so far, we've gone over four different classification models, we've done SVM,

1:39:11

logistic regression, naive Bayes and cannon. And these are just simple ways on how to implement

1:39:17

them. Each of these they have different, you know, they have different hyper parameters that you can

1:39:23

go and you can toggle. And you can try to see if that helps later on or not. But for the most part,

1:39:31

they perform, they give us around 70 to 80% accuracy. Okay, with SVM being the best. Now,

1:39:40

let's see if we can actually beat that using a neural net. Now the final type of model that

1:39:45

I wanted to talk about is known as a neural net or neural network. And neural nets look something

1:39:51

like this. So you have an input layer, this is where all your features would go. And they have

1:39:58

all these arrows pointing to some sort of hidden layer. And then all these arrows point to some

1:40:03

sort of output layer. So what is what is all this mean? Each of these layers in here, this is

1:40:10

something known as a neuron. Okay, so that's a neuron. In a neural net. These are all of our

1:40:18

features that we're inputting into the neural net. So that might be x zero x one all the way through

1:40:23

x n. Right. And these are the features that we talked about there, they might be you know,

1:40:28

the pregnancy, the BMI, the age, etc. Now all of these get weighted by some value. So they

1:40:38

are multiplied by some w number that applies to that one specific category that one specific

1:40:44

feature. So these two get multiplied. And the sum of all of these goes into that neuron. Okay,

1:40:51

so basically, I'm taking w zero times x zero. And then I'm adding x one times w one and then

1:40:58

I'm adding you know, x two times w two, etc, all the way to x n times w n. And that's getting

1:41:05

input into the neuron. Now I'm also adding this bias term, which just means okay, I might want

1:41:11

to shift this by a little bit. So I might add five or I might add 0.1 or I might subtract 100,

1:41:17

I don't know. But we're going to add this bias term. And the output of all these things. So

1:41:24

the sum of this, this, this and this, go into something known as an activation function,

1:41:31

okay. And then after applying this activation function, we get an output. And this is what a

1:41:38

neuron would look like. Now a whole network of them would look something like this.

1:41:46

So I kind of gloss over this activation function. What exactly is that? This is how a neural net

1:41:53

looks like if we have all our inputs here. And let's say all of these arrows represent some sort

1:41:58

of addition, right? Then what's going on is we're just adding a bunch of times, right? We're adding

1:42:08

the some sort of weight times these input layer a bunch of times. And then if we were to go back

1:42:13

and factor that all out, then this entire neural net is just a linear combination of these input

1:42:22

layers, which I don't know about you, but that just seems kind of useless, right? Because we could

1:42:27

literally just write that out in a formula, why would we need to set up this entire neural network,

1:42:33

we wouldn't. So the activation function is introduced, right? So without an activation

1:42:40

function, this just becomes a linear model. An activation function might look something like

1:42:46

this. And as you can tell, these are not linear. And the reason why we introduce these is so that

1:42:52

our entire model doesn't collapse on itself and become a linear model. So over here, this is

1:42:58

something known as a sigmoid function, it runs between zero and one, tanh runs between negative

1:43:04

one all the way to one. And this is ReLU, which anything less than zero is zero, and then anything

1:43:10

greater than zero is linear. So with these activation functions, every single output of a neuron

1:43:18

is no longer just the linear combination of these, it's some sort of altered linear state, which means

1:43:24

that the input into the next neuron is, you know, it doesn't it doesn't collapse on itself, it doesn't

1:43:32

become linear, because we've introduced all these nonlinearities. So this is a training set, the

1:43:39

model, the loss, right? And then we do this thing called training, where we have to feed the loss

1:43:45

back into the model, and make certain adjustments to the model to improve this predicted output.

1:43:55

Let's talk a little bit about the training, what exactly goes on during that step.

1:44:00

Let's go back and take a look at our L2 loss function. This is what our L2 loss function

1:44:07

looks like it's a quadratic formula, right? Well, up here, the error is really, really, really, really

1:44:15

large. And our goal is to get somewhere down here, where the loss is decreased, right? Because that

1:44:23

means that our predicted value is closer to our true value. So that means that we want to go

1:44:30

this way. Okay. And thanks to a lot of properties of math, something that we can do is called

1:44:39

gradient descent, in order to follow this slope down this way. This quadratic is, it has different

1:44:53

different slopes with respect to some value. Okay, so the loss with respect to some weight

1:45:03

w zero, versus w one versus w n, they might all be different. Right? So some way that I kind of

1:45:12

think about it is, to what extent is this value contributing to our loss. And we can actually

1:45:18

figure that out through some calculus, which we're not going to touch up on in this specific course.

1:45:24

But if you want to learn more about neural nets, you should probably also learn some calculus

1:45:29

and figure out what exactly back propagation is doing, in order to actually calculate, you know,

1:45:35

how much do we have to backstep by. So the thing is here, you might notice that this follows

1:45:41

this curve at all of these different points. And the closer we get to the bottom, the smaller

1:45:48

this step becomes. Now stick with me here. So my new value, this is what we call a weight update,

1:45:57

I'm going to take w zero, and I'm going to set some new value for w zero. And what I'm going to

1:46:04

set for that is the old value of w zero, plus some factor, which I'll just call alpha for now,

1:46:13

times whatever this arrow is. So that's basically saying, okay, take our old w zero, our old weight,

1:46:23

and just decrease it this way. So I guess increase it in this direction, right, like take a step in

1:46:30

this direction. But this alpha here is telling us, okay, don't don't take a huge step, right,

1:46:34

just in case we're wrong, take a small step, take a small step in that direction, see if we get any

1:46:38

closer. And for those of you who, you know, do want to look more into the mathematics of things,

1:46:45

the reason why I use a plus here is because this here is the negative gradient, right, if this were

1:46:51

just the if you were to use the actual gradient, this should be a minus.

1:46:54

Now this alpha is something that we call the learning rate. Okay, and that adjusts how quickly

1:47:00

we're taking steps. And that might, you know, tell our that that will ultimately control

1:47:07

how long it takes for our neural net to converge. Or sometimes if you set it too high, it might even

1:47:13

diverge. But with all of these weights, so here I have w zero, w one, and then w n. We make the same

1:47:21

update to all of them after we calculate the loss, the gradient of the loss with respect to that

1:47:29

weight. So that's how back propagation works. And that is everything that's going on here. After we

1:47:37

calculate the loss, we're calculating gradients, making adjustments in the model. So we're setting

1:47:42

all the all the weights to something adjusted slightly. And then we're going to calculate the

1:47:50

gradient. And then we're saying, Okay, let's take the training set and run it through the model

1:47:55

again, and go through this loop all over again. So for machine learning, we already have seen some

1:48:01

libraries that we use, right, we've already seen SK learn. But when we start going into neural

1:48:09

networks, this is kind of what we're trying to program. And it's not very fun to try to

1:48:19

do this from scratch, because not only will we probably have a lot of bugs, but also probably

1:48:25

not going to be fast enough, right? Wouldn't it be great if there are just some, you know,

1:48:30

full time professionals that are dedicated to solving this problem, and they could literally

1:48:35

just give us their code that's already running really fast? Well, the answer is, yes, that exists.

1:48:43

And that's why we use TensorFlow. So TensorFlow makes it really easy to define these models. But

1:48:49

we also have enough control over what exactly we're feeding into this model. So for example,

1:48:55

this line here is basically saying, Okay, let's create a sequential neural net. So sequential is

1:49:02

just, you know, what we've seen here, it just goes one layer to the next. And a dense layer means that

1:49:08

a dense layer means that all of them are interconnected. So here, this is interconnected with all of these

1:49:13

nodes, and this one's all these, and then this one gets connected to all of the next ones, and so on.

1:49:19

So we're going to create 16 dense nodes with relu activation functions. And then we're going

1:49:26

to create another layer of 16 dense nodes with relu activation. And then our output layer is going

1:49:34

to be just one node. Okay. And that's how easy it is to define something in TensorFlow. So TensorFlow

1:49:43

is an open source library that helps you develop and train your ML models. Let's implement this

1:49:51

for a neural net. So we're using a neural net for classification. Now, so our neural net model,

1:49:58

we are going to use TensorFlow, and I don't think I imported that up here. So we are going to import

1:50:03

that down here. So I'm going to import TensorFlow as TF. And enter. Cool. So my neural net model

1:50:19

is going to be, I'm going to use this. So essentially, this is saying layer all these

1:50:28

things that I'm about to pass in. So yeah, layer them linear stack of layers, layer them as a model.

1:50:35

And what that means, nope, not that. So what that means is I can pass in

1:50:42

some sort of layer, and I'm just going to use a dense layer.

1:50:46

Oops, dot dense. And let's say we have 32 units. Okay, I will also

1:51:01

set the activation as really. And at first we have to specify the input shape. So here we have 10,

1:51:09

and comma. Alright. Alright, so that's our first layer. Now our next layer, I'm just going to have

1:51:19

another dense layer of 32 units all using relu. And that's it. So for the final layer, this is

1:51:28

just going to be my output layer, it's going to just be one node. And the activation is going to

1:51:35

be sigmoid. So if you recall from our logistic regression, what happened there was when we had

1:51:43

a sigmoid, it looks something like this, right? So by creating a sigmoid activation on our last layer,

1:51:49

we're essentially projecting our predictions to be zero or one, just like in logistic regression.

1:51:57

And that's going to help us, you know, we can just round to zero or one and classify that way.

1:52:03

Okay. So this is my neural net model. And I'm going to compile this. So in TensorFlow,

1:52:12

we have to compile it. It's really cool, because I can just literally pass in what type of optimizer

1:52:17

I want, and it'll do it. So here, if I go to optimizers, I'm actually going to use atom.

1:52:24

And you'll see that, you know, the learning rate is 0.001. So I'm just going to use that default.

1:52:31

So 0.001. And my loss is going to be binary cross entropy. And the metrics that I'm also going to

1:52:44

include on here, so it already will consider loss, but I'm, I'm also going to tack on accuracy.

1:52:50

So we can actually see that in a plot later on. Alright, so I'm going to run this.

1:52:55

And one thing that I'm going to also do is I'm going to define these plot definitions. So I'm

1:53:01

actually copying and pasting this, I got these from TensorFlow. So if you go on to some TensorFlow

1:53:06

tutorial, they actually have these, this like, defined. And that's exactly what I'm doing here.

1:53:13

So I'm actually going to move this cell up, run that. So we're basically plotting the loss

1:53:18

over all the different epochs. epochs means like training cycles. And we're going to run that. So

1:53:23

means like training cycles. And we're going to plot the accuracy over all the epochs.

1:53:28

Alright, so we have our model. And now all that's left is, let's train it. Okay.

1:53:37

So I'm going to say history. So TensorFlow is great, because it keeps track of the history

1:53:42

of the training, which is why we can go and plot it later on. Now I'm going to set that equal to

1:53:47

this neural net model. And fit that with x train, y train, I'm going to make the number of epochs

1:53:59

equal to let's say just let's just use 100 for now. And the batch size, I'm going to set equal to,

1:54:06

let's say 32. Alright. And the validation split. So what the validation split does, if it's down

1:54:18

here somewhere. Okay, so yeah, this validation split is just the fraction of the training data

1:54:23

to be used as validation data. So essentially, every single epoch, what's going on is TensorFlow

1:54:31

saying, leave certain if this is point two, then leave 20% out. And we're going to test how the

1:54:37

model performs on that 20% that we've left out. Okay, so it's basically like our validation data

1:54:42

set. But TensorFlow does it on our training data set during the training. So we have now a measure

1:54:48

outside of just our validation data set to see, you know, what's going on. So validation split,

1:54:54

I'm going to make that 0.2. And we can run this. So if I run that, all right, and I'm actually going

1:55:05

to set verbose equal to zero, which means, okay, don't print anything, because printing something

1:55:13

for 100 epochs might get kind of annoying. So I'm just going to let it run, let it train,

1:55:19

and then we'll see what happens. Cool, so it finished training. And now what I can do is

1:55:31

because you know, I've already defined these two functions, I can go ahead and I can plot the loss,

1:55:36

oops, loss of that history. And I can also plot the accuracy throughout the training.

1:55:45

So this is a little bit ish what we're looking for. We definitely are looking for a steadily

1:55:52

decreasing loss and an increasing accuracy. So here we do see that, you know, our validation

1:55:59

accuracy improves from around point seven, seven or something all the way up to somewhere around

1:56:07

point, maybe eight one. And our loss is decreasing. So this is good. It is expected that the validation

1:56:16

loss and accuracy is performing worse than the training loss or accuracy. And that's because

1:56:23

our model is training on that data. So it's adapting to that data. Whereas the validation stuff is,

1:56:28

you know, stuff that it hasn't seen yet. So, so that's why. So in machine learning, as we saw above,

1:56:35

we could change a bunch of the parameters, right? Like I could change this to 64. So now it'd be

1:56:40

a row of 64 nodes, and then 32, and then one. So I can change some of these parameters.

1:56:47

And a lot of machine learning is trying to find, hey, what do we set these hyper parameters to?

1:56:54

So what I'm actually going to do is I'm going to rewrite this so that we can do something what's

1:57:02

known as a grid search. So we can search through an entire space of hey, what happens if, you know,

1:57:08

we have 64 nodes and 64 nodes, or 16 nodes and 16 nodes, and so on. And then on top of all that,

1:57:19

we can, you know, we can change this learning rate, we can change how many epochs we can change,

1:57:26

you know, the batch size, all these things might affect our training. And just for kicks,

1:57:33

I'm also going to add what's known as a dropout layer in here. And what dropout is doing is

1:57:42

saying, hey, randomly choose with at this rate, certain nodes, and don't train them in, you know,

1:57:51

in a certain iteration. So this helps prevent overfitting. Okay, so I'm actually going to

1:57:59

define this as a function called train model, we're going to pass in x train, y train,

1:58:07

the number of nodes, the dropout, you know, the probability that we just talked about

1:58:15

learning rate. So I'm actually going to say lr batch size. And we can also pass in number epochs,

1:58:27

right? I mentioned that as a parameter. So indent this, so it goes under here. And with these two,

1:58:34

I'm going to set this equal to number of nodes. And now with the two dropout layers, I'm going

1:58:40

to set dropout prob. So now you know, the probability of turning off a node during the training

1:58:48

is equal to dropout prob. And I'm going to keep the output layer the same. Now I'm compiling it,

1:58:55

but this here is now going to be my learning rate. And I still want binary cross entropy and

1:59:00

accuracy. We are actually going to train our model inside of this function. But here we can do the

1:59:12

epochs equal epochs, and this is equal to whatever, you know, we're passing in x train,

1:59:19

y train belong right here. Okay, so those are getting passed in as well. And finally, at the

1:59:25

end, I'm going to return this model and the history of that model. Okay. So now what I'll do

1:59:40

is let's just go through all of these. So let's say let's keep epochs at 100. And now what I can

1:59:46

do is I can say, hey, for a number of nodes in, let's say, let's do 1632 and 64, to see what

1:59:53

happens for the different dropout probabilities. And I mean, zero would be nothing. Let's use 0.2.

2:00:02

Also, to see what happens. You know, for the learning rate in 0.005, 0.001. And you know,

2:00:17

maybe we want to throw on 0.1 in there as well. And then for the batch size, let's do 1632,

2:00:27

64 as well. Actually, and let's also throw in 128. Actually, let's get rid of 16. Sorry,

2:00:33

so 128 in there. That should be 01. I'm going to record the model and history using this

2:00:44

train model here. So we're going to do x train y train, the number of nodes is going to be,

2:00:54

you know, the number of nodes that we've defined here, dropout, prob, LR, batch size, and epochs.

2:01:04

Okay. And then now we have both the model and the history. And what I'm going to do is again,

2:01:10

I want to plot the loss for the history. I'm also going to plot the accuracy.

2:01:19

Probably should have done them side by side, that probably would have been easier.

2:01:26

Okay, so what I'm going to do is split up, split this up. And that will be

2:01:34

the subplots. So now this is just saying, okay, I want one row and two columns in that row for my

2:01:41

plots. Okay, so I'm going to plot on my axis one, the loss. I don't actually know this is going to

2:01:56

work. Okay, we don't care about the grid. Yeah, let's let's keep the grid. And then now my other.

2:02:09

So now on here, I'm going to plot all the accuracies on the second plot.

2:02:20

I might have to debug this a bit.

2:02:21

We should be able to get rid of that. If we run this, we already have history saved as a variable

2:02:27

in here. So if I just run it on this, okay, it has no attribute x label. Oh, I think it's because

2:02:36

it's like set x label or something. Okay, yeah, so it's, it's set instead of just x label, y label.

2:02:47

So let's see if that works. All right, cool. Um, and let's actually make this a bit larger.

2:02:55

Okay, so we can actually change the figure size that I'm gonna set. Let's see what happens if I

2:02:59

set that to. Oh, that's not the way I wanted it. Okay, so that looks reasonable.

2:03:08

And that's just going to be my plot history function. So now I can plot them side by side.

2:03:15

Here, I'm going to plot the history. And what I'm actually going to do is I so here, first,

2:03:23

I'm going to print out all these parameters. So I'm going to print out

2:03:27

the F string to print out all of this stuff. So here, I'm going to print out all these parameters.

2:03:34

Uh, all of this stuff. So here, I'm printing out how many nodes, um, the dropout probability,

2:03:45

uh, the learning rate.

2:03:55

And we already know how many you found, so I'm not even going to bother with that.

2:03:57

So once we plot this, uh, let's actually also figure out what the, um, what the validation

2:04:10

losses on our validation set that we have that we created all the way back up here.

2:04:16

Alright, so remember, we created three data sets. Let's call our model and evaluate what the

2:04:23

validation data with the validation data sets loss would be. And I actually want to record,

2:04:33

let's say I want to record whatever model has the least validation loss. So

2:04:40

first, I'm going to initialize that to infinity so that you know, any model will beat that score.

2:04:45

So if I do float infinity, that will set that to infinity. And maybe I'll keep

2:04:53

track of the parameters. Actually, it doesn't really matter. I'm just going to keep track of

2:04:58

the model. And I'm gonna set that to none. So now down here, if the validation loss is ever

2:05:06

less than the least validation loss, then I am going to simply come down here and say,

2:05:13

Hey, this validation for this least validation loss is now equal to the validation loss.

2:05:21

And the least loss model is whatever this model is that just earned that validation loss. Okay.

2:05:31

So we are actually just going to let this run for a while. And then we're going to get our least

2:05:40

last model after that. So let's just run. All right, and now we wait.

2:05:51

All right, so we've finally finished training. And you'll notice that okay, down here, the loss

2:06:12

actually gets to like 0.29. The accuracy is around 88%, which is pretty good. So you might be wondering,

2:06:19

okay, why is this accuracy in this? Like, these are both the validation. So this accuracy here

2:06:26

is on the validation data set that we've defined at the beginning, right? And this one here,

2:06:30

this is actually taking 20% of our tests, our training set every time during the training,

2:06:35

and saying, Okay, how much of it do I get right now? You know, after this one step where I didn't

2:06:41

train with any of that. So they're slightly different. And actually, I realized later on

2:06:46

that I probably you know, probably what I should have done is over here, when we were defining

2:06:54

the model fit, instead of the validation split, you can define the validation data.

2:07:00

And you can pass in the validation data, I don't know if this is the proper syntax. But

2:07:05

that's probably what I should have done. But instead, you know, we'll just stick with what

2:07:09

we have here. So you'll see at the end, you know, with the 64 nodes, it seems like this is our best

2:07:16

performance 64 nodes with a dropout of 0.2, a learning rate of 0.001, and a batch size of 64.

2:07:25

And it does seem like yes, the validation, you know, the fake validation, but the validation

2:07:34

loss is decreasing, and then the accuracy is increasing, which is a good sign. Okay,

2:07:40

so finally, what I'm going to do is I'm actually just going to predict. So I'm going to take

2:07:45

this model, which we've called our least loss model, I'm going to take this model,

2:07:50

and I'm going to predict x test on that. And you'll see that it gives me some values that

2:07:58

are really close to zero and some that are really close to one. And that's because we have a sigmoid

2:08:02

output. So if I do this, and what I can do is I can cast them. So I'm going to say anything that's

2:08:11

greater than 0.5, set that to one. So if I actually, I think what happens if I do this?

2:08:22

Oh, okay, so I have to cast that as type. And so now you'll see that it's ones and zeros. And I'm

2:08:29

actually going to transform this into a column as well. So here I'm going to Oh, oops, I didn't

2:08:40

I didn't mean to do that. Okay, no, I wanted to just reshape it to that. So now it's one dimensional.

2:08:49

Okay. And using that we can actually just rerun the classification report based on these this

2:08:57

neural net output. And you'll see that okay, the the F ones are the accuracy gives us 87%. So it

2:09:04

seems like what happened here is the precision on class zero. So the hadrons has increased a bit,

2:09:12

but the recall decreased. But the F one score is still at a good point eight one. And for the other

2:09:19

class, it looked like the precision decreased a bit the recall increased for an overall F one score.

2:09:25

That's also been increased. I think I interpreted that properly. I mean, we went through all this

2:09:31

work and we got a model that performs actually very, very similarly to the SVM model that we

2:09:37

had earlier. And the whole point of this exercise was to demonstrate, okay, these are how you can

2:09:43

define your models. But it's also to say, hey, maybe, you know, neural nets are very, very

2:09:48

powerful, as you can tell. But sometimes, you know, an SVM or some other model might actually be more

2:09:55

appropriate. But in this case, I guess it didn't really matter which one we use at the end. An 87%

2:10:04

accuracy score is still pretty good. So yeah, let's now move on to regression.

2:10:11

We just saw a bunch of different classification models. Now let's shift gears into regression,

2:10:17

the other type of supervised learning. If we look at this plot over here, we see a bunch of scattered

2:10:23

data points. And here we have our x value for those data points. And then we have the corresponding y

2:10:31

value, which is now our label. And when we look at this plot, well, our goal in regression is to find

2:10:40

the line of best fit that best models this data. Essentially, we're trying to let's say we're given

2:10:48

some new value of x that we don't have in our sample, we're trying to say, okay, what would my

2:10:54

prediction for y be for that given x value. So that, you know, might be somewhere around there.

2:11:03

I don't know. But remember, in regression that, you know, given certain features,

2:11:08

we're trying to predict some continuous numerical value for y.

2:11:12

In linear regression, we want to take our data and fit a linear model to this data. So in this case,

2:11:21

our linear model might look something along the lines of here. Right. So this here would be

2:11:30

considered as maybe our line of best fit. And this line is modeled by the equation, I'm going to write

2:11:41

it down here, y equals b zero, plus b one x. Now b zero just means it's this y intercept. So if we

2:11:51

extend this y down here, this value here is b zero, and then b one defines the source of the

2:11:58

line, defines the slope of this line. Okay. All right. So that's the that's the formula

2:12:09

for linear regression. And how exactly do we come up with that formula? What are we trying to do

2:12:17

with this linear regression? You know, we could just eyeball where the line be, but humans are

2:12:23

not very good at eyeballing certain things like that. I mean, we can get close, but a computer is

2:12:29

better at giving us a precise value for b zero and b one. Well, let's introduce the concept of

2:12:37

something known as a residual. Okay, so residual, you might also hear this being called the error.

2:12:47

And what that means is, let's take some data point in our data set. And we're going to evaluate how

2:12:55

far off is our prediction from a data point that we already have. So this here is our y, let's say,

2:13:04

this is 12345678. So this is y eight, let's call it, you'll see that I use this y i in order to

2:13:15

I in order to represent, hey, just one of these points. Okay. So this here is why and this here

2:13:23

would be the prediction. Oops, this here would be the prediction for y eight, which I've labeled

2:13:30

with this hat. Okay, if it has a hat on it, that means hey, this is what this is my guess this is

2:13:35

my prediction for you know, this specific value of x. Okay. Now the residual would be this distance

2:13:48

here between y eight and y hat eight. So y eight minus y hat eight. All right, because that would

2:13:58

give us this here. And I'm just going to take the absolute value of this. Because what if it's below

2:14:04

the line, right, then you would get a negative value, but distance can't be negative. So we're

2:14:08

just going to put a little hat, or we're going to put a little absolute value around this quantity.

2:14:15

And that gives us the residual or the error. So let me rewrite that. And you know, to generalize

2:14:23

to all the points, I'm going to say the residual can be calculated as y i minus y hat of i. Okay.

2:14:32

So this just means the distance between some given point, and its prediction, its corresponding

2:14:39

prediction on the line. So now, with this residual, this line of best fit is generally trying to

2:14:47

decrease these residuals as much as possible. So now that we have some value for the error,

2:14:55

our line of best fit is trying to decrease the error as much as possible for all of the different

2:15:00

data points. And that might mean, you know, minimizing the sum of all the residuals. So this

2:15:07

here, this is the sum symbol. And if I just stick the residual calculation in there,

2:15:16

it looks something like that, right. And I'm just going to say, okay, for all of the eyes in our

2:15:21

data set, so for all the different points, we're going to sum up all the residuals. And I'm going

2:15:27

to try to decrease that with my line of best fit. So I'm going to find the B0 and B1, which gives

2:15:33

me the lowest value of this. Okay. Now in other, you know, sometimes in different circumstances,

2:15:41

we might attach a squared to that. So we're trying to decrease the sum of the squared residuals.

2:15:49

And what that does is it just, you know, it adds a higher penalty for how far off we are from,

2:16:03

you know, points that are further off. So that is linear regression, we're trying to find

2:16:08

this equation, some line of best fit that will help us decrease this measure of error

2:16:15

with respect to all the data points that we have in our data set, and try to come up with

2:16:19

the best prediction for all of them. This is known as simple linear regression.

2:16:30

And basically, that means, you know, our equation looks something like this. Now, there's also

2:16:39

multiple linear regression, which just means that hey, if we have more than one value for x, so like

2:16:52

think of our feature vectors, we have multiple values in our x vector, then our predictor might

2:16:58

look something more like this. Actually, I'm just going to say etc, plus b n, x n. So now I'm coming

2:17:11

up with some coefficient for all of the different x values that I have in my vector. Now you guys

2:17:18

might have noticed that I have some assumptions over here. And you might be asking, okay, Kylie,

2:17:23

what in the world do these assumptions mean? So let's go over them.

2:17:26

So let's go over them. The first one is linearity.

2:17:33

And what that means is, let's say I have a data set. Okay.

2:17:43

Linearity just means, okay, my does my data follow a linear pattern? Does y increase as x

2:17:50

increases? Or does y decrease at as x increases? Does so if y increases or decreases at a constant

2:17:59

rate as x increases, then you're probably looking at something linear. So what's the example of a

2:18:04

nonlinear data set? Let's say I had data that might look something like that. Okay. So now just

2:18:12

visually judging this, you might say, okay, seems like the line of best fit might actually be some

2:18:18

curve like this. Right. And in this case, we don't satisfy that linearity assumption anymore.

2:18:29

So with linearity, we basically just want our data set to follow some sort of linear trajectory.

2:18:39

And independence, our second assumption

2:18:42

just means this point over here, it should have no influence on this point over here,

2:18:50

or this point over here, or this point over here. So in other words, all the points,

2:18:56

all the samples in our data set should be independent. Okay, they should not rely on

2:19:03

one another, they should not affect one another.

2:19:05

Okay, now, normality and homoscedasticity, those are concepts which use this residual. Okay. So if

2:19:17

I have a plot that looks something like this, and I have a plot that looks like this. Okay,

2:19:31

something like this. And my line of best fit is somewhere here, maybe it's something like that.

2:19:47

In order to look at these normality and homoscedasticity assumptions, let's look at

2:19:52

the residual plot. Okay. And what that means is I'm going to keep my same x axis. But instead

2:20:03

of plotting now where they are relative to this y, I'm going to plot these errors. So now I'm

2:20:09

going to plot y minus y hat like this. Okay. And now you know, this one is slightly positive,

2:20:19

so it might be here, this one down here is negative, it might be here. So our residual plot,

2:20:25

it's literally just a plot of how you know, the values are distributed around our line of best

2:20:30

fit. So it looks like it might, you know, look something like this. Okay. So this might be our

2:20:42

residual plot. And what normality means, so our assumptions are normality and homoscedasticity,

2:20:59

I might have butchered that spelling, I don't really know. But what normality is saying is

2:21:05

saying, okay, these residuals should be normally distributed. Okay, around this line of best fit,

2:21:12

it should follow a normal distribution. And now what homoscedasticity says, okay, our variants

2:21:21

of these points should remain constant throughout. So this spread here should be approximately the

2:21:28

same as this spread over here. Now, what's an example of where you know, homoscedasticity is

2:21:35

not held? Well, let's say that our original plot actually looks something like this.

2:21:46

Okay, so now if we looked at the residuals for that, it might look something

2:21:51

like that. And now if we look at this spread of the points, it decreases, right? So now the spread

2:22:03

is not constant, which means that homoscedasticity, this assumption would not be fulfilled, and it

2:22:12

might not be appropriate to use linear regression. So that's just linear regression. Basically,

2:22:18

we have a bunch of data points, we want to predict some y value for those. And we're trying to come

2:22:25

up with this line of best fit that best describes, hey, given some value x, what would be my best

2:22:32

guess of what y is. So let's move on to how do we evaluate a linear regression model. So the first

2:22:43

measure that I'm going to talk about is known as mean absolute error, or MAE

2:22:52

for short, okay. And mean absolute error is basically saying, all right, let's take

2:22:59

all the errors. So all these residuals that we talked about, let's sum up the distance

2:23:06

for all of them, and then take the average. And then that can describe, you know, how far off are

2:23:11

we. So the mathematical formula for that would be, okay, let's take all the residuals.

2:23:21

Alright, so this is the distance. Actually, let me redraw a plot down here. So

2:23:27

suppose I have a data set, look like this. And here are all my data points, right. And now let's

2:23:41

say my line looks something like that. So my mean absolute error would be summing up all of these

2:23:52

values. This was a mistake. So summing up all of these, and then dividing by how many data points

2:24:01

I have. So what would be all the residuals, it would be y i, right, so every single point,

2:24:08

minus y hat i, so the prediction for that on here. And then we're going to sum over all of

2:24:16

all of the different i's in our data set. Right, so i, and then we divide by the number of points

2:24:24

we have. So actually, I'm going to rewrite this to make it a little clearer. So i is equal to

2:24:29

whatever the first data point is all the way through the nth data point. And then we divide

2:24:33

it by n, which is how many points there are. Okay, so this is our measure of mae. And this is basically

2:24:42

telling us, okay, in on average, this is the distance between our predicted value and the

2:24:50

actual value in our training set. Okay. And mae is good because it allows us to, you know, when we

2:25:01

get this value here, we can literally directly compare it to whatever units the y value is in.

2:25:08

So let's say y is we're talking, you know, the prediction of the price of a house, right, in

2:25:17

dollars. Once we have once we calculate the mae, we can literally say, oh, the average, you know,

2:25:24

price, the average, how much we're off by is literally this many dollars. Okay. So that's the

2:25:34

mean absolute error. An evaluation technique that's also closely related to that is called the mean

2:25:40

squared error. And this is MSE for short. Okay. Now, if I take this plot again, and I duplicated

2:25:53

and move it down here, well, the gist of mean squared error is kind of the same, but instead

2:25:59

of the absolute value, we're going to square. So now the MSE is something along the lines of,

2:26:06

okay, let's sum up something, right, so we're going to sum up all of our errors.

2:26:13

So now I'm going to do y i minus y hat i. But instead of absolute valuing them,

2:26:19

I'm going to square them all. And then I'm going to divide by n in order to find the mean. So

2:26:25

basically, now I'm taking all of these different values, and I'm squaring them first before I add

2:26:33

them to one another. And then I divide by n. And the reason why we like using mean squared error

2:26:42

is that it helps us punish large errors in the prediction. And later on, MSE might be important

2:26:49

because of differentiability, right? So a quadratic equation is differentiable, you know,

2:26:55

if you're familiar with calculus, a quadratic equation is differentiable, whereas the absolute

2:27:00

value function is not totally differentiable everywhere. But if you don't understand that,

2:27:05

don't worry about it, you won't really need it right now. And now one downside of mean squared

2:27:10

error is that once I calculate the mean squared error over here, and I go back over to y, and I

2:27:16

want to compare the values. Well, it gets a little bit trickier to do that because now my mean squared

2:27:25

error is in terms of y squared, right? It's this is now squared. So instead of just dollars, how,

2:27:33

you know, how many dollars off am I I'm talking how many dollars squared off am I. And that,

2:27:40

you know, to humans, it doesn't really make that much sense. Which is why we have created

2:27:45

something known as the root mean squared error. And I'm just going to copy this diagram over here

2:27:53

because it's very, very similar to mean squared error. Except now we take a big squared root.

2:28:03

Okay, so this is our messy, and we take the square root of that mean squared error. And so now the

2:28:10

term in which you know, we're defining our error is now in terms of that dollar sign symbol again.

2:28:17

So that's a pro of root mean squared error is that now we can say, okay, our error according

2:28:23

to this metric is this many dollar signs off from our predictor. Okay, so it's in the same unit,

2:28:30

which is one of the pros of root mean squared error. And now finally, there is the coefficient

2:28:37

of determination, or r squared. And this is a formula for r squared. So r squared is equal

2:28:43

to one minus RSS over TSS. Okay, so what does that mean? Basically, RSS stands for the sum

2:28:56

of the squared residuals. So maybe it should be SSR instead, but

2:29:03

RSS sum of the squared residuals, and this is equal to if I take the sum of all the values,

2:29:14

and I take y i minus y hat, i, and square that, that is my RSS, right, it's a sum of the squared

2:29:24

residuals. Now TSS, let me actually use a different color for that.

2:29:30

So TSS is the total sum of squares.

2:29:41

And what that means is that instead of being with respect to this prediction,

2:29:48

we are instead going to

2:29:52

take each y value and just subtract the mean of all the y values, and square that.

2:30:00

Okay, so if I drew this out,

2:30:13

and if this were my

2:30:16

actually, let's use a different color. Let's use green. If this were my predictor,

2:30:24

so RSS is giving me this measure here, right? It's giving me some estimate of how far off we are from

2:30:33

our regressor that we predicted. Actually, I'm gonna take this one, and I'm gonna take this one,

2:30:41

and actually, I'm going to use red for that. Well, TSS, on the other hand, is saying, okay,

2:30:52

how far off are these values from the mean. So if we literally didn't do any calculations for the

2:30:59

line of best fit, if we just took all the y values and average all of them, and said, hey,

2:31:04

this is the average value for every single x value, I'm just going to predict that average value

2:31:10

instead, then it's asking, okay, how far off are all these points from that line?

2:31:19

Okay, and remember that this square means that we're punishing larger errors, right? So even if

2:31:26

they look somewhat close in terms of distance, the further a few data points are, then the further

2:31:32

the larger our total sum of squares is going to be. Sorry, that was my dog. So the total sum of

2:31:39

squares is taking all of these values and saying, okay, what is the sum of squares, if I didn't do

2:31:44

any regressor, and I literally just calculated the average of all the y values in my data set,

2:31:51

and for every single x value, I'm just going to predict that average, which means that okay,

2:31:55

like, that means that maybe y and x aren't associated with each other at all. Like the

2:32:00

best thing that I can do for any new x value, just predict, hey, this is the average of my data set.

2:32:05

And this total sum of squares is saying, okay, well, with respect to that average,

2:32:12

what is our error? Right? So up here, the sum of the squared residuals, this is telling us what is

2:32:19

our what what is our error with respect to this line of best fit? Well, our total sum of squares

2:32:26

saying what is the error with respect to, you know, just the average y value. And if our line

2:32:34

of best fit is a better fit, then this total sum of squares, that means that you know, this numerator,

2:32:46

that means that this numerator is going to be smaller than this denominator, right?

2:32:52

And if our errors in our line of best fit are much smaller, then that means that this ratio

2:32:59

of the RSS over TSS is going to be very small, which means that R squared is going to go towards

2:33:06

one. And now when R squared is towards one, that means that that's usually a sign that we have a

2:33:14

good predictor. It's one of the signs, not the only one. So over here, I also have, you know,

2:33:24

that there's this adjusted R squared. And what that does, it just adjusts for the number of terms.

2:33:29

So x1, x2, x3, etc. It adjusts for how many extra terms we add, because usually when we,

2:33:37

you know, add an extra term, the R squared value will increase because that'll help us predict

2:33:42

y some more. But the value for the adjusted R squared increase if the new term actually

2:33:48

improves this model fit more than expected, you know, by chance. So that's what adjusted

2:33:54

R squared is. I'm not, you know, it's out of the scope of this one specific course.

2:33:58

And now that's linear regression. Basically, I've covered the concept of residuals or errors.

2:34:05

And, you know, how do we use that in order to find the line of best fit? And you know,

2:34:11

our computer can do all the calculations for us, which is nice. But behind the scenes,

2:34:15

it's trying to minimize that error, right? And then we've gone through all the different

2:34:20

ways of actually evaluating a linear regression model and the pros and cons of each one.

2:34:26

So now let's look at an example. So we're still on supervised learning. But now we're just going to

2:34:31

talk about regression. So what happens when you don't just want to predict, you know, type 123?

2:34:37

What happens if you actually want to predict a certain value? So again, I'm on the UCI machine

2:34:43

learning repository. And here I found this data set about bike sharing in Seoul, South Korea.

2:34:55

So this data set is predicting rental bike count. And here it's the kind of bikes rented at each

2:35:01

hour. So what we're going to do, again, you're going to go into the data folder, and you're going

2:35:08

to download this CSV file. And we're going to move over to collab again. And here I'm going to name

2:35:19

this FCC bikes and regression. I don't remember what I called the last one. But yeah, FCC bikes

2:35:29

regression. Now I'm going to import a bunch of the same things that I did earlier. And, you know,

2:35:39

I'm going to also continue to import the oversampler and the standard scaler. And then I'm actually

2:35:46

also just going to let you guys know that I have a few more things I wanted import. So this is a

2:35:52

library that lets us copy things. Seaborn is a wrapper over a matplotlib. So it also allows us

2:35:59

to plot certain things. And then just letting you know that we're also going to be using

2:36:03

TensorFlow. Okay, so one more thing that we're also going to be using, we're going to use the

2:36:07

sklearn linear model library. Actually, let me make my screen a little bit bigger. So yeah,

2:36:15

awesome. Run this and that'll import all the things that we need. So again, I'm just going to,

2:36:25

you know, give some credit to where we got this data set. So let me copy and paste this UCI thing.

2:36:38

And I will also give credit to this here.

2:36:46

Okay, cool. All right, cool. So this is our data set. And again, it tells us all the different

2:36:54

attributes that we have right here. So I'm actually going to go ahead and paste this in here.

2:37:05

Feel free to copy and paste this if you want me to read it out loud, so you can type it.

2:37:09

It's byte count, hour, temp, humidity, wind, visibility, dew point, temp, radiation, rain,

2:37:18

snow, and functional, whatever that means. Okay, so I'm going to come over here and import my data

2:37:27

by dragging and dropping. All right. Now, one thing that you guys might actually need to do is

2:37:34

you might actually have to open up the CSV because there were, at first, a few like forbidding

2:37:41

characters in mine, at least. So you might have to get rid of like, I think there was a degree here,

2:37:46

but my computer wasn't recognizing it. So I got rid of that. So you might have to go through

2:37:50

and get rid of some of those labels that are incorrect. I'm going to do this. Okay. But

2:37:59

after we've done that, we've imported in here, I'm going to create a data a data frame from that. So,

2:38:07

all right, so now what I can do is I can read that CSV file and I can get the data into here.

2:38:12

So so like data dot CSV. Okay, so now if I call data dot head, you'll see that I have all the

2:38:21

various labels, right? And then I have the data in there. So I'm going to from here, I'm actually

2:38:32

going to get rid of some of these columns that, you know, I don't really care about. So here,

2:38:37

I'm going to, when I when I type this in, I'm going to drop maybe the date, whether or not it's a

2:38:44

holiday, and the various seasons. So I'm just not going to care about these things. Access equals

2:38:53

one means drop it from the columns. So now you'll see that okay, we still have, I mean,

2:38:59

I guess you don't really notice it. But if I set the data frames columns equal to data set calls,

2:39:05

and I look at, you know, the first five things, then you'll see that this is now our data set.

2:39:11

It's a lot easier to read. So another thing is, I'm actually going to

2:39:18

df functional. And we're going to create this. So remember that our computers are not very good

2:39:24

at language, we want it to be in zeros and ones. So here, I will convert that.

2:39:30

Well, if this is equal to yes, then that that gets mapped as one. So then set type integer. All right.

2:39:41

Great. Cool. So the thing is, right now, these by counts are for whatever hour. So

2:39:48

to make this example simpler, I'm just going to index on an hour, and I'm gonna say, okay,

2:39:52

we're only going to use that specific hour. So I'm just going to index on an hour, and I'm

2:39:59

going to use an hour. So here, let's say. So this data frame is only going to be data frame where

2:40:07

the hour, let's say it equals 12. Okay, so it's noon. All right. So now you'll see that all the

2:40:17

equal to 12. And I'm actually going to now drop that column. Our access equals one. Alright,

2:40:31

so we run this cell. Okay, so now we got rid of the hour in here. And we just have the by count,

2:40:38

the temperature, humidity, wind, visibility, and yada, yada, yada. Alright, so what I want to do

2:40:45

is I'm going to actually plot all of these. So for i in all the columns, so the range, length of

2:40:55

whatever its data frame is, and all the columns, because I don't have by count as

2:41:00

actually, it's my first thing. So what I'm going to do is say for a label in data frame,

2:41:06

columns, everything after the first thing, so that would give me the temperature and

2:41:10

onwards. So these are all my features, right? I'm going to just scatter. So I want to see how that

2:41:19

label how that specific data, how that affects the by count. So I'm going to plot the bike count on

2:41:29

the y axis. And I'm going to plot, you know, whatever the specific label is on the x axis.

2:41:35

And I'm going to title this, whatever the label is. And, you know, make my y label, the bike count

2:41:46

at noon. And the x label as just the label. Okay, now, I guess we don't even need the legend.

2:41:58

We don't even need the legend. So just show that plot. All right. So it seems like functional is

2:42:10

not really doesn't really give us any utility. So then snow rain seems like this radiation,

2:42:21

you know, is fairly linear dew point temperature, visibility, wind doesn't really seem like it does

2:42:31

much humidity, kind of maybe like an inverse relationship. But the temperature definitely

2:42:37

looks like there's a relationship between that and the number of bikes, right. So what I'm actually

2:42:41

going to do is I'm going to drop some of the ones that don't don't seem like they really matter. So

2:42:46

maybe wind, you know, visibility. Yeah, so I'm going to get rid of when visibility and functional.

2:42:59

So now data frame, and I'm going to drop wind, visibility, and functional. All right. And the

2:43:13

axis again is the column. So that's one. So if I look at my data set, now, I have just the

2:43:21

temperature, the humidity, the dew point temperature, radiation, rain, and snow. So again,

2:43:27

what I want to do is I want to split this into my training, my validation and my test data set,

2:43:34

just as we talked before. Here, we can use the exact same thing that we just did. And we can say

2:43:42

numpy dot split, and sample, you know that the whole sample, and then create our splits

2:43:54

of the data frame. And we're going to do that. But now set this to eight. Okay.

2:44:04

So I don't really care about, you know, the the full grid, the full array. So I'm just going to

2:44:10

use an underscore for that variable. But I will get my training x and y's. And actually, I don't

2:44:19

have a function for getting the x and y's. So here, I'm going to write a function defined,

2:44:30

get x y. And I'm going to pass in the data frame. And I'm actually going to pass in what the name

2:44:36

of the y label is, and what the x what specific x labels I want to look at. So here, if that's none,

2:44:47

then I'm just like, like, I'm only going to I'm going to get everything from the data set. That's

2:44:51

not the wildlife. So here, I'm actually going to make first a deep copy of my data frame.

2:45:00

And that basically means I'm just copying everything over. If, if like x labels is none,

2:45:08

so if not x labels, then all I'm going to do is say, all right, x is going to be whatever this

2:45:14

data frame is. And I'm just going to take all the columns. So C for C, and data frame, dot columns,

2:45:22

if C does not equal the y label, right, and I'm going to get the values from that. But if there

2:45:32

is the x labels, well, okay, so in order to index only one thing, so like, let's say I pass in only

2:45:40

one thing in here, then my data frame is, so let me make a case for that. So if the length of x

2:45:50

labels is equal to one, then what I'm going to do is just say that this is going to be x labels,

2:46:00

and add that just that label values, and I actually need to reshape to make this 2d.

2:46:08

So I'm going to pass in negative one comma one there. Now, otherwise, if I have like a list of

2:46:15

specific x labels that I want to use, then I'm actually just going to say x is equal to data

2:46:20

frame of those x labels, dot values. And that should suffice. Alright, so now that's just me

2:46:28

extracting x. And in order to get my y, I'm going to do y equals data frame, and then passing the y

2:46:36

label. And at the very end, I'm going to say data equals NP dot h stack. So I'm stacking them horizontally

2:46:45

one next to each other. And I'll take x and y, and return that. Oh, but this needs to be values.

2:46:54

And I'm actually going to reshape this to make it 2d as well so that we can do this h stack.

2:46:59

And I will return data x, y. So now I should be able to say, okay, get x, y, and take that data

2:47:10

frame. And the y label, so my y label is byte count. And actually, so for the x label, I'm actually

2:47:18

going to let's just do like one dimension right now. And earlier, I got rid of the plots, but we

2:47:24

had seen that maybe, you know, the temperature dimension does really well. And we might be able

2:47:30

to use that to predict why. So I'm going to label this also that, you know, it's just using the

2:47:38

temperature. And I am also going to do this again for, oh, this should be train. And this should be

2:47:48

validation. And this should be a test. Because oh, that's Val. Right. But here, it should be Val.

2:48:01

And this should be test. Alright, so we run this and now we have our training validation and test

2:48:08

data sets for just the temperature. So if I look at x train temp, it's literally just the temperature.

2:48:16

Okay, and I'm doing this first to show you simple linear regression. Alright, so right now I can

2:48:23

create a regressor. So I can say the temp regressor here. And then I'm going to, you know, make a

2:48:30

linear regression model. And just like before, I can simply fix fit my x train temp, y train temp

2:48:40

in order to train train this linear regression model. Alright, and then I can also, I can print

2:48:49

this regressor is coefficients and the intercept. So if I do that, okay, this is the coefficient

2:49:02

for whatever the temperature is, and then the the x intercept, okay, or the y intercept, sorry. All

2:49:11

right. And I can, you know, score, so I can get the the r squared score. So I can score x test

2:49:25

and y test. All right, so it's an r squared of around point three eight, which is better than

2:49:35

zero, which would mean, hey, there's absolutely no association. But it's also not, you know, like,

2:49:42

good, it depends on the context. But, you know, the higher that number, it means the higher that

2:49:47

the two variables would be correlated, right? Which here, it's all right. It just means there's

2:49:53

maybe some association between the two. But the reason why I want to do this one D was to show

2:50:00

you, you know, if we plotted this, this is what it would look like. So if I create a scatterplot,

2:50:07

and let's take the training. So this is our data. And then let's make it blue. And then if I

2:50:22

also plotted, so something that I can do is say, you know, the x range, I'm going to plot it,

2:50:29

is when space, and this goes from negative 20 to 40, this piece of data. So I'm going to just say,

2:50:36

let's take 100 things from there. So I'm going to plot x, and I'm going to take this temper,

2:50:47

this, like, regressor, and predict x with that. Okay, and this label, I'm going to label that

2:50:57

the fit. And this color, let's make this red. And let's actually set the line with, so I can,

2:51:08

I can change how thick that value is. Okay. Now at the very end, let's create a legend. And let's,

2:51:21

all right, let's also create, you know, title, all these things that matter, in some sense. So

2:51:30

here, let's just say, this would be the bikes, versus the temperature, right? And the y label

2:51:39

would be number of bikes. And the x label would be the temperature. So I actually think that this

2:51:48

might cause an error. Yeah. So it's expecting a 2d array. So we actually have to reshape this.

2:51:57

Okay, there we go. So I just had to make this an array and then reshape it. So it was 2d. Now,

2:52:15

we see that, all right, this increases. But again, remember those assumptions that we had about

2:52:20

linear regression, like this, I don't really know if this fits those assumptions, right? I just

2:52:26

wanted to show you guys though, that like, all right, this is what a line of s fit through this

2:52:32

data would look like. Okay. Now, we can do multiple linear regression, right. So I'm going to go ahead

2:52:46

and do that as well. Now, if I take my data set, and instead of the labels, it's actually what's

2:52:58

my current data set right now. Alright, so let's just use all of these except for the byte count,

2:53:09

right. So I'm going to just say for the x labels, let's just take the data frames columns and just

2:53:18

remove the byte count. So does that work? So if this part should be of x labels is none. And then

2:53:30

this should work now. Oops, sorry. Okay, so I have Oh, but this here, because it's not just the

2:53:39

temperature anymore, we should actually do this, let's say all, right. So I'm just going to quickly

2:53:48

rerun this piece here so that we have our temperature only data set. And now we have our

2:53:53

all data set. Okay. And this regressor, I can do the same thing. So I can do the all regressor.

2:54:02

And I'm going to make this the linear regression. And I'm going to fit this to x train all and y

2:54:12

train all. Okay. Alright, so let's go ahead and also score this regressor. And let's see how the

2:54:20

R squared performs now. So if I test this on the test data set, what happens? Alright, so our R

2:54:30

square seems to improve it went from point four to point five, two, which is a good sign. Okay.

2:54:38

And I can't necessarily plot, you know, every single dimension. But this just this is just

2:54:44

to say, okay, this is this is improved, right? Alright, so one cool thing that you can do with

2:54:49

tensorflow is you can actually do regression, but with the neural net. So here, I'm going

2:55:00

to we already have our our training data for just the temperature and just, you know, for all the

2:55:08

different columns. So I'm not going to bother with splitting up the data again, I'm just going to go

2:55:13

ahead and start building the model. So in this linear regression model, typically, you know,

2:55:20

it does help if we normalize it. So that's very easy to do with tensorflow, I can just create some

2:55:28

normalizer layer. So I'm going to do tensorflow Keras layers, and get the normalization layer.

2:55:37

And the input shape for that will just be one because let's just do it again on just the

2:55:43

temperature and the access I will make none. Now for this temp normalizer, and I should have had

2:55:53

an equal sign there. I'm going to adapt this to X train temp, and reshape this to just a single vector.

2:56:06

So that should work great. Now with this model, so temp neural net model, what I can do is I can do,

2:56:14

you know, dot keras, sequential. And I'm going to pass in this normalizer layer. And then I'm

2:56:23

going to say, hey, just give me one single dense layer with one single unit. And what that's doing

2:56:29

is saying, all right, well, one single node just means that it's linear. And if you don't add any

2:56:37

sort of activation function to it, the output is also linear. So here, I'm going to have tensorflow

2:56:43

Keras layers dot dense. And I'm just going to have one unit. And that's going to be my model. Okay.

2:56:54

So with this model, let's compile. And for our optimizer, let's use,

2:57:06

let's use the atom again, dot atom, and we have to pass in the learning rate. So learning rate,

2:57:16

and our learning rate, let's do 0.01. And now, the loss, we actually let's get this one 0.1. And the

2:57:26

loss, I'm going to do mean squared error. Okay, so we run that we've compiled it, okay, great.

2:57:34

And just like before, we can call history. And I'm going to fit this model. So here,

2:57:41

if I call fit, I can just fit it, and I'm going to take the x train with the temperature,

2:57:49

but reshape it. Y train for the temperature. And I'm going to set verbose equal to zero so

2:57:57

that it doesn't, you know, display stuff. I'm actually going to set epochs equal to, let's do

2:58:04

1000. And the validation data should be let's pass in the validation data set here

2:58:16

as a tuple. And I know I spelled that wrong. So let's just run this.

2:58:22

And up here, I've copied and pasted the plot loss from our previous but changed the y label

2:58:27

to MSC. Because now we're talking we're dealing with mean squared error. And I'm going to plot

2:58:34

the loss of this history after it's done. So let's just wait for this to finish training and then to

2:58:39

plot. Okay, so this actually looks pretty good. We see that the value is still the same. So

2:58:50

this actually looks pretty good. We see that the values are converging. So now what I can do is

2:58:56

I'm going to go back up and take this plot. And we are going to just run that plot again. So

2:59:07

here, instead of this temperature regressor, I'm going to use the neural net regressor.

2:59:16

This neural net model.

2:59:17

And if I run that, I can see that, you know, this also gives me a linear regressor,

2:59:26

you'll notice that this this fit is not entirely the same as the one

2:59:31

up here. And that's due to the training process of, you know, of this neural net. So just two

2:59:38

different ways to try and try to find the best linear regressor. Okay, but here we're using back

2:59:45

propagation to train a neural net node, whereas in the other one, they probably are not doing that.

2:59:50

Okay, they're probably just trying to actually compute the line of s fit. So, okay, given this,

2:59:59

well, we can repeat the exact same exercise with our with our multiple linear regressions. Okay,

3:00:09

but I'm actually going to skip that part. I will leave that as an exercise to the viewer. Okay,

3:00:14

so now what would happen if we use a neural net, a real neural net instead of just, you know,

3:00:19

one single node in order to predict this. So let's start on that code, we already have our

3:00:24

normalizer. So I'm actually going to take the same setup here. But instead of, you know, this

3:00:31

one dense layer, I'm going to set this equal to 32 units. And for my activation, I'm going to use

3:00:37

Relu. And now let's duplicate that. And for the final output, I just want one answer. So I just

3:00:46

want one cell. And this activation is also going to be Relu, because I can't ever have less than

3:00:52

zero bytes. So I'm just going to set that as Relu. I'm just going to name this the neural net model.

3:00:57

Okay. And at the bottom, I'm going to have this neural net model. I'm going to have this neural

3:01:04

net model, I'm going to compile. And I will actually use the same compiler here. But instead of

3:01:18

instead of a learning rate of 0.01, I'll use 0.001. Okay. And I'm going to train this here.

3:01:27

So the history is this neural net model. And I'm going to fit that against x train temp, y train

3:01:39

temp, and valid validation data, I'm going to set this again equal to x val temp, and y val temp.

3:01:54

Now, for the verbose, I'm going to say equal to zero epochs, let's do 100. And here for the batch

3:02:03

size, actually, let's just not do a batch size right now. Let's just try it. Let's see what happens

3:02:08

here. And again, we can plot the loss of this history after it's done training. So let's just

3:02:18

run this. And that's not what we're supposed to get. So what is going on? Here is sequential,

3:02:26

we have our temperature normalizer, which I'm wondering now if we have to redo that.

3:02:39

Do that. Okay, so we do see this decline, it's an interesting curve, but we do we do see it eventually.

3:02:53

So this is our loss, which all right, if decreasing, that's a good sign.

3:02:57

And actually, what's interesting is let's just let's plot this model again. So here instead of that.

3:03:04

And you'll see that we actually have this like, curve that looks something like this. So actually,

3:03:09

what if I got rid of this activation? Let's train this again. And see what happens.

3:03:21

Alright, so even even when I got rid of that really at the end, it kind of knows, hey, you know, if

3:03:27

it's not the best model, if we had maybe one more layer in here, these are just things that you have

3:03:36

to play around with. When you're, you know, working with machine learning, it's like, you don't really

3:03:41

know what the best model is going to be. For example, this also is not brilliant. But I guess

3:03:53

it's okay. So my point is, though, that with a neural net, I mean, this is not brilliant, but also

3:04:00

there's like no data down here, right? So it's kind of hard for our model to predict. In fact,

3:04:04

we probably should have started the prediction somewhere around here. My point, though, is that

3:04:09

with this neural net model, you can see that this is no longer a linear predictor, but yet we still

3:04:14

get an estimate of the value, right? And we can repeat this exact same exercise, right? So let's

3:04:21

do that. Right. And we can repeat this exact same exercise with the multiple inputs. So here,

3:04:33

if I now pass in all of the data, so this is my all normalizer,

3:04:40

and I should just be able to pass in that. So let's move this to the next cell. Here,

3:04:54

I'm going to pass in my all normalizer. And let's compile it. Yeah, those parameters look good.

3:05:02

Great. So here with the history, when we're trying to fit this model, instead of temp,

3:05:10

we're going to use our larger data set with all the features. And let's just train that.

3:05:22

And of course, we want to plot the loss.

3:05:31

Okay, so that's what our loss looks like. So an interesting curve, but it's decreasing.

3:05:37

So before we saw that our R squared score was around point five, two. Well, we don't really have

3:05:44

that with a neural net anymore. But one thing that we can measure is hey, what is the mean squared

3:05:49

error, right? So if I come down here, and I compare the two mean squared errors, so

3:05:59

so I can predict x test all right. So these are my predictions using that linear regressor,

3:06:14

will linear multiple multiple linear regressor. So these are my live predictions, linear regression.

3:06:20

Okay. I'm actually going to do that at the bottom. So let me just copy and paste that cell and bring

3:06:32

it down here. So now I'm going to calculate the mean squared error for both the linear regressor

3:06:41

and the neural net. Okay, so this is my linear and this is my neural net. So if I do my neural net

3:06:51

model, and I predict x test all, I get my two, you know, different y predictions. And I can calculate

3:07:03

the mean squared error, right? So if I want to get the mean squared error, and I have y prediction

3:07:11

and y real, I can do numpy dot square, and then I would need the y prediction minus, you know, the

3:07:19

real. So this this is basically squaring everything. And this should be a vector. So if I just take

3:07:31

this entire thing and take the mean of that, that should give me the MSC. So let's just try that out.

3:07:44

And the y real is y test all, right? So that's my mean squared error for the linear regressor.

3:07:52

And this is my mean squared error for the neural net. So that's interesting. I will debug this live,

3:08:04

I guess. So my guess is that it's probably coming from this normalization layer. Because this input

3:08:14

shape is probably just six. And okay, so that works now. And the reason why is because, like,

3:08:33

my inputs are only for every vector, it's only a one dimensional vector of length six. So I should

3:08:39

have I should have just had six, comma, which is a tuple of size six from the start, or it's a it's

3:08:46

a tuple containing one element, which is a six. Okay, so it's actually interesting that my neural

3:08:54

net results seem like they they have a larger mean squared error than my linear regressor.

3:09:00

One thing that we can look at is, we can actually plot the real versus, you know, the the actual

3:09:09

results versus what the predictions are. So if I say, some access, and I use plt dot axes, and make

3:09:21

axes and make these equal, then I can scatter the the y, you know, the test. So what the actual

3:09:31

values are on the x axis, and then what the prediction are on the x axis. Okay. And I can

3:09:40

label this as the linear regression predictions. Okay, so then let me just label my axes. So the

3:09:50

x axis, I'm going to say is the true values. The y axis is going to be my linear regression predictions.

3:10:04

Or actually, let's plot. Let's just make this predictions.

3:10:09

And then at the end, I'm going to plot. Oh, let's set some limits.

3:10:22

Because I think that's like approximately the max number of bikes.

3:10:28

So I'm going to set my x limit to this and my y limit to this.

3:10:35

So here, I'm going to pass that in here too. And all right, this is what we actually get for our

3:10:46

linear regressor. You see that actually, they align quite well, I mean, to some extent. So 2000 is

3:10:54

probably too much 2500. I mean, looks like maybe like 1800 would be enough here for our limits.

3:11:03

And I'm actually going to label something else, the neural net predictions.

3:11:12

Let's add a legend. So you can see that our neural net for the larger values, it seems like

3:11:22

it's a little bit more spread out. And it seems like we tend to underestimate a little bit down

3:11:28

here in this area. Okay. And for some reason, these are way off as well.

3:11:37

But yeah, so we've basically used a linear regressor and a neural net. Honestly, there are

3:11:44

sometimes where a neural net is more appropriate and a linear regressor is more appropriate.

3:11:49

I think that it just comes with time and trying to figure out, you know, and just literally seeing

3:11:54

like, hey, what works better, like here, a linear, a multiple linear regressor might actually work

3:11:59

better than a neural net. But for example, with the one dimensional case, a linear regressor would

3:12:05

never be able to see this curve. Okay. I mean, I'm not saying this is a great model either, but I'm

3:12:12

just saying like, hey, you know, sometimes it might be more appropriate to use something that's not

3:12:19

linear. So yeah, I will leave regression at that. Okay, so we just talked about supervised learning.

3:12:29

And in supervised learning, we have data, we have some a bunch of features and for a bunch of

3:12:34

different samples. But each of those samples has some sort of label on it, whether that's a number,

3:12:39

a category, a class, etc. Right, we were able to use that label in order to try to predict

3:12:46

right, we were able to use that label in order to try to predict new labels of other points that

3:12:51

we haven't seen yet. Well, now let's move on to unsupervised learning. So with unsupervised

3:12:59

learning, we have a bunch of unlabeled data. And what can we do with that? You know, can we learn

3:13:05

anything from this data? So the first algorithm that we're going to discuss is known as k means

3:13:13

clustering. What k means clustering is trying to do is it's trying to compute k clusters from the data.

3:13:25

So in this example below, I have a bunch of scattered points. And you'll see that this

3:13:31

is x zero and x one on the two axes, which means I'm actually plotting two different features,

3:13:38

right of each point, but we don't know what the y label is for those points. And now, just looking

3:13:44

at these scattered points, we can kind of see how there are different clusters in the data set,

3:13:51

right. So depending on what we pick for k, we might have different clusters. Let's say k equals two,

3:14:00

right, then we might pick, okay, this seems like it could be one cluster, but this here is also

3:14:05

another cluster. So those might be our two different clusters. If we have k equals three,

3:14:13

for example, then okay, this seems like it could be a cluster. This seems like it could be a

3:14:18

cluster. And maybe this could be a cluster, right. So we could have three different clusters in the

3:14:23

data set. Now, this k here is predefined, if I can spell that correctly, by the person who's running

3:14:33

the model. So that would be you. All right. And let's discuss how you know, the computer actually

3:14:42

goes through and computes the k clusters. So I'm going to write those steps down here.

3:14:52

Now, the first step that happens is we actually choose well, the computer chooses three random

3:15:01

points on this plot to be the centroids. And by centuries, I just mean the center of the clusters.

3:15:11

Okay. So three random points, let's say we're doing k equals three, so we're choosing three

3:15:16

random points to be the centroids of the three clusters. If it were two, we'd be choosing two

3:15:21

random points. Okay. So maybe the three random points I'm choosing might be here.

3:15:27

Here, here, and here. All right. So we have three different points. And the second thing that we do

3:15:44

is we actually calculate

3:15:46

the distance for each point to those centroids. So between all the points and the centroid.

3:16:01

So basically, I'm saying, all right, this is this distance, this distance, this distance,

3:16:07

all of these distances, I'm computing between oops, not those two, between the points, not the

3:16:13

centroids themselves. So I'm computing the distances for all of these plots to each of the centroids.

3:16:20

Okay. And that comes with also assigning those points to the closest centroid.

3:16:34

What do I mean by that? So let's take this point here, for example, so I'm computing

3:16:42

this distance, this distance, and this distance. And I'm saying, okay, it seems like the red one

3:16:46

is the closest. So I'm actually going to put this into the red centroid. So if I do that for

3:16:54

all of these points, it seems slightly closer to red, and this one seems slightly closer to red,

3:17:03

right? Now for the blue, I actually wouldn't put any blue ones in here, but we would probably

3:17:13

actually, that first one is closer to red. And now it seems like the rest of them are probably

3:17:21

closer to green. So let's just put all of these into green here, like that. And cool. So now we

3:17:31

have, you know, our two, three, technically centroid. So there's this group here, there's

3:17:38

this group here. And then blue is kind of just this group here, it hasn't really touched any

3:17:44

of the points yet. So the next step, three that we do is we actually go and we recalculate the

3:17:54

centroid. So we compute new centroids based on the points that we have in all the centroids.

3:18:04

And by that, I just mean, okay, well, let's take the average of all these points. And where is that

3:18:10

new centroid? That's probably going to be somewhere around here, right? The blue one, we don't have

3:18:15

any points in there. So we won't touch and then the screen one, we can put that probably somewhere

3:18:22

over here, oops, somewhere over here. Right. So now if I erase all of the previously computed centroids,

3:18:38

I can go and I can actually redo step two over here, this calculation.

3:18:45

Alright, so I'm going to go back and I'm going to iterate through everything again,

3:18:48

and I'm going to recompute my three centroids. So let's see, we're going to take this red point,

3:18:55

these are definitely all red, right? This one still looks a bit red. Now,

3:19:03

this part, we actually start getting closer to the blues.

3:19:08

So this one still seems closer to a blue than a green, this one as well. And I think the rest

3:19:16

would belong to green. Okay, so now our three centroids are three, sorry, our three clusters

3:19:26

would be this, this, and then this, right? Those are our three centroids. And so now we go back

3:19:39

and we compute the new sorry, those would be the three clusters. So now we go back and we compute

3:19:44

the three centroids. So I'm going to get rid of this, this and this. And now where would this

3:19:50

red be centered, probably closer, you know, to this point here, this blue might be closer to

3:19:57

up here. And then this green would probably be somewhere. It's pretty similar to what we had

3:20:05

before. But it seems like it'd be pulled down a bit. So probably somewhere around there for green.

3:20:10

All right. And now, again, we go back and we compute the distance between all the points

3:20:20

and the centroids. And then we assign them to the closest centroid. Okay. So the reds are all here,

3:20:27

it's very clear. Actually, let me just circle that. And this it actually seems like this point is

3:20:36

it actually seemed like this point is closer to this blue now. So the blues seem like they would

3:20:43

be maybe this point looks like it'd be blue. So all these look like they would be blue now.

3:20:50

And the greens would probably be this cluster right here. So we go back, we compute the centroids,

3:20:58

bam. This one probably like almost here, bam. And then the green looks like it would be probably

3:21:10

here ish. Okay. And now we go back and we compute the we compute the clusters again.

3:21:21

So red, still this blue, I would argue is now this cluster here. And green is this cluster here.

3:21:33

Okay, so we go and we recompute the centroids, bam, bam. And, you know, bam. And now if I were

3:21:48

to go and assign all the points to clusters again, I would get the exact same thing. Right. And so

3:21:54

that's when we know that we can stop iterating between steps two and three is when we've

3:21:59

converged on some solution when we've reached some stable point. And so now because none of

3:22:06

these points are really changing out of their clusters anymore, we can go back to the user

3:22:10

and say, Hey, these are our three clusters. Okay. And this process, something known as

3:22:20

expectation maximization. This part where we're assigning the points to the closest centroid,

3:22:33

this is something this is our expectation step. And this part where we're computing the new

3:22:41

centroids, this is our maximization step. Okay, so that's expectation maximization.

3:22:55

And we use this in order to compute the centroids, assign all the points to clusters,

3:23:02

according to those centroids. And then we're recomputing all that over again, until we reach

3:23:07

some stable point where nothing is changing anymore. Alright, so that's our first example

3:23:13

of unsupervised learning. And basically, what this is doing is trying to find some structure,

3:23:19

some pattern in the data. So if I came up with another point, you know, might be somewhere here,

3:23:25

I can say, Oh, it looks like that's closer to if this is a, b, c, it looks like that's closest to

3:23:32

cluster B. And so I would probably put it in cluster B. Okay, so we can find some structure

3:23:38

in the data based on just how, how the points are scattered relative to one another. Now,

3:23:46

the second unsupervised learning technique that I'm going to discuss with you guys, something noted,

3:23:50

principal component analysis. And the point of principal component analysis is very often it's

3:23:57

used as a dimensionality reduction technique. So let me write that down. It's used for dimensionality

3:24:07

reduction. And what do I mean by dimensionality reduction is if I have a bunch of features like

3:24:15

x1 x2 x3 x4, etc. Can I just reduce that down to one dimension that gives me the most information

3:24:23

about how all these points are spread relative to one another. And that's what PCA is for. So PCA

3:24:29

principal component analysis. Let's say I have some points in the x zero and x one feature space.

3:24:42

Okay, so these points might be spread, you know, something like this.

3:24:59

Okay. So for example, if this were something to do with housing prices, right,

3:25:08

this here might be x zero might be hey, years since built, right, since the house was built,

3:25:19

and x one might be square footage of the house. Alright, so like years since built, I mean, like

3:25:29

right now it's been, you know, 22 years since a house in 2000 was built. Now principal component

3:25:36

analysis is just saying, alright, let's say we want to build a model, or let's say we want to,

3:25:40

you know, display something about our data, but we don't we don't have two axes to show it on.

3:25:49

How do we display, you know, how do we how do we demonstrate that this point is a further away from

3:25:56

this point than this point. And we can do that using principal component analysis. So

3:26:04

take what you know about linear regression and just forget about it for a second. Otherwise,

3:26:07

you might get confused. PCA is a way of trying to find direction in the space with the largest

3:26:16

variance. So this principal component, what that means is basically the component.

3:26:23

So some direction in this space with the largest variance, okay, it tells us the most about our

3:26:38

data set without the two different dimensions. Like, let's say we have these two different

3:26:42

mentions, and somebody's telling us, hey, you only get one dimension in order to show your data set.

3:26:48

What dimension do you want to show us? Okay, so let's say we want to show our data set,

3:26:53

what dimension like what do we do, we want to project our data onto a single dimension.

3:27:00

Alright, so that in this case might be a dimension that looks something like

3:27:06

this. And you might say, okay, we're not going to talk about linear regression, okay.

3:27:11

We don't have a y value. So linear regression, this would be why this is not why, okay, we don't

3:27:16

have a label for that. Instead, what we're doing is we're taking the right angle projection. So

3:27:23

all of these take that's not very visible. But take this right angle projection onto this line.

3:27:33

And what PCA is doing is saying, okay, map all of these points onto this one dimensional space.

3:27:39

So the transformed data set would be here.

3:27:44

This one's on the data sets are on the line. So we just put that there. But now this would be our

3:27:49

new one dimensional data set. Okay, it's not our prediction or anything. This is our new data set.

3:27:57

If somebody came to us said you only get one dimension, you only get one number to represent

3:28:02

each of these 2d points. What number would you give us? What number would you give us?

3:28:06

So this would be our new one dimensional data set. Okay, it's not our prediction or anything.

3:28:13

What number would you give me? This would be the number that we gave. Okay, this in this direction,

3:28:24

this is where our points are the most spread out. Right? If I took this plot,

3:28:31

and let me actually duplicate this so I don't have to rewrite anything.

3:28:36

Or so I don't have to erase and then redraw anything. Let me get rid of some of this stuff.

3:28:47

And I just got rid of a point there too. So let me draw that back.

3:28:54

Alright, so if this were my original data point, what if I had taken, you know, this to be

3:29:01

the PCA dimension? Okay, well, I then would have points that let me actually do that in different

3:29:12

color. So if I were to draw a right angle to this for every point, my points would look something

3:29:24

like this. And so just intuitively looking at these two different plots, this top one and this one,

3:29:37

we can see that the points are squished a little bit closer together. Right? Which means that the

3:29:43

variance that's not the space with the largest variance. The thing about the largest variance

3:29:48

is that this will give us the most discrimination between all of these points. The larger the

3:29:55

variance, the further spread out these points will likely be. Now, and so that's the that's the

3:30:01

dimension that we should project it on a different way to actually look at that, like what is the

3:30:07

dimension with the largest variance. It's actually it also happens to be the dimension that decreases

3:30:14

to be the dimension that decreases that minimizes the residuals. So if we take all the points, and

3:30:25

we take the residual from that the XY residual, so in linear regression, in linear regression,

3:30:33

we were looking only at this residual, the differences between the predictions right between

3:30:37

y and y hat, it's not that here in principal component analysis, we're taking the difference

3:30:44

from our current point in two dimensional space, and then it's projected point. Okay, so we're

3:30:52

taking that dimension. And we're saying, alright, how much, you know, how much distance is there

3:31:00

between that projection residual, and we're trying to minimize that for all of these points. So that

3:31:08

actually equates to this largest variance dimension, this dimension here, the PCA dimension,

3:31:21

you can either look at it as minimizing, minimize, let me get rid of this,

3:31:34

the projection residuals. So that's the stuff in orange.

3:31:42

Or to maximizing the variance between the points.

3:31:48

Okay. And we're not really going to talk about, you know, the method that we need in order to

3:31:55

calculate out the principal components, or like what that projection would be, because you will

3:32:00

need to understand linear algebra for that, especially eigenvectors and eigenvalues, which

3:32:06

I'm not going to cover in this class. But that's how you would find the principal components. Okay,

3:32:12

now, with this two dimensional data set here, sorry, this one dimensional data set, we started

3:32:16

from a 2d data set, and we now boil it down to one dimension. Well, we can go and take that

3:32:22

dimension, and we can do other things with it. Right, we can, like if there were a y label,

3:32:27

then we can now show x versus y, rather than x zero and x one in different plots with that y.

3:32:35

Now we can just say, oh, this is a principal component. And we're going to plot that with

3:32:38

the y. Or for example, if there were 100 different dimensions, and you only wanted to take five of

3:32:44

them, well, you could go and you could find the top five PCA dimensions. And that might be a lot

3:32:51

more useful to you than 100 different feature vector values. Right. So that's principal component

3:32:58

analysis. Again, we're taking, you know, certain data that's unlabeled, and we're trying to make

3:33:05

some sort of estimation, like some guess about its structure from that original data set, if we

3:33:13

wanted to take, you know, a 3d thing, so like a sphere, but we only have a 2d surface to draw it

3:33:20

on. Well, what's the best approximation that we can make? Oh, it's a circle. Right PCA is kind of

3:33:26

the same thing. It's saying if we have something with all these different dimensions, but we can't

3:33:30

show all of them, how do we boil it down to just one dimension? How do we extract the most

3:33:35

information from that multiple dimensions? And that is exactly either you minimize the projection

3:33:43

residuals, or you maximize the variance. And that is PCA. So we'll go through an example of that.

3:33:50

Now, finally, let's move on to implementing the unsupervised learning part of this class.

3:33:57

Here, again, I'm on the UCI machine learning repository. And I have a seeds data set where,

3:34:04

you know, I have a bunch of kernels that belong to three different types of wheat. So there's

3:34:09

comma, Rosa and Canadian. And the different features that we have access to are, you know,

3:34:17

geometric parameters of those wheat kernels. So the area perimeter, compactness, length, width,

3:34:23

width, asymmetry, and the length of the kernel groove. Okay, so all of these are real values,

3:34:30

which is easy to work with. And what we're going to do is we're going to try to predict,

3:34:36

or I guess we're going to try to cluster the different varieties of the wheat.

3:34:41

So let's get started. I have a colab notebook open again. Oh, you're gonna have to, you know,

3:34:46

go to the data folder, download this. And so I'm going to go to the data folder, download this,

3:34:52

and let's get started. So the first thing to do is to import our seeds data set into our colab

3:35:04

notebook. So I've done that here. Okay, and then we're going to import all the classics again,

3:35:11

so pandas. And then I'm also going to import seedborn because I'm going to want that for this

3:35:28

specific class. Okay. Great. So now our columns that we have in our seed data set are the area,

3:35:40

the perimeter, the compactness, the length, with asymmetry, groove, length, I mean, I'm just going

3:35:54

to call it groove. And then the class, right, the wheat kernels class. So now we have to import this,

3:36:00

I'm going to do that using pandas read CSV. And it's called seeds data.csv. So I'm going to turn

3:36:11

that into a data frame. And the names are equal to the columns over here. So what happens if I just

3:36:19

do that? Oops, what did I call this seeds data set text? Alright, so if we actually look at our

3:36:29

data frame right now, you'll notice something funky. Okay. And here, you know, we have all the

3:36:36

stuff under area. And these are all our numbers with some dash t. So the reason is because we

3:36:42

haven't actually told pandas what the separator is, which we can do like this. And this t that's

3:36:50

just a tab. So in order to ensure that like all whitespace gets recognized as a separator,

3:36:56

we can actually this is for like a space. So any spaces are going to get recognized as data

3:37:04

separators. So if I run that, now our this, you know, this is a lot better. Okay. Okay.

3:37:14

So now let's actually go and like visualize this data. So what I'm actually going to do is plot

3:37:20

each of these against one another. So in this case, pretend that we don't have access to the

3:37:26

class, right? Pretend that so this class here, I'm just going to show you in this example,

3:37:31

that like, hey, we can predict our classes using unsupervised learning. But for this example,

3:37:36

in unsupervised learning, we don't actually have access to the class. So I'm going to just try to

3:37:41

plot these against one another and see what happens. So for some I in range, you know,

3:37:49

the columns minus one because the classes in the columns. And I'm just going to say for j in range,

3:37:57

so take everything from I onwards, you know, so I like the next thing after I until the end of this.

3:38:06

So this will give us basically a grid of all the different like combinations. And our x label is

3:38:15

going to be columns I our y label is going to be the columns j. So those are our labels up here.

3:38:25

And I'm going to use seaborne this time. And I'm going to say scatter my data. So our x is going

3:38:34

to be our x label. Or y is going to be our y label. And our data is going to be the data frame that

3:38:46

we're passing in. So what's interesting here is that we can say hue. And what this will do is say,

3:38:53

like if I give this class, it's going to separate the three different classes into three different

3:38:57

hues. So now what we're doing is we're basically comparing the area and the perimeter or the area

3:39:03

and the compactness. But we're going to visualize, you know, what classes they're in. So let's go

3:39:10

ahead and I might have to show. So great. So basically, we can see perimeter and area we give

3:39:22

we get these three groups. The area compactness, we get these three groups, and so on. So these all

3:39:31

kind of look honestly like somewhat similar. Right, so Wow, look at this one. So this one,

3:39:40

we have the compactness and the asymmetry. And it looks like there's not really I mean,

3:39:44

it just looks like they're blobs, right? Sure, maybe class three is over here more, but

3:39:50

one and two kind of look like they're on top of each other. Okay. I mean, there are some that

3:39:55

might look slightly better in terms of clustering. But let's go through some of the some of the

3:40:00

clustering examples that we talked about, and try to implement those. The first thing that we're

3:40:05

going to do is just straight up clustering. So what we learned about was k means clustering.

3:40:16

So from SK learn, I'm going to import k means. Okay. And just for the sake of being able to run,

3:40:29

you know, any x and any y, I'm just going to say, hey, let's use some x. What's a good one, maybe.

3:40:40

I mean, perimeter asymmetry could be a good one. So x could be perimeter, y could be asymmetry.

3:40:47

Okay. And for this, the x values, I'm going to just extract those specific values.

3:40:59

Alright, well, let's make a k means algorithm, or let's, you know, define this. So k means,

3:41:09

and in this specific case, we know that the number of clusters is three. So let's just use that. And

3:41:15

I'm going to fit this against this x that I've just defined right here. Right. So, you know, if I

3:41:27

create this clusters, so one thing, one cool thing is I can actually go to this clusters, and I can

3:41:33

say k mean dot labels. And it'll give give me if I can type correctly, it'll give me what its

3:41:43

predictions for all the clusters are. And our actual, oops, not that. If we go to the data frame,

3:41:52

and we get the class, and the values from those, we can actually compare these two and say, hey,

3:41:59

like, you know, everything in general, most of the zeros that it's predicted, are the ones, right.

3:42:05

And in general, the twos are the twos here. And then this third class one, okay, that corresponds

3:42:11

to three. Now remember, these are separate classes. So the labels, what we actually call them don't

3:42:16

really matter. We can say a map zero to one map two to two and map one to three. Okay, and our,

3:42:23

you know, our mapping would do fairly well. But we can actually visualize this. And in order to do

3:42:30

that, I'm going to create this cluster cluster data frame. So I'm going to create a data frame.

3:42:40

And I'm going to pass in a horizontally stacked array with x, so my values for x and y. And then

3:42:51

the clusters that I have here, but I'm going to reshape them. So it's 2d.

3:42:58

Okay. And the columns, the labels for that are going to be x, y, and plus. Okay. So I'm going

3:43:14

to go ahead and do that same seaborne scatter plot. Again, where x is x, y is y. And now,

3:43:23

the hue is again the class. And the data is now this cluster data frame. Alright, so this here,

3:43:35

this here is my k means like, I guess classes.

3:43:42

So k means kind of looks like this. If I come down here and I plot, you know, my original data frame,

3:43:54

this is my original classes with respect to this specific x and y. And you'll see that, honestly,

3:44:01

like it doesn't do too poorly. Yeah, there's I mean, the colors are different, but that's fine.

3:44:07

For the most part, it gets information of the clusters, right. And now we can do that with

3:44:16

higher dimensions. So with the higher dimensions, if we make x equal to, you know, all the columns,

3:44:25

except for the last one, which is our class, we can do the exact same thing.

3:44:31

We can do the exact same thing. So here, and we can

3:44:43

predict this. But now, our columns are equal to our data frame columns all the way to the last one.

3:44:55

And then with this class, actually, so we can literally just say data frame columns.

3:45:02

And we can fit all of this. And now, if I want to plot the k means classes.

3:45:11

Alright, so this was my that's my clustered and my original. So actually, let me see if I can

3:45:20

get these on the same page. So yeah, I mean, pretty similar to what we just saw. But what's

3:45:27

actually really cool is even something like, you know, if we change. So what's one of them

3:45:36

where they were like on top of each other? Okay, so compactness and asymmetry, this one's messy.

3:45:47

Right. So if I come down here, and I say compactness and asymmetry, and I'm trying to do this in 2d,

3:45:58

this is what my scatterplot. So this is what you know, my k means is telling me for these two

3:46:05

dimensions for compactness and asymmetry, if we just look at those two, these are our three classes,

3:46:12

right? And we know that the original looks something like this. And are these two remotely

3:46:18

alike? No. Okay, so now if I come back down here, and I rerun this higher dimensions one,

3:46:25

but actually, this clusters, I need to get the labels of the k means again.

3:46:34

Okay, so if I rerun this with higher dimensions,

3:46:38

well, if we zoom out, and we take a look at these two, sure, the colors are mixed up. But in general,

3:46:45

there are the three groups are there, right? This does a much better job at assessing, okay,

3:46:52

what group is what. So, for example, we could relabel the one in the original class to two.

3:47:01

And then we could make sorry, okay, this is kind of confusing. But for example, if this light pink

3:47:08

were projected onto this darker pink here, and then this dark one was actually the light pink,

3:47:15

and this light one was this dark one, then you kind of see like these correspond to one another,

3:47:21

right? Like even these two up here are the same class as all the other ones over here, which are

3:47:26

the same in the same color. So you don't want to compare the two colors between the plots,

3:47:31

you want to compare which points are in what colors in each of the plots. So that's one cool

3:47:37

application. So this is how k means functions, it's basically taking all the data sets and saying,

3:47:44

All right, where are my clusters given these pieces of data? And then the next thing that we

3:47:50

talked about is PCA. So PCA, we're reducing the dimension, but we're mapping all these like,

3:47:58

you know, seven dimensions. I don't know if there are seven, I made that number up, but we're

3:48:02

mapping multiple dimensions into a lower dimension number. Right. And so let's see how that works.

3:48:10

So from SK learn decomposition, I can import PCA and that will be my PCA model.

3:48:16

So if I do PCA component, so this is how many dimensions you want to map it into.

3:48:22

And you know, for this exercise, let's do two. Okay, so now I'm taking the top two dimensions.

3:48:29

And my transformed x is going to be PCA dot fit transform, and the same x that I had up here.

3:48:39

And the same x that I had up here. Okay, so all the other all the values basically, area,

3:48:46

perimeter, compactness, length, width, asymmetry, groove. Okay. So let's run that. And we've

3:48:54

transformed it. So let's look at what the shape of x used to be. So they're okay. So seven was right,

3:49:02

I had 210 samples, each seven, seven features long, basically. And now my transformed x

3:49:14

is 210 samples, but only of length two, which means that I only have two dimensions now that

3:49:20

I'm plotting. And we can actually even take a look at, you know, the first five things.

3:49:27

Okay, so now we see each each one is a two dimensional point,

3:49:30

each sample is now a two dimensional point in our new in our new dimensions.

3:49:38

So what's cool is I can actually scatter these

3:49:46

zero and transformed x. So I actually have to

3:49:53

take the columns here. And if I show that,

3:50:01

basically, we've just taken this like seven dimensional thing, and we've made it into a

3:50:06

single or I guess to a two dimensional representation. So that's a point of PCA.

3:50:13

And actually, let's go ahead and do the same clustering exercise as we did up here. If I take

3:50:20

the k means this PCA data frame, I can let's construct data frame out of that. And the data

3:50:29

frame is going to be H stack. I'm going to take this transformed x and the clusters that reshape.

3:50:40

So actually, instead of clusters, I'm going to use k means dot labels. And I need to reshape this.

3:50:46

So it's 2d. So we can do the H stack. And for the columns, I'm going to set this to PCA one PCA two,

3:50:59

and the class. All right. So now if I take this, I can also do the same for the truth.

3:51:08

But instead of the k means labels, I want from the data frame the original classes.

3:51:13

And I'm just going to take the values from that. And so now I have a data frame for the k means

3:51:20

with PCA and then a data frame for the truth with also the PCA. And I can now plot these similarly

3:51:27

to how I plotted these up here. So let me actually take these two.

3:51:32

Instead of the cluster data frame, I want the this is the k means PCA data frame. This is still going

3:51:41

to be class, but now x and y are going to be the two PCA dimensions. Okay. So these are my two PCA

3:51:51

dimensions. And you can see that the data frame is going to be the same as the cluster data frame.

3:51:58

So these are my two PCA dimensions. And you can see that, you know, they're, they're pretty spread

3:52:05

out. And then here, I'm going to go to my truth classes. Again, it's PCA one PCA two, but instead

3:52:14

of k means this should be truth PCA data frame. So you can see that like in the truth data frame

3:52:22

along these two dimensions, we actually are doing fairly well in terms of separation, right? It does

3:52:29

seem like this is slightly more separable than the other like dimensions that we had been looking at

3:52:36

up here. So that's a good sign. And up here, you can see that hey, some of these correspond to one

3:52:45

another. I mean, for the most part, our algorithm or unsupervised clustering algorithm is able to

3:52:51

to give us is able to spit out, you know, what the proper labels are. I mean, if you map these

3:52:59

specific labels to the different types of kernels. But for example, this one might all be the comma

3:53:05

kernel kernels and same here. And then these might all be the Canadian kernels. And these might all

3:53:09

be the Canadian kernels. So it does struggle a little bit with, you know, where they overlap.

3:53:14

But for the most part, our algorithm is able to find the three different categories, and do a

3:53:21

fairly good job at predicting them without without any information from us, we haven't given our

3:53:26

algorithm any labels. So that's a gist of unsupervised learning. I hope you guys enjoyed

3:53:32

this course. I hope you know, a lot of these examples made sense. If there are certain things

3:53:38

that I have done, and you know, you're somebody with more experience than me, please let me know

3:53:44

in the comments and we can all as a community learn from this together. So thank you all for watching.

Machine Learning for Everybody – Full Course

More from freeCodeCamp.org

Trending Transcripts