But what is a neural network? | Deep learning chapter 1

0:04 This is a 3.

0:06 It's sloppily written and rendered

0:07 at an extremely low resolution of 28x28 pixels,

0:10 but your brain has no trouble recognizing it as a 3.

0:14 And I want you to take a moment to appreciate

0:16 how crazy it is that brains can do this so effortlessly.

0:19 I mean, this, this and this are also recognizable as 3s,

0:22 even though the specific values of each pixel

0:25 is very different from one image to the next.

0:28 The particular light-sensitive cells in your eye that are firing when you see

0:33 this 3 are very different from the ones firing when you see this 3.

0:37 But something in that crazy-smart visual cortex

0:40 of yours resolves these as representing the same idea,

0:43 while at the same time recognizing other images as their own distinct ideas.

0:49 But if I told you, hey, sit down and write for me a program that takes in a grid

0:54 of 28x28 pixels like this and outputs a single number between 0 and 10,

0:59 telling you what it thinks the digit is,

1:01 well the task goes from comically trivial to dauntingly difficult.

1:07 Unless you've been living under a rock,

1:08 I think I hardly need to motivate the relevance and importance

1:11 of machine learning and neural networks to the present and to the future.

1:15 But what I want to do here is show you what a neural network actually is,

1:18 assuming no background,

1:19 and to help visualize what it's doing, not as a buzzword but as a piece of math.

1:25 My hope is that you come away feeling like the structure itself is motivated,

1:28 and to feel like you know what it means when you read,

1:31 or you hear about a neural network quote-unquote learning.

1:35 This video is just going to be devoted to the structure

1:37 component of that, and the following one is going to tackle learning.

1:40 What we're going to do is put together

1:42 a neural network that can learn to recognize handwritten digits.

1:49 This is a somewhat classic example for introducing the topic,

1:52 and I'm happy to stick with the status quo here,

1:54 because at the end of the two videos I want to point

1:56 you to a couple good resources where you can learn more,

1:58 and where you can download the code that does

2:01 this and play with it on your own computer.

2:05 There are many many variants of neural networks,

2:07 and in recent years there's been sort

2:09 of a boom in research towards these variants,

2:12 but in these two introductory videos you and I are just going

2:15 to look at the simplest plain vanilla form with no added frills.

2:19 This is kind of a necessary prerequisite

2:21 for understanding any of the more powerful modern variants,

2:24 and trust me it still has plenty of complexity for us to wrap our minds around.

2:29 But even in this simplest form it can learn to recognize handwritten digits,

2:33 which is a pretty cool thing for a computer to be able to do.

2:37 And at the same time you'll see how it does fall

2:39 short of a couple hopes that we might have for it.

2:43 As the name suggests neural networks are inspired by the brain,

2:46 but let's break that down.

2:48 What are the neurons, and in what sense are they linked together?

2:52 Right now when I say neuron all I want you

2:55 to think about is a thing that holds a number,

2:58 specifically a number between 0 and 1.

3:00 It's really not more than that.

3:03 For example the network starts with a bunch of neurons

3:07 corresponding to each of the 28x28 pixels of the input image,

3:12 which is 784 neurons in total.

3:14 Each one of these holds a number

3:17 that represents the grayscale value of the corresponding pixel,

3:20 ranging from 0 for black pixels up to 1 for white pixels.

3:25 This number inside the neuron is called its activation,

3:28 and the image you might have in mind here is that each

3:31 neuron is lit up when its activation is a high number.

3:36 So all of these 784 neurons make up the first layer of our network.

3:46 Now jumping over to the last layer,

3:48 this has 10 neurons, each representing one of the digits.

3:52 The activation in these neurons, again some number that's between 0 and 1,

3:56 represents how much the system thinks

3:58 that a given image corresponds with a given digit.

4:03 There's also a couple layers in between called the hidden layers,

4:06 which for the time being should just be a giant question mark

4:09 for how on earth this process of recognizing digits is going to be handled.

4:14 In this network I chose two hidden layers,

4:16 each one with 16 neurons, and admittedly that's kind of an arbitrary choice.

4:21 To be honest I chose two layers based on how

4:23 I want to motivate the structure in just a moment,

4:25 and 16, well that was just a nice number to fit on the screen.

4:28 In practice there is a lot of room

4:30 for experiment with a specific structure here.

4:33 The way the network operates,

4:34 activations in one layer determine the activations of the next layer.

4:39 And of course the heart of the network

4:41 as an information processing mechanism comes down

4:43 to exactly how those activations from one

4:46 layer bring about activations in the next layer.

4:49 It's meant to be loosely analogous to how in biological networks of neurons,

4:53 some groups of neurons firing cause certain others to fire.

4:58 Now the network I'm showing here has already been trained to recognize digits,

5:01 and let me show you what I mean by that.

5:03 It means if you feed in an image, lighting up all 784 neurons of the input layer

5:08 according to the brightness of each pixel in the image,

5:11 that pattern of activations causes some very specific pattern in the next

5:15 layer which causes some pattern in the one after it,

5:18 which finally gives some pattern in the output layer.

5:22 And the brightest neuron of that output layer is the network's choice,

5:26 so to speak, for what digit this image represents.

5:32 And before jumping into the math for how one layer influences the next,

5:36 or how training works, let's just talk about why it's even reasonable

5:40 to expect a layered structure like this to behave intelligently.

5:44 What are we expecting here?

5:45 What is the best hope for what those middle layers might be doing?

5:48 Well, when you or I recognize digits, we piece together various components.

5:54 A 9 has a loop up top and a line on the right.

5:57 An 8 also has a loop up top, but it's paired with another loop down low.

6:01 A 4 basically breaks down into three specific lines, and things like that.

6:07 Now in a perfect world, we might hope that each neuron in the second

6:11 to last layer corresponds with one of these subcomponents,

6:14 that anytime you feed in an image with, say, a loop up top,

6:18 like a 9 or an 8, there's some specific

6:21 neuron whose activation is going to be close to 1.

6:24 And I don't mean this specific loop of pixels,

6:26 the hope would be that any generally loopy

6:29 pattern towards the top sets off this neuron.

6:32 That way, going from the third layer to the last one

6:35 just requires learning which combination

6:37 of subcomponents corresponds to which digits.

6:41 Of course, that just kicks the problem down the road,

6:43 because how would you recognize these subcomponents,

6:45 or even learn what the right subcomponents should be?

6:48 And I still haven't even talked about how one layer influences the next,

6:51 but run with me on this one for a moment.

6:53 Recognizing a loop can also break down into subproblems.

6:57 One reasonable way to do this would be to first

6:59 recognize the various little edges that make it up.

7:03 Similarly, a long line, like the kind you might see in the digits 1 or 4 or 7,

7:08 is really just a long edge,

7:10 or maybe you think of it as a certain pattern of several smaller edges.

7:15 So maybe our hope is that each neuron in the second

7:18 layer of the network corresponds with the various relevant little edges.

7:23 Maybe when an image like this one comes in, it lights up

7:26 all of the neurons associated with around 8 to 10 specific little edges,

7:31 which in turn lights up the neurons associated

7:33 with the upper loop and a long vertical line,

7:36 and those light up the neuron associated with a 9.

7:40 Whether or not this is what our final network actually does is another question,

7:44 one that I'll come back to once we see how to train the network,

7:47 but this is a hope that we might have,

7:50 a sort of goal with the layered structure like this.

7:53 Moreover, you can imagine how being able to detect edges and patterns

7:56 like this would be really useful for other image recognition tasks.

8:00 And even beyond image recognition,

8:02 there are all sorts of intelligent things you might

8:04 want to do that break down into layers of abstraction.

8:08 Parsing speech, for example,

8:09 involves taking raw audio and picking out distinct sounds,

8:12 which combine to make certain syllables, which combine to form words,

8:16 which combine to make up phrases and more abstract thoughts, etc.

8:21 But getting back to how any of this actually works,

8:23 picture yourself right now designing how exactly the activations

8:26 in one layer might determine the activations in the next.

8:30 The goal is to have some mechanism

8:33 that could conceivably combine pixels into edges,

8:35 or edges into patterns, or patterns into digits.

8:39 And to zoom in on one very specific example,

8:42 let's say the hope is for one particular neuron in the second layer to pick

8:46 up on whether or not the image has an edge in this region here.

8:51 The question at hand is what parameters should the network have?

8:55 What dials and knobs should you be able to tweak

8:58 so that it's expressive enough to potentially capture this pattern,

9:01 or any other pixel pattern,

9:03 or the pattern that several edges can make a loop, and other such things?

9:08 Well, what we'll do is assign a weight to each one

9:11 of the connections between our neuron and the neurons from the first layer.

9:16 These weights are just numbers.

9:18 Then take all of those activations from the first

9:21 layer and compute their weighted sum according to these weights.

9:27 I find it helpful to think of these weights

9:29 as being organized into a little grid of their own,

9:32 and I'm going to use green pixels to indicate positive weights,

9:35 and red pixels to indicate negative weights,

9:37 where the brightness of that pixel is

9:39 some loose depiction of the weight's value.

9:42 Now if we made the weights associated with almost all of the pixels

9:46 zero except for some positive weights in this region that we care about,

9:50 then taking the weighted sum of all

9:52 the pixel values really just amounts to adding

9:54 up the values of the pixel just in the region that we care about.

9:59 And if you really wanted to pick up on whether there's an edge here,

10:02 what you might do is have some

10:04 negative weights associated with the surrounding pixels.

10:07 Then the sum is largest when those middle

10:09 pixels are bright but the surrounding pixels are darker.

10:14 When you compute a weighted sum like this, you might come out with any number,

10:18 but for this network what we want is

10:20 for activations to be some value between 0 and 1.

10:24 So a common thing to do is to pump this weighted sum into some

10:27 function that squishes the real number line into the range between 0 and 1.

10:32 And a common function that does this is called the sigmoid function,

10:35 also known as a logistic curve.

10:38 Basically very negative inputs end up close to 0,

10:41 positive inputs end up close to 1,

10:43 and it just steadily increases around the input 0.

10:49 So the activation of the neuron here is basically

10:52 a measure of how positive the relevant weighted sum is.

10:57 But maybe it's not that you want the neuron

10:59 to light up when the weighted sum is bigger than 0.

11:02 Maybe you only want it to be active when the sum is bigger than say 10.

11:06 That is, you want some bias for it to be inactive.

11:11 What we'll do then is just add in some other number like negative

11:14 10 to this weighted sum before

11:16 plugging it through the sigmoid squishification function.

11:20 That additional number is called the bias.

11:23 So the weights tell you what pixel pattern

11:25 this neuron in the second layer is picking up

11:28 on, and the bias tells you how high the weighted

11:31 sum needs to be before the neuron starts getting meaningfully active.

11:36 And that is just one neuron.

11:38 Every other neuron in this layer is going to be

11:41 connected to all 784 pixel neurons from the first layer,

11:45 and each one of those 784 connections has its own weight associated with it.

11:51 Also, each one has some bias, some other number that you add on to the weighted

11:55 sum before squishing it with the sigmoid.

11:58 And that's a lot to think about!

11:59 With this hidden layer of 16 neurons,

12:02 that's a total of 784 times 16 weights, along with 16 biases.

12:08 And all of that is just the connections from the first layer to the second.

12:12 The connections between the other layers also have

12:14 a bunch of weights and biases associated with them.

12:18 All said and done, this network has

12:21 almost exactly 13,000 total weights and biases.

12:23 13,000 knobs and dials that can be tweaked

12:26 and turned to make this network behave in different ways.

12:31 So when we talk about learning,

12:32 what that's referring to is getting the computer to find a valid setting for all

12:37 of these many many numbers so that it'll actually solve the problem at hand.

12:42 One thought experiment that is at once fun and kind of horrifying is

12:46 to imagine sitting down and setting all of these weights and biases by hand,

12:50 purposefully tweaking the numbers so that the second layer picks up on edges,

12:54 the third layer picks up on patterns, etc.

12:56 I personally find this satisfying rather than just

12:59 treating the network as a total black box,

13:01 because when the network doesn't perform the way you anticipate,

13:04 if you've built up a little bit of a relationship

13:07 with what those weights and biases actually mean,

13:09 you have a starting place for experimenting

13:11 with how to change the structure to improve.

13:14 Or when the network does work but not for the reasons you might expect,

13:18 digging into what the weights and biases are doing is a good way

13:21 to challenge your assumptions and really

13:23 expose the full space of possible solutions.

13:26 By the way, the actual function here is

13:28 a little cumbersome to write down, don't you think?

13:32 So let me show you a more

13:34 notationally compact way that these connections are represented.

13:37 This is how you'd see it if you choose to read up more about neural networks.

13:40 Organize all of the activations from one layer into a column as a vector.

13:48 Then organize all of the weights as a matrix,

13:49 where each row of that matrix corresponds to the connections

13:51 between one layer and a particular neuron in the next layer.

13:58 What that means is that taking the weighted sum

14:01 of the activations in the first layer according to these weights

14:04 corresponds to one of the terms in the matrix

14:07 vector product of everything we have on the left here.

14:14 By the way, so much of machine learning just

14:16 comes down to having a good grasp of linear algebra,

14:18 so for any of you who want a nice

14:20 visual understanding for matrices and what matrix vector multiplication means,

14:24 take a look at the series I did on linear algebra, especially chapter 3.

14:29 Back to our expression, instead of talking about adding the bias

14:32 to each one of these values independently,

14:34 we represent it by organizing all those biases into a vector,

14:38 and adding the entire vector to the previous matrix vector product.

14:43 Then as a final step, I'll wrap a sigmoid around the outside here,

14:46 and what that's supposed to represent is that you're going to apply

14:50 the sigmoid function to each specific component of the resulting vector inside.

14:55 So once you write down this weight

14:58 matrix and these vectors as their own symbols,

15:00 you can communicate the full transition of activations from one layer

15:03 to the next in an extremely tight and neat little expression,

15:07 and this makes the relevant code both a lot simpler and a lot faster,

15:11 since many libraries optimize the heck out of matrix multiplication.

15:17 Remember how earlier I said these neurons are simply things that hold numbers?

15:22 Well of course the specific numbers that they hold depends on the image you

15:26 feed in, so it's actually more accurate to think of each neuron as a function,

15:31 one that takes in the outputs of all the neurons

15:34 in the previous layer and spits out a number between 0 and 1.

15:39 Really the entire network is just a function,

15:41 one that takes in 784 numbers as an input and spits out 10 numbers as an output.

15:47 It's an absurdly complicated function,

15:49 one that involves 13,000 parameters in the forms

15:52 of these weights and biases that pick up on certain patterns,

15:55 and which involves iterating many matrix

15:57 vector products and the sigmoid squishification function,

16:00 but it's just a function nonetheless.

16:03 And in a way it's kind of reassuring that it looks complicated.

16:07 I mean if it were any simpler, what hope would we have that it

16:10 could take on the challenge of recognizing digits?

16:13 And how does it take on that challenge?

16:15 How does this network learn the appropriate

16:17 weights and biases just by looking at data?

16:20 Well that's what I'll show in the next video,

16:21 and I'll also dig a little more into what

16:23 this particular network we're seeing is really doing.

16:27 Now is the point I suppose I should say subscribe to stay

16:30 notified about when that video or any new videos come out,

16:33 but realistically most of you don't

16:35 actually receive notifications from YouTube, do you?

16:38 Maybe more honestly I should say subscribe so

16:40 that the neural networks that underlie YouTube's recommendation algorithm

16:43 are primed to believe that you want to see

16:45 content from this channel get recommended to you.

16:48 Anyway, stay posted for more.

16:50 Thank you very much to everyone supporting these videos on Patreon.

16:54 I've been a little slow to progress in the probability series this summer,

16:57 but I'm jumping back into it after this project,

16:59 so patrons you can look out for updates there.

17:03 To close things off here I have with me Lisha Li

17:05 who did her PhD work on the theoretical side of deep

17:08 learning and who currently works at a venture capital firm called

17:11 Amplify Partners who kindly provided some of the funding for this video.

17:15 So Lisha one thing I think we should quickly bring up is this sigmoid function.

17:19 As I understand it early networks use this to squish

17:22 the relevant weighted sum into that interval between zero and one,

17:25 you know kind of motivated by this biological

17:27 analogy of neurons either being inactive or active.

17:30 Exactly.

17:30 But relatively few modern networks actually use sigmoid anymore.

17:34 Yeah.

17:34 It's kind of old school right?

17:35 Yeah or rather ReLU seems to be much easier to train.

17:39 And ReLU, ReLU stands for rectified linear unit?

17:42 Yes it's this kind of function where you're just taking a max of zero

17:47 and a where a is given by what you were explaining in the video

17:51 and what this was sort of motivated from I think was a partially

17:55 by a biological analogy with how neurons would either be activated or not.

18:01 And so if it passes a certain threshold it

18:03 would be the identity function but if it did

18:06 not then it would just not be activated so

18:08 it'd be zero so it's kind of a simplification.

18:11 Using sigmoids didn't help training or it was

18:14 very difficult to train at some point and people

18:17 just tried ReLU and it happened to work

18:21 very well for these incredibly deep neural networks.

18:25 All right thank you Lisha.