Computer Vision: Crash Course Computer Science #35

0:03 Hi, I’m Carrie Anne, and welcome to Crash Course Computer Science!

0:05 Today, let’s start by thinking about how important vision can be.

0:09 Most people rely on it to prepare food, walk around obstacles,

0:12 read street signs, watch videos like this, and do hundreds of other tasks.

0:16 Vision is the highest bandwidth sense,

0:17 and it provides a firehose of information about the state

0:20 of the world and how to act on it.

0:22 For this reason, computer scientists have been trying

0:24 to give computers vision for half a century,

0:27 birthing the sub-field of computer vision.

0:29 Its goal is to give computers the ability

0:31 to extract high-level understanding from digital images and videos.

0:35 As everyone with a digital camera or smartphone knows,

0:37 computers are already really good at capturing photos with incredible

0:40 fidelity and detail– much better than humans in fact.

0:43 But as computer vision professor Fei-Fei Li recently said,

0:46 “Just like to hear is the not the same as to listen.

0:49 To take pictures is not the same as to see.” INTRO As a refresher,

1:02 images on computers are most often stored as big grids of pixels.

1:05 Each pixel is defined by a color,

1:07 stored as a combination of three additive primary colors: red, green and blue.

1:11 By combining different intensities of these three colors,

1:13 what’s called a RGB value, we can represent any color.

1:17 Perhaps the simplest computer vision algorithm– and a good

1:20 place to start– is to track a colored object, like a bright pink ball.

1:23 The first thing we need to do is record the ball’s color.

1:26 For that, we’ll take the RGB value of the centermost pixel.

1:30 With that value saved, we can give a computer program an image,

1:32 and ask it to find the pixel with the closest color match.

1:35 An algorithm like this might start in the upper right corner,

1:38 and check each pixel, one at time,

1:40 calculating the difference from our target color.

1:43 Now, having looked at every pixel,

1:44 the best match is very likely a pixel from our ball.

1:47 We’re not limited to running this algorithm on a single photo;

1:50 we can do it for every frame in a video,

1:52 allowing us to track the ball over time.

1:54 Of course, due to variations in lighting, shadows, and other effects,

1:57 the ball on the field is almost certainly not going

1:59 to be the exact same RGB value as our target color,

2:03 but merely the closest match.

2:04 In more extreme cases, like at a game at night, the tracking might be poor.

2:08 And if one of the team's jerseys used the same color as the ball,

2:11 our algorithm would get totally confused.

2:13 For these reasons, color marker tracking and similar algorithms are rarely used,

2:18 unless the environment can be tightly controlled.

2:20 This color tracking example was able to search pixel-by-pixel,

2:23 because colors are stored inside of single pixels.

2:27 But this approach doesn’t work for features larger than a single pixel,

2:30 like edges of objects, which are inherently made up of many pixels.

2:34 To identify these types of features in images,

2:36 computer vision algorithms have to consider

2:38 small regions of pixels, called patches.

2:40 As an example, let’s talk about an algorithm

2:43 that finds vertical edges in a scene,

2:45 let’s say to help a drone navigate safely through a field of obstacles.

2:48 To keep things simple, we’re going to convert our image into grayscale,

2:51 although most algorithms can handle color.

2:53 Now let’s zoom into one of these poles to see what an edge looks like up close.

2:57 We can easily see where the left edge of the pole starts,

2:59 because there’s a change in color that persists across many pixels vertically.

3:04 We can define this behavior more formally by creating a rule

3:07 that says the likelihood of a pixel being a vertical

3:09 edge is the magnitude of the difference in color between

3:12 some pixels to its left and some pixels to its right.

3:15 The bigger the color difference between these two sets of pixels,

3:17 the more likely the pixel is on an edge.

3:20 If the color difference is small, it’s probably not an edge at all.

3:23 The mathematical notation for this operation looks

3:25 like this– it’s called a kernel or filter.

3:28 It contains the values for a pixel-wise multiplication,

3:30 the sum of which is saved into the center pixel.

3:33 Let’s see how this works for our example pixel.

3:36 I’ve gone ahead and labeled all of the pixels with their grayscale values.

3:39 Now, we take our kernel, and center it over our pixel of interest.

3:43 This specifies what each pixel value underneath should be multiplied by.

3:46 Then, we just add up all those numbers.

3:49 In this example, that gives us 147.

3:51 That becomes our new pixel value.

3:54 This operation, of applying a kernel to a patch of pixels,

3:57 is call a convolution.

3:58 Now let’s apply our kernel to another pixel.

4:00 In this case, the result is 1.

4:02 Just 1.

4:03 In other words, it’s a very small color difference, and not an edge.

4:06 If we apply our kernel to every pixel in the photo,

4:09 the result looks like this, where the highest

4:11 pixel values are where there are strong vertical edges.

4:13 Note that horizontal edges,

4:15 like those platforms in the background, are almost invisible.

4:18 If we wanted to highlight those features,

4:20 we’d have to use a different kernel– one that’s sensitive to horizontal edges.

4:23 Both of these edge enhancing kernels are called Prewitt Operators,

4:27 named after their inventor.

4:29 These are just two examples of a huge variety of kernels,

4:32 able to perform many different image transformations.

4:34 For example, here’s a kernel that sharpens images.

4:37 And here’s a kernel that blurs them.

4:39 Kernels can also be used like little

4:41 image cookie cutters that match only certain shapes.

4:43 So, our edge kernels looked for image patches with strong

4:46 differences from right to left or up and down.

4:49 But we could also make kernels that are good at finding lines,

4:52 with edges on both sides.

4:53 And even islands of pixels surrounded by contrasting colors.

4:57 These types of kernels can begin to characterize simple shapes.

5:00 For example, on faces,

5:01 the bridge of the nose tends to be brighter than the sides of the nose,

5:05 resulting in higher values for line-sensitive kernels.

5:08 Eyes are also distinctive– a dark circle sounded

5:10 by lighter pixels– a pattern other kernels are sensitive to.

5:14 When a computer scans through an image,

5:16 most often by sliding around a search window,

5:18 it can look for combinations of features indicative of a human face.

5:22 Although each kernel is a weak face detector by itself,

5:25 combined, they can be quite accurate.

5:26 It’s unlikely that a bunch of face-like features

5:29 will cluster together if they’re not a face.

5:31 This was the basis of an early

5:33 and influential algorithm called Viola-Jones Face Detection.

5:35 Today, the hot new algorithms on the block are Convolutional Neural Networks.

5:40 We talked about neural nets last episode, if you need a primer.

5:42 In short, an artificial neuron– which is the building

5:45 block of a neural network– takes a series of inputs,

5:48 and multiplies each by a specified weight,

5:50 and then sums those values all together.

5:53 This should sound vaguely familiar, because it’s a lot like a convolution.

5:56 In fact, if we pass a neuron 2D pixel data,

5:59 rather than a one-dimensional list of inputs, it’s exactly like a convolution.

6:03 The input weights are equivalent to kernel values,

6:06 but unlike a predefined kernel,

6:08 neural networks can learn their own useful kernels

6:10 that are able to recognize interesting features in images.

6:13 Convolutional Neural Networks use banks of these neurons to process image data,

6:17 each outputting a new image, essentially digested by different learned kernels.

6:21 These outputs are then processed by subsequent layers of neurons,

6:25 allowing for convolutions on convolutions on convolutions.

6:28 The very first convolutional layer might find things like edges,

6:31 as that’s what a single convolution can recognize, as we’ve already discussed.

6:35 The next layer might have neurons that convolve

6:37 on those edge features to recognize simple shapes,

6:39 comprised of edges, like corners.

6:42 A layer beyond that might convolve on those corner features,

6:45 and contain neurons that can recognize simple objects, like mouths and eyebrows.

6:49 And this keeps going, building up in complexity,

6:51 until there’s a layer that does a convolution that puts it together:

6:55 eyes, ears, mouth, nose, the whole nine yards,

6:57 and says “ah ha, it’s a face!” Convolutional

7:00 neural networks aren’t required to be many layers deep,

7:03 but they usually are, in order to recognize complex objects and scenes.

7:07 That’s why the technique is considered deep learning.

7:09 Both Viola-Jones and Convolutional Neural Networks can be

7:12 applied to many image recognition problems, beyond faces,

7:15 like recognizing handwritten text,

7:16 spotting tumors in CT scans and monitoring traffic flow on roads.

7:20 But we’re going to stick with faces.

7:22 Regardless of what algorithm was used, once we’ve isolated a face in a photo,

7:26 we can apply more specialized computer

7:28 vision algorithms to pinpoint facial landmarks,

7:30 like the tip of the nose and corners of the mouth.

7:33 This data can be used for determining things like if the eyes are open,

7:37 which is pretty easy once you have

7:39 the landmarks– it’s just the distance between points.

7:41 We can also track the position of the eyebrows;

7:43 their relative position to the eyes can be an indicator of surprise, or delight.

7:47 Smiles are also pretty straightforward to detect

7:50 based on the shape of mouth landmarks.

7:52 All of this information can be interpreted by emotion recognition algorithms,

7:56 giving computers the ability to infer when you’re happy,

8:00 sad, frustrated, confused and so on.

8:02 In turn, that could allow computers to intelligently adapt their behavior...

8:06 maybe offer tips when you’re confused,

8:08 and not ask to install updates when you’re frustrated.

8:11 This is just one example of how vision

8:13 can give computers the ability to be context sensitive,

8:16 that is, aware of their surroundings.

8:18 And not just the physical surroundings– like if

8:19 you're at work or on a train– but also

8:21 your social surroundings– like if you’re in a formal

8:24 business meeting versus a friend’s birthday party.

8:27 You behave differently in those surroundings,

8:29 and so should computing devices, if they’re smart.

8:32 Facial landmarks also capture the geometry of your face,

8:34 like the distance between your eyes and the height of your forehead.

8:38 This is one form of biometric data,

8:40 and it allows computers with cameras to recognize you.

8:43 Whether it’s your smartphone automatically unlocking itself when it sees you,

8:46 or governments tracking people using CCTV cameras,

8:49 the applications of face recognition seem limitless.

8:52 There have also been recent breakthroughs

8:54 in landmark tracking for hands and whole bodies,

8:56 giving computers the ability to interpret a user’s body language,

8:59 and what hand gestures they’re frantically

9:01 waving at their internet connected microwave.

9:03 As we’ve talked about many times in this series,

9:05 abstraction is the key to building complex systems,

9:08 and the same is true in computer vision.

9:10 At the hardware level, you have engineers building better and better cameras,

9:14 giving computers improved sight with each passing year,

9:16 which I can’t say for myself.

9:18 Using that camera data, you have computer vision algorithms crunching pixels

9:22 to find things like faces and hands.

9:25 And then, using output from those algorithms,

9:27 you have even more specialized algorithms for interpreting

9:29 things like user facial expression and hand gestures.

9:32 On top of that, there are people building novel interactive experiences,

9:36 like smart TVs and intelligent tutoring systems,

9:38 that respond to hand gestures and emotion.

9:41 Each of these levels are active areas of research,

9:44 with breakthroughs happening every year.

9:46 And that’s just the tip of the iceberg.

9:47 Today, computer vision is everywhere– whether

9:50 it’s barcodes being scanned at stores, self-driving cars waiting at red lights,

9:54 or snapchat filters superimposing mustaches.

9:56 And, the most exciting thing is

9:58 that computer scientists are really just getting started,

10:01 enabled by recent advances in computing, like super fast GPUs.

10:05 Computers with human-like ability to see is going

10:08 to totally change how we interact with them.

10:10 Of course, it’d also be nice if they could hear and speak,

10:13 which we’ll discuss next week.

10:15 I’ll see you then.