How does Ray Tracing Work in Video Games and Movies?

0:00 Every new TV show and movie that uses

0:03 computer-generated images and special effects relies on Ray Tracing.

0:07 For example, in order to build an interstellar battle,

0:11 set in a galaxy far, far away, 3D artists model and texture the spaceships,

0:18 position them around the scene with lights, a background,

0:21 and a camera, and then render the scene.

0:24 Rendering is a computational process that simulates how rays

0:28 of light bounce off of and illuminate each of the models,

0:33 thus transforming a scene full of simple 3D models into a realistic environment.

0:38 There are many different ray tracing algorithms used to render scenes,

0:44 but the current industry standard in TV shows and movies is called path tracing.

0:49 This algorithm requires an unimaginable number of calculations.

0:53 For example, if you had the entire population

0:57 of the world working together and performing 1 calculation every second,

1:02 it would take 12 days of nonstop problem

1:05 solving to turn this scene into this image.

1:09 Due to these incredible computational requirements,

1:12 path tracing was considered impossible

1:15 for anything but super computers for decades.

1:18 In fact, this algorithm for simulating light was first conceptualized in 1986,

1:24 however it took 30 years before movies like Zootopia, Moana,

1:30 Finding Dory and Coco could be rendered using path tracing and even then,

1:36 rendering these movies required a server farm

1:40 of 1000s of computers and multiple months to complete.

1:44 So, why does path tracing require quadrillions of calculations?

1:48 And how does Ray Tracing work?

1:51 Well, in this video, we’ll answer these two questions, and in the process,

1:57 you’ll get a better understanding of how Computer Generated Images

2:01 or CGI and special effects are created for TV and movies.

2:06 After that we’ll open up this GPU and see

2:10 how its architecture is specifically designed to execute ray tracing,

2:14 enabling it to render this scene in only a few minutes.

2:19 And finally, we’ll investigate how Video games like Cyberpunk

2:22 or the Unreal Engine Lumen Renderer use Ray Tracing.

2:27 So, let’s dive right in.

2:36 This video is sponsored by Brilliant.org Let’s

2:40 first see how Path Tracing works and how

2:43 this dragon and kingdom are created and turned

2:46 into a setting for a fantasy show.

2:48 To make the scene, an artist first spends a few months modeling everything,

2:54 the islands, the castles, the houses, the trees, and of course, the dragon.

2:59 Although these models may have some smooth curves or squares and polygons,

3:04 they’re actually all broken down into small triangles.

3:08 In short, GPUs almost exclusively work with 3D scenes made of triangles,

3:15 and this scene is built from 3.2 million triangles.

3:19 After a model is built,

3:22 the 3D artist assigns a texture to it which defines both the color,

3:27 as well as material attributes,

3:29 such as whether the surface is rough, smooth, metallic, glass,

3:33 water-like, or composed of a wide range of other materials.

3:37 Next, the completed models are properly positioned around

3:41 the scene and the artist adds lights such as the sky and the sun and adjusts

3:48 their intensity and direction to simulate the time of day.

3:51 Finally, a virtual camera is added

3:54 and the scene is rendered and brought to life.

3:58 As mentioned earlier, path tracing simulates how light interacts

4:02 with and bounces off every surface in the scene,

4:06 thereby producing realistic effects such as smooth shadows

4:10 across the buildings or the way light interacts

4:13 with the water and produces bright highlights

4:15 in some areas and water covered sand in others.

4:19 In the real world, light rays start at the sun,

4:23 and when they hit a surface such as this red roof,

4:26 some light is absorbed while the red light is reflected,

4:30 thus tinting the light based on the color of the object.

4:34 These now tinted light rays bounce off the surface and make

4:38 their way to the camera and produce a 2D image.

4:42 With this scene, a near infinite number of light rays are produced by the sun

4:47 and sky and only a small fraction of them actually reach the camera.

4:53 Calculating an infinite number of light rays is impossible

4:56 and only the light rays that reach the camera are useful,

5:00 and therefore with path tracing we don’t send

5:03 rays out from the sky or light source,

5:07 but rather we send out rays from a virtual camera and into the scene.

5:12 We then determine which objects the rays hit and calculate

5:15 how those objects are illuminated by the light sources.

5:19 With computer-generated images or CGI,

5:21 the 2D image is represented by a view plane in front of the virtual camera.

5:28 This view plane has the same pixel count as the final image,

5:32 so a 4K image has 8.3 million pixels.

5:36 Furthermore, by animating the camera around or changing its field of view,

5:42 the view plane will correspondingly change.

5:45 Let’s transition to an indoor scene such as this barbershop,

5:50 which contains 8 million triangles and is

5:53 actually more complicated than the island kingdom.

5:56 In order to create this image on the view plane, a total of 8.3 billion rays,

6:02 which are a thousand rays per pixel,

6:05 are sent out from the virtual camera through the view plane and into the scene.

6:10 Ray Tracing is a massively parallel operation because

6:14 each pixel is independent from all other pixels.

6:17 This means that the thousand rays from one pixel can be calculated

6:22 at the same time as the rays from the next pixel over and so on.

6:27 Once a single pixel’s rays finish flying around the scene,

6:30 the results are combined with the other rays and pixels to form a single image.

6:35 If we were to show billions of rays,

6:39 the scene would quickly become inundated with lines,

6:42 so let’s simplify it down to a single

6:45 ray running through one pixel of the viewing plane.

6:48 This ray starts at the camera,

6:50 travels through a random point in the pixel and into the scene.

6:54 It flies straight and eventually hits a triangle, and once it does,

7:00 that object’s color becomes associated with that ray and pixel.

7:04 For example, when the ray hits this chair, then the pixel becomes red.

7:09 The other nearby rays running through random places in the same pixel

7:14 will hit pretty close to this ray and have their colors averaged together.

7:19 These rays are called primary rays and they

7:22 answer the question of what triangle and object

7:24 do the rays first hit and what basic color should be in that specific pixel.

7:30 Another example is that these rays running through this pixel hit

7:34 the blue stripe on the barbershop pole turning the pixel blue.

7:38 The other billions of rays do the same thing resulting

7:42 in a single image with the proper 3D perspective from the virtual camera.

7:47 This image is fairly flat colored because each pixel just

7:51 has the simple color of the object the rays hit.

7:55 So the next question is:

7:57 how is the location where the primary ray hits illuminated by the light

8:02 sources and how bright or dark should the pixel’s color be shaded.

8:06 For example, when you look at the blue stripe of the barbershop pole,

8:12 the entire stripe is just blue, but in the rendered image,

8:16 there’s a gradient from bright to dark across a number of pixels

8:20 depending on how the triangles are facing the lights and the window.

8:25 Specifically, the dark blue backside doesn’t face any of the light sources

8:31 and therefore its illumination comes only

8:33 from light bouncing off the nearby walls.

8:36 Furthermore, when the lighting conditions change

8:39 and more light enters the scene, the entire barbershop pole brightens up.

8:44 This accurate lighting applies to all the objects in the scene

8:48 and is what transforms the scene and makes it look realistic.

8:52 In order to accurately determine the brightness of these blue pixels,

8:56 ray tracing first needs to determine how

8:59 the surface is illuminated directly by the light sources,

9:03 which is called direct illumination, and second,

9:06 how the surface is illuminated by light bouncing off other objects,

9:10 which is called indirect illumination.

9:12 Combining direct and indirect illumination is called global illumination.

9:19 In order to calculate direct illumination,

9:21 we start at the intersection point where the primary ray hits

9:25 the triangle in the barbershop pole and then we generate additional rays

9:30 called shadow rays and send them in the direction of each light

9:34 source such as the light bulbs or the sun outside the window.

9:38 If there are no objects between the intersection point and a light source,

9:42 then that means that this point on the blue

9:45 stripe is directly illuminated by that light source.

9:48 For each light source that directly illuminates this point,

9:51 we factor in the light source’s brightness, size,

9:55 color, distance, and the direction of the surface

9:58 that the triangle inside the blue stripe is facing.

10:02 All these factors are multiplied by the Red, Blue,

10:05 and Green or RGB values of the blue stripe,

10:09 which in turn changes the shading or brightness

10:12 of the pixel that the primary ray went through.

10:15 Let’s brighten the room again,

10:17 and you can see the RGB values increase for this pixel.

10:22 Now let’s dim the room once more and look at a different

10:26 pixel whose primary ray hits the backside of the barbershop pole.

10:30 A similar set of shadow rays are sent

10:32 out from this intersection point to each light source,

10:36 but each of these rays is blocked by other triangles in the pole,

10:41 and thus this point doesn’t receive any

10:43 direct illumination from any of the light sources, leaving the pixel dark.

10:48 These rays are called shadow rays because they determine whether a location

10:53 is directly illuminated by a light source or whether it’s in a shadow.

10:58 You might think that this backside should

11:00 be entirely black because it’s in the shadows

11:03 and none of the light rays from the light sources can reach it.

11:08 However, this backside still has color because

11:10 it’s illuminated by light bouncing off the walls.

11:14 This light is called indirect illumination, and in order to calculate it,

11:19 we take the intersection point from the primary ray

11:22 and generate a secondary ray that bounces off it.

11:25 This secondary ray then hits a new surface such as this point on the wall.

11:30 From this secondary point we send out a new set of shadow rays to each light

11:36 source to see whether the point on the wall

11:40 is in shadows or whether it’s directly illuminated.

11:43 The results from these new shadow rays and the attributes of the corresponding

11:48 light sources are combined with the color of the wall’s surface,

11:51 essentially turning this point on the wall into a light

11:55 source that illuminates the backside of the barbershop pole.

11:58 Sometimes this point is still in shadows,

12:01 so we create an additional secondary ray from the point on the wall

12:06 and send it in a new direction and see what it hits.

12:10 Then we calculate how that third point is

12:13 directly illuminated using yet another set of shadow

12:16 rays thereby turning this third point

12:18 into a light source that illuminates the previous point.

12:23 This secondary ray bouncing happens multiple times,

12:26 and each time we send shadow rays to the light

12:30 sources and check how that point is illuminated.

12:33 The purpose of bouncing the secondary rays around and sending

12:36 out shadow rays at each point is to find

12:40 different paths where light bounces off different surfaces and indirectly

12:44 illuminates the original point where the primary ray hits.

12:49 Furthermore, by sending a thousand rays through random points in a single pixel,

12:54 and by having thousands of secondary rays bounce in different directions,

12:59 we get an accurate approximation for indirect illumination or how

13:04 this pixel is illuminated by light bouncing off the other objects.

13:09 It's called path tracing because by using these primary rays,

13:14 secondary rays and shadow rays,

13:17 we’re finding billions of paths from the camera through

13:20 different points in the scene and to the light sources.

13:24 One additional benefit of indirect illumination and the use of secondary

13:28 rays is that color can bounce from one object to another.

13:33 For example, when we place a red balloon

13:35 next to the wall and brighten the scene,

13:38 some secondary light rays are tinted red by the balloon,

13:42 and this reddish color can be seen on the wall itself.

13:45 An important detail is that the direction the secondary rays bounce off

13:50 the surface depends on the material

13:53 and texture properties assigned to the object.

13:55 For example, here is a set of spheres that are all gray,

14:00 but have different roughness values that drastically change their look.

14:05 Essentially, for a perfectly smooth surface with no roughness,

14:09 the object becomes a mirror because every one of the secondary

14:13 rays will bounce off in the same perfect reflection direction,

14:17 and whatever the secondary rays hit will combine

14:21 together and become visible in the mirror-like surface.

14:24 However, when a material has a roughness set to 100%,

14:29 then the secondary rays will bounce in entirely

14:32 random directions resulting in a flat gray surface.

14:36 Furthermore, if an object is assigned a glass material,

14:40 then additional refraction rays that pass through the glass are generated,

14:46 and the color and brightness of the pixels in the glass will depend

14:49 mostly on the direction of the refraction rays and what those rays hit.

14:54 Here’s an interesting scene of some glass and mirror

14:58 objects that truly show the power of path tracing,

15:01 and you can see multiple mirror bounces in some

15:04 of the objects and proper refraction in the glass.

15:08 Note that for this barbershop scene a thousand rays per pixel

15:12 and four secondary bounces are the render settings we chose during scene setup.

15:18 Other scenes use different numbers of rays per pixel,

15:22 secondary bounces, and light sources.

15:24 When we multiply these values together with the number of pixels in an image

15:29 we get the total number of rays required to generate a single image.

15:35 Furthermore, animations typically have 24 frames a second,

15:40 so a 20-minute animation requires over a quadrillion rays,

15:44 and that’s why path tracing was considered computationally

15:48 impossible for TV shows and movies for decades.

15:52 The other key problem was figuring out which one triangle

15:56 out of 8 million each of the rays hits first.

16:01 So let’s see how these problems are solved and we’ll start

16:04 by transitioning to a new scene

16:06 and see how ray-triangle intersections are calculated.

16:10 Let’s simplify the scene down to one ray

16:13 and two triangles and find which one the ray hits.

16:17 We start by extending the planes that the triangles are on and then,

16:22 using the equations of the planes and the ray,

16:25 we calculate the point at which they intersect.

16:28 Now that we have a set of intersection points on separate planes,

16:32 we find whether the point is inside each corresponding triangle.

16:37 If it is, then that means the ray hits the triangle,

16:41 and if it isn’t that means it misses the triangle.

16:45 These steps are relatively simple, and with 10 triangles,

16:49 we can do this over and over, once for each triangle.

16:54 If multiple triangles are hit we do

16:57 a distance calculation to find the closest one.

17:00 However, when a scene has millions of triangles,

17:03 finding which one triangle a single ray

17:07 hits first becomes incredibly repetitive and computationally problematic.

17:11 We solve this by using what’s called a bounding volume hierarchy or BVH.

17:17 Essentially, we take triangles in the scene and, using their 3D coordinates,

17:24 we divide them into two separate boxes called bounding volumes.

17:28 Each of these boxes contains half of all the triangles in the scene.

17:33 Then we take these 2 boxes with their 1.5 million

17:37 triangles and divide them again into boxes with 750,000 triangles.

17:43 We keep on dividing the triangles into more and more

17:48 progressively smaller pairs of boxes for a total of 19 divides.

17:53 In the end we’ve separated 3 million triangles into a hierarchy of 19 divisions

17:59 of boxes with a total of 525 thousand very small boxes at the bottom,

18:07 each with around 6 triangles inside.

18:10 The key is that all of these boxes

18:13 have their sides aligned with the coordinate axes,

18:16 which makes a far easier calculation.

18:18 For example e, if we have a ray and two axes aligned boxes,

18:23 finding whether it hits box A or box B is just

18:27 a matter of finding the intercept with the plane of Y equals six,

18:31 and then seeing whether the intercept coordinates fall

18:35 between box A’s bounds or between Box B’s bounds.

18:39 Then we do the same thing inside Box B but using

18:44 the axes aligned coordinates of the two smaller boxes inside of it.

18:49 For a scene of 3 million triangles,

18:51 these 19 box divide branches form a binary tree or hierarchy,

18:57 hence the name bounding volume hierarchy.

19:00 At each branch we perform a simple ray-box intersection

19:04 calculation to see which box the ray hits first,

19:08 and then the ray travels to the next branch.

19:12 At the very bottom, once a ray

19:15 finishes traveling through all the bounding volume branches,

19:18 which is called BVH traversal, we end up with a small box of only 6 triangles.

19:25 We then do the ray-triangle intersection calculation

19:28 that we mentioned earlier with just these 6 triangles.

19:33 As a result, BVH trees and traversal

19:36 reduce tens of millions of calculations down to a handful of simple ray box

19:43 intersections followed by 6 ray triangle intersections.

19:47 Using BVHs helps to solve which triangle a ray will hit first but doesn’t

19:53 fix the fact that a single frame

19:55 of animation requires over a hundred billion rays.

19:59 The solution is in the incredibly powerful GPUs we now have.

20:04 When we open up this GPU, we find a rather large microchip that has 10496 CUDA

20:12 or shading cores and 82 Ray Tracing or RT cores.

20:17 The CUDA cores perform basic arithmetic while the ray tracing

20:22 cores are specially designed and optimized to execute Ray Tracing.

20:27 Inside the RT cores are two sections,

20:30 the BVH traversal section takes in all the coordinates of the boxes

20:34 and the direction of the ray and executes BVH traversal in nanoseconds.

20:40 Then, the ray triangle intersection section uses

20:43 the coordinates of the six or so

20:46 triangles in the smallest bounding volume and quickly

20:50 finds which triangle the ray hits first.

20:53 The RT cores operate in parallel with one another and pipeline

20:57 the operations so that a few billion rays can be handled every second,

21:02 and a complex scene like this one can be rendered in 4 minutes.

21:08 Overall Path Tracing’s computationally impossible problems are solved by using

21:13 bounding volume hierarchies along with improvements in GPU hardware.

21:18 One crazy fact is that the most powerful

21:22 supercomputer in the year 2000 was the ASCI White,

21:26 which cost 110 million dollars and could

21:30 perform 12.3 trillion operations a second.

21:34 Compare this with the NVidia 3090 GPU which

21:37 cost a few thousand dollars when it first came out in 2022 and the CUDA

21:44 or shading cores perform 36 trillion operations a second.

21:49 It’s mind-boggling how such an incredible amount

21:51 of computing power can fit into a graphics card the size of a shoebox and how

21:57 computer-generated images or CGI and special effects,

22:00 which used to be only for high-budget films,

22:04 can now be created on a desktop computer.

22:07 Ray Tracing is a fusion of a variety

22:10 of different disciplines from the physics of light,

22:14 to trigonometry, vectors, and matrices,

22:17 and then also computer science, algorithms and hardware.

22:21 Covering all these topics would require multiple hour-long

22:25 videos which we don’t have time to do, but luckily Brilliant,

22:29 the sponsor of this video,

22:31 already has several free and easy to access courses that explore these topics.

22:37 Brilliant is where you learn by doing,

22:40 and is a website filled with thousands of fun and interactive modules,

22:45 loaded with subjects ranging from the fundamentals of math to quantum

22:49 mechanics to programming in python to biology, and much more.

22:53 When I learn new things on Brilliant, I like to think about Steve Jobs,

22:59 and how he took a calligraphy class at college.

23:02 Although at the time it had no practical application in his life,

23:06 10 years later when designing the Macintosh computer,

23:10 he applied all the lessons from that calligraphy course

23:14 to designing the typefaces and proportionally spaced fonts of the Mac.

23:18 The key is that as you progress

23:21 through Brilliant’s interactive lessons and learn new things,

23:24 you may not know how those lessons apply to your job or life,

23:29 but there will be one or two courses that will

23:32 click into place and change the trajectory of your career.

23:35 However, if you don’t try out their courses, then you’ll never know.

23:39 The other reason why Steve Jobs is applicable

23:42 to ray tracing is because he was the CEO

23:46 of Pixar from 1986 until 2006 and helped to design

23:50 the computers that rendered some of its first movies.

23:53 To be a successful inventor like Steve Jobs,

23:56 you need to be well versed in a wide range of disciplines.

24:01 For the viewers of this channel,

24:03 Brilliant is providing a free 30-day trial with access to all

24:07 their thousands of lessons and is also offering 20% off an annual subscription.

24:13 Just go to brilliant.org/brancheducation.

24:16 The link is in the description below.

24:21 We loved making this video because path tracing is

24:24 an algorithm that we use daily due to the fact

24:28 that all our animations are created and rendered using a software

24:31 called Blender which uses path tracing in its rendering engine.

24:36 Specifically, here are all the scenes we used and some

24:40 statistics that you can pause the video and look at.

24:43 It takes a ton of work to create high quality educational videos.

24:49 Researching this video, writing the script,

24:51 and then animating the scenes has taken us over 800 hours,

24:56 so if you could take a quick second to like this video,

25:00 subscribe to the channel, write a comment below and share it with someone who

25:05 watches TV or movies it would help us a ton.

25:08 Furthermore, we’d like to give a shout-out to the Blender Dev Team.

25:13 Blender is an incredibly powerful, free-to-use modeling and animation software.

25:17 Each of these scenes was made by an incredible artist

25:22 and you can download them for free from the Blender website.

25:26 Finally, one question you may have is:

25:29 how is ray tracing is used in video games.

25:33 There are many different methods, so we’ll cover just a few of them.

25:37 The first one is similar to path tracing but with some shortcuts.

25:42 For a given environment in a video game,

25:45 a very low-resolution duplicate of all the models in the scene is created.

25:50 Path tracing is then used to determine direct

25:53 and indirect lighting for each of these low-resolution

25:57 objects and the results are saved

25:59 into a light map on the low-resolution duplicate.

26:02 Then the light map is applied to the high-resolution

26:05 version of the objects in the scene,

26:08 creating realistic indirect lighting and shadows on the high-resolution objects.

26:13 This method is pretty good at approximating indirect lighting and is

26:17 one of the ray tracing techniques used in Unreal Engine’s Lumen renderer.

26:23 The second and completely different method for using ray

26:26 tracing in video games is called screen space ray tracing.

26:30 It doesn’t use the scene’s geometries but rather uses

26:33 the images and data generated from the video game graphics rendering

26:38 pipeline where all the objects in the scene undergo 3D

26:42 transformations to build a flat 2D image on the viewscreen.

26:46 During the video game graphics process, additional data is created,

26:51 such as a depth map that shows how far

26:54 each object and the corresponding pixels are from the camera,

26:58 as well as a normal map that shows

27:01 the direction each of the objects and pixels are facing.

27:04 By combining the view screen, the depth map,

27:07 and the normal map, we can generate an approximation for the X,

27:11 Y, and Z values of the various objects in the scene,

27:16 as well as determine what direction each pixel is facing.

27:20 Now that we have a simplified scene, let’s say this lake is reflective,

27:24 and we want to know what pixels should be shown in its reflection.

27:28 To figure it out, we use ray tracing with this simplified screen space 3D

27:33 representation and bounce the rays off

27:36 of the lake’s pixels using the normal map.

27:39 These rays then continue through the simplified

27:42 geometry and hit the trees behind it,

27:45 thus producing a reflection of the trees on the lake.

27:50 One problematic issue with screen space ray tracing is

27:53 that it can only use the data that’s on the screen.

27:57 As a result, when the camera moves, the trees move out of view,

28:01 and thus the trees are removed from the screen space

28:04 data and it’s impossible to see them in the reflection.

28:08 Additionally, screen space ray tracing doesn’t allow

28:11 for reflections of objects behind the camera.

28:14 This type of ray tracing along with other

28:17 rendering algorithms are used in games like Cyberpunk.

28:21 Additionally, if you’re curious as to how video game graphics work,

28:25 we have a separate video that explores all the steps such as Vertex Shading,

28:31 Rasterization, and Fragment Shading.

28:33 The video game graphics rendering pipeline

28:36 is entirely different from Ray Tracing, so we recommend you check it out.

28:41 And, that’s pretty much it for Ray Tracing.

28:44 We’d like to give a shoutout to Cem Yuksel,

28:47 a professor at the School of Computing at the University of Utah.

28:52 On his YouTube channel,

28:53 you can find his lecture series on computer graphics and interactive graphics,

28:58 which were both instrumental in the research for this video.

29:02 This is Branch Education, and we create 3D animations that dive deeply

29:08 into the technology that drives our modern world.

29:11 Watch another Branch video by clicking one

29:16 of these cards or click here to subscribe.