Generating 3D Models with Diffusion - Computerphile

0:00 Today I want to talk about specifically um generative

0:03 AI in 3D and how it sort of all started.

0:06 Um so obviously if you keep up with AI

0:09 um there's been large language models for text um 2D

0:13 diffusion for images and video diffusion for videos but there's

0:17 not been that one big sort of model for 3D.

0:22 Um it's been a bit you know bit slower on that end.

0:25 Um, so what I want to talk about is how we can generate objects in 3D.

0:31 Um, and why it's a lot harder to do it in 3D than it is to do it in 2D.

0:38 If we're going to talk about 3D object generation,

0:41 we need to talk about 2D first.

0:43 Um, so 2D um, is all done using something called diffusion,

0:47 which you've probably heard before,

0:48 but for those who haven't, I'm just going to do a quick recap.

0:51 you give it an image prompt.

0:53 So for this I'm going to do frogs on stilts,

0:57 which is what I've been doing for the past few videos.

0:59 It's based on Mike's prompt from his amazing video about 2D diffusion.

1:03 So if you don't understand this, I recommend going and watching that.

1:06 Fogs on stilts.

1:07 I'll have to have to try that later.

1:10 So what you do is you have an image that is complete noise.

1:14 Okay, random values in there.

1:16 And what you do is you give that to a diffusion model.

1:21 with your prompt and that diffusion model goes okay if

1:25 you want an image that looks like frogs on stilts you

1:29 need to take away this amount of noise from that image

1:33 you'll start getting an image that starts looking a little

1:37 bit like a frog on stills so what happens is

1:40 that at time step t equals z you'll have complete noise

1:43 and then you pass that through the diffusion model several

1:46 times and then let's say this is now t equals 10.

1:50 So you've done it 10 times.

1:52 You'll start getting a frog on stilts like that.

1:57 Oh god, I've run out of space.

2:00 Um, but at t equals, let's say 20, you'll have your frog on stilts.

2:05 So here's the thing with 2D diffusion models, which is just so cool,

2:09 is the fact that they can merge these different concepts.

2:12 So you have a frog and you have stilts.

2:15 Never the twain shall meet.

2:17 [laughter] There's no photo that exists in real life.

2:20 Well, maybe there is.

2:21 Someone's probably done some abstract [laughter] some abstract stuff with that.

2:25 But it's just those two concepts couldn't be further apart,

2:29 but they're able to join together.

2:31 And the reason for that is that these 2D um image um diffusion

2:35 models um are trained on billions and billions of image caption uh data.

2:43 So these training sets are absolutely massive and that's how

2:47 they're able to learn how to draw a frog very well,

2:49 how to draw stilts very well and and put them together.

2:53 3D is more tricky.

2:56 So in 3D data sets, I think the largest one has around a couple of million.

3:01 If you want to generate a frog on stilts,

3:03 what you'll find is that there aren't that many frogs in the data

3:06 set and there not that there aren't that many stilts in the data set.

3:09 So when you ask it to generate a frog on stilts,

3:12 it can't can't really do it that well.

3:14 Um so that's the first problem is that the data sets are smaller.

3:17 The second problem um is that 3D is just so much

3:22 harder than 2D because you've got to think about every single angle.

3:26 So it has to look good from every angle and it has to look realistic which

3:31 a lot of this um 3D data just it doesn't look as real as real life.

3:37 Anyway, that's the problem with 3D data.

3:40 How do we solve it?

3:41 [clears throat] So, just to preface,

3:43 um these days there are some 3D um generation models that just work on 3D data.

3:49 Mesh GPT is a good one if you want to um have a look at that.

3:53 But in general, it's still early days for that sort of thing.

3:56 But what has happened in 2022 um there

4:00 was a model called Dream Fusion that came out.

4:03 Um, and this model was very very it was groundbreaking because it

4:07 was the first time that 2D diffusion was used to generate 3D models.

4:12 So, as I say, 2D diffusion models are very,

4:15 very good at making abstract concepts like frogs on stills.

4:18 So, we want to use that to generate 3D content.

4:22 So, Dream Fusion was the first model that came out that did that and it

4:26 was revolutionary because what they did is they

4:28 leveraged 2D diffusion and projected it into 3D.

4:33 So we're able to get these abstract concepts

4:35 like a frog on stills in three dimensions.

4:38 And this came out like three years ago.

4:40 So you know and that wasn't long after 2D diffusion came out.

4:43 So it was quite a big big thing.

4:46 So we've talked about we have 2D diffusion models.

4:49 So that um I [snorts] stuff like stable

4:51 diffusion is used for that because it's quick,

4:53 it's nice, uh generates good images.

4:55 Um for the 3D representation we don't use a mesh.

4:59 Um we use a view symphysis model.

5:02 Now a view Simpsons model is something

5:04 like no radiance fields which we've spoken about.

5:07 Oh nerfs nerfs which we've spoken about on this channel and now the latest

5:10 sort of craze 3D gaus and splatting which is my favorite thing ever.

5:15 But what these models can do is you give it a series of 2D

5:18 images and they're able to reconstruct the scene um in 3D via some optimization.

5:24 So 2D to 3D.

5:26 Are you starting to sense a pattern here?

5:27 So we have diffusion that can generate 2D images of what we want.

5:32 And then we have Gaussian spatting on Nerf

5:34 that can generate a 3D model based on 2D.

5:38 Right?

5:38 So we have these two things.

5:41 Let's put them together.

5:42 And how they put them together

5:43 is using something called score distillation sampling.

5:46 You have your prompts.

5:48 I'm going to do frog on stilts.

5:51 And what you do is you start off with a blank 3D scene.

5:55 For Gaussian spatting, we have to initialize it with some Gaussians.

5:58 Um, and for but for nerf, we don't need to.

6:01 Actually, to preface, I'm just going to talk about Gaussian spatting uh

6:04 for this one because um that's what people use more now.

6:07 But Dream Fusion used Nerf.

6:10 So, you have a blank scene here.

6:13 So, what you then do is you render a 2D image from a perspective.

6:18 So, I'm going to draw a camera there for example.

6:22 So that camera has rendered this image here.

6:25 Okay.

6:26 So what we then do is we need to add some noise to it

6:33 as we've done with um 2D diffusion when you added noise to that.

6:38 So this image is slightly noised up.

6:40 Then what you do is you pass

6:42 this into the diffusion model with your prompt and say

6:49 if I'm going to dn noiseise this image to make it look like frogs on stilts,

6:52 what noise do I need?

6:54 And the diffusion model goes you need this amount of noise.

6:57 You then take it away from that image.

6:59 And what happens is that that image will start to look like a frog.

7:04 And then what you do is you use that image and optimize

7:08 that 3D scene as if you were using um 3D reconstruction basically.

7:12 And then what you find is that this scene starts to look a bit more like a frog.

7:17 What happens in real life is that you do this many many many times.

7:20 But for now I've drawn let's let's just do a really noisy frog.

7:23 There you go.

7:24 That explains it.

7:24 So you've done it once and now that there's a there's a frog that when you

7:28 render from this camera angle here it starts to look a bit more like a frog.

7:33 But what you do now is you take a different camera angle.

7:36 You have your camera here.

7:38 And then what you do is you render a new image from here.

7:41 And now you start having this frog, very noisy frog in this image.

7:47 You add some noise to it.

7:49 Ask the diffusion model,

7:51 what would this frog look like if I'm going to make it look better?

7:55 It then generates what that frog look like,

7:58 what should look like from that angle.

7:59 And then you optimize it.

8:00 And suddenly, let's say, I don't know,

8:03 this frog now has eyes from that camera angle.

8:06 And so what you do is you do that again with another camera angle,

8:10 again with another camera angle, again and again and again.

8:13 And that's why it's called score distillation sampling

8:15 because you're distilling that sort of update over many,

8:19 many camera angles, and do that many, many, many times.

8:21 And what happens is that you start getting a 3D model of a of a frog on stills.

8:27 So each of that that noisy frog is being made out of gaussians is it?

8:31 Yes.

8:31 Yes.

8:32 So this is all Gaussians or if you're using Nerf it

8:35 will it will just be um a neural network but essentially

8:39 splats of color or whatever

8:40 flats of color that make up a frog which is it's quite nice isn't it?

8:44 It's quite cool.

8:45 Um but yes essentially you start off with nothing or if

8:48 it's gases back you need to initialize gases but I

8:50 won't get into that but you start off with something

8:52 that looks very bad and then what you do is you

8:54 use this to you render an image ask the 2D diffusion

8:57 model what should this actually look like it generates an image

9:00 of what it thinks it should look like and then

9:02 you do that you update it to look more like that.

9:05 But here's the thing.

9:06 If you just did it from one camera angle many, many many many times,

9:10 what you do is you get a frog that looks great from that camera angle,

9:13 but as soon as you move over, it would look completely just terrible.

9:18 So, what you do is you do it over many camera angles and do it so

9:23 that it updates it all together and starts

9:25 to look like um your frog that you want.

9:29 There are some issues that they found

9:31 that we're just going to talk about very briefly.

9:33 So there's this thing that they found which I think is just awesome.

9:36 It's called the multiface Janus problem.

9:39 Now Janus is the Roman god with two faces.

9:43 Um which I think is just the perfect analogy for this.

9:46 So what happens is that let's say you have your blank scene and you render

9:51 an image blah blah blah and then you ask a diffusion model to generate a frog.

9:56 You ask it to generate a frog on stilts.

9:58 what it will draw and I'm really gonna try and do this but I'm

10:02 probably not gonna it will try and draw it will draw a frog

10:05 looking at the camera because that's what it's trained on is trained on frogs

10:09 where the frog is looking at the camera that's what it just assumes it's

10:12 going to be but if you generate uh you do the same

10:16 on a camera angle like this blah blah do the same do the same what

10:21 it will generate it will say yes that's what it should look like there

10:26 and what happens is that you end up with a frog with two heads.

10:32 [laughter] Um cuz it every single image

10:37 it presumes is the dominant front-facing image.

10:42 Um it doesn't understand that this image is

10:45 over here and this image is over here.

10:47 Um so and and that's how you get something called content drift

10:51 as well where from certain angles things will merge and look really weird.

10:55 I I'll show you all examples of this um in a second.

10:58 But so how did they solve this issue?

11:01 Well, what they did is that let's they said that right in this 3D scene,

11:06 this angle here is going to be the predominant angle.

11:10 So when you then um generate an image,

11:13 you pass in the prompt that is converted into a clip embedding and then

11:17 passed in of a a frog on stilts from the front front-facing camera.

11:23 But then if they had one here which is um from the back

11:26 they would pass in the prompt um a frog on silts from the back.

11:31 So it would learn to generate it from the front and the back.

11:34 But if you want to generate one from here which is

11:36 in between what they did is that they had the clip embedding

11:40 of a frog on stilts from the front a clip embedding

11:43 of a frog on stilts from the back and then they'd interpolate between it.

11:48 Let's say this is zero, this is 90 and this is 60.

11:50 They do it so it was let's say 60% frog on stilts from the back

11:55 and 30% or 40% frog on silts from the front if that makes sense.

11:59 So the diffusion model would learn okay it's roughly this sort of what we need.

12:04 So that solved or helped solve the multiface Janice problem

12:09 because I was thinking of that very old pitch on Monday version of that.

12:12 It is it is yeah that's spot on.

12:14 [laughter] Yeah, I'll show you some examples cuz it did

12:17 fail and it has generated like some some weird stuff.

12:20 One other thing they realized as well is that if they just use

12:22 the same amount of noise throughout let's say the first epoch to the last epoch,

12:27 it wouldn't converge very well.

12:29 But what they realized that if at the start they have a lot of noise,

12:33 then what happens is that the diffusion model is able to generate

12:36 the weird sort of structures that it that it needs for a frog.

12:40 So it's able to generate eyes and generate legs and stuff like that.

12:44 Um but as you lower the noise as the number of um cameras that you

12:48 generate goes on, it's able to then get that fine texture if that makes sense.

12:52 So at the start it generates like really weird stuff,

12:55 but it's just generating the overall structure and then

12:58 towards the end it's finalizing it and only

13:00 adding a little bit of noise so it's able to converge on a nice um frog.

13:05 So it's generated a frog.

13:08 Um this is using um Gaussian dreamer which

13:11 is a score distillation sampling model that uses Gaussian.

13:14 So if I click play you can see that it's a frog on stilts.

13:18 I wouldn't say that's super impressive but it is a frog on stilts.

13:22 If I go now I can show you what it looked like when it was generating.

13:28 This is a training process.

13:29 So when it starts off it doesn't really look like a frog on stilts that much.

13:33 But as you go on, we're on 1,000 now.

13:38 And finally 2,000, you get your final finished frog.

13:42 So that looks okay.

13:44 Doesn't look too bad.

13:45 But now I'm going to show you something that doesn't work so much.

13:49 And this is quite funny, actually.

13:51 Hopefully people will laugh at it.

13:52 Here we have a frog that doesn't quite look very, very good.

13:56 Um, this is a frog on frog's legs rather than a frog on stilts.

14:02 And also it's grown some sort of it's grown antenna.

14:05 Is that the word antenna?

14:07 So now what I'm going to show you

14:08 is what it looks like during the training process.

14:11 So this is at um epoch zero.

14:14 And then you have on the left is what it looks like when it's rendered.

14:17 On the right is what the diffusion model is saying it should look like.

14:20 So at the start you can see

14:22 that it generates some absolutely crazy images of frogs.

14:27 And that's because the noise is quite high.

14:29 But as you go on, you can see firstly that the frog

14:32 is starting to look a bit more like a frog.

14:35 But there are some problems with this.

14:38 Here we go.

14:39 So from this angle, when it generates the frog,

14:43 firstly it's a frog with two heads.

14:46 And secondly, what should be the stilts there at the bottom,

14:50 it thinks is a reflection,

14:52 which is why it's generated these frog extra legs because it

14:57 thinks that that should be a reflection rather than the stilts.

15:01 And I can go over more and more examples here.

15:04 And also, you've got like this this frog has like a hat.

15:08 And also [snorts] this the reason why I

15:11 believe it has those weird antenna at the back

15:16 with the with the sort of um fake eyes on it

15:19 is because when it generates an image from the back,

15:22 it assumes that that's the frog's eyes from the back.

15:25 So the multifface Janice problem not quite been solved here.

15:30 [laughter] But as you can see, even though there are problems,

15:33 it's still able to generate um something.

15:36 So that was 3 years ago.

15:38 Um what has there been improvements?

15:41 Yes, there have.

15:42 There are models now called um multiv- view latent diffusion

15:45 models which can solve a lot of this uh these problems.

15:49 Um which maybe is another video in itself.

16:00 You're kind of talking about how to make random images effectively.

16:03 How do we direct this?

16:04 So that's where the complexity starts ramping up, right?

16:07 We've got a structure where we can train a network to produce random images,

16:10 but it's not guided.

16:11 There's no way of saying I want a frog rabbit hybrid, right?

16:15 which I've done and it's very weird.