NVIDIA's New AI Turns One Photo Into A World That Never Breaks

NVIDIA's New AI Turns One Photo Into A World That Never Breaks

Two Minute Papers

0:00 I can’t believe that we are getting this for free.

0:03 What is this?

0:04 Well, hold on to your papers Fellow Scholars because you take just one image,

0:12 and it creates a 3D explorable world out of it.

0:16 Super cool.

0:17 They call it Lyra 2.0.

0:19 It sounds almost too good to be true, and it often is.

0:24 I’ll tell you why.

0:26 I grew up in Budapest,

0:28 and now I live in southern Hungary in a different, beautiful city called Pécs.

0:33 And whenever I visit Budapest, I love to walk around the parts where I grew up,

0:39 it is always an incredible feeling.

0:40 And I’m thinking that if we can even use research technology

0:44 to deliver just a fraction of that feeling, that is fantastic.

0:50 Or, if you need just one image, you can take a Street View image and it

0:55 will create a video game world out of it.

0:57 Drop in a robot and have it train there safely and learn how to be a good robot.

1:03 A different variant of this concept is called Cosmos, it’s a bit different.

1:09 It creates simulation data for training robots and self-driving

1:13 cars.I recently tried a self-driving car in San Francisco,

1:18 and it was incredible… even though only

1:22 part of its training comes from simulated data, that part is crucial.

1:27 This is a testament to how important and useful simulations are.

1:33 They unlocks unexpected solutions for tough problems.

1:37 But, not so fast.

1:39 This isn’t so easy because unfortunately…we have a big problem.

1:45 These worlds break down.

1:47 Also, wait a second, DeepMind also did this earlier, didn’t they?

1:52 Genie 3.

1:53 An image goes in, a game comes out.

1:57 It can even be a drawing, painting, whatever you wish.

2:01 So how is this different?

2:04 Is this the same thing?

2:08 Well, no.

2:09 Okay, let me try to explain.

2:11 A bit more than a year ago an amazing AI appeared that claimed to have

2:17 watched 1,000,000 hours of Minecraft videos and thus

2:21 remade the very coarse Minecraft game for us.

2:24 And the interesting thing was this.

2:27 We look at something, look away, look back and…whoops!

2:31 Yes, you saw it right.

2:34 That just happened.

2:35 When we ask, okay little AI, what was there, it says: “I dunno”.

2:43 It did not have object permanence.

2:45 Nearly every human toddler has object permanence.

2:49 So it had very limited memory, so long-term consistency was really hard.

2:54 But then, Genie 3 took one image

2:58 and generated interactive worlds with multi-minute consistency.

3:02 All this progress in just a bit more than a year.

3:10 That is…insane.

3:12 However, this is still not that practical

3:15 because over a few minutes, it still forgets.

3:18 What I want to see is long term coherence.

3:21 So what is the solution?

3:23 How do we get a world from a photo that doesn’t break down?

3:28 Is that even possible?

3:29 Dear Fellow Scholars, this is Two Minute Papers with Dr.

3:33 Károly Zsolnai-Fehér.

3:34 Well, most of these techniques see 2D pixels on a flat screen.

3:39 No 3D geometry just a bunch of numbers for pixel colors.

3:44 Here, the core generator is a diffusion transformer, kind of like OpenAI’s Sora.

3:51 Not new.

3:52 But wait a second… it still somehow always remembers.

3:57 The worlds never break.

3:59 In this one, looking away and looking back

4:02 will always give you back what you saw earlier.

4:06 That is possible for video games with 3D geometry made by artists.

4:10 But…how is that even possible with this kind of system?

4:15 Well, that is the point…almost exactly like that!

4:19 You see, it is using a per-frame 3D geometry cache.

4:26 Okay, so what does that mean?

4:28 Well, it means that it keeps a small 3D memory of the scene.

4:33 In simpler words, it doesn’t remember the whole world as is,

4:37 it just remembers the scaffolding of the world.

4:41 And then, it is able to recreate the rest consistently.

4:45 So when you look away, and look back,

4:48 it doesn’t make up something new from scratch, no.

4:51 Instead, it thinks, wait, what was there a moment ago?

4:56 Got it!

4:57 Now, it does not store the whole scene as is, it has a depth map,

5:02 something they call a downsampled point cloud, and some camera movement info.

5:07 That is fantastic.

5:09 But it turns out…that’s not quite fantastic enough.

5:14 This depth map is not for the whole global scene.

5:17 Because if you try to fuse everything into one giant 3D world,

5:22 it is done in a way that errors accumulate over time.

5:27 Tiny mistakes start piling up, and then over time,

5:30 it gets more and more corrupted.

5:32 It is kind of like doing a photocopy of something.

5:36 And then, a photocopy of the photocopy, and then…you know how that goes.

5:42 It just gets lower and lower quality over each step.

5:47 Not good.

5:48 Okay, so now, what is the solution?

5:52 Well, instead, it keeps a separate little 3D snapshot for each view.

5:57 Then later, when it comes back, it can ask:

6:02 which earlier views saw this place best?

6:06 And it uses those as memory.

6:08 That is an incredible idea.

6:11 So, does that really work?

6:13 The ablation study reveals the answer.

6:16 This is a good paper so it proposes a bunch of puzzle pieces,

6:21 and it doesn’t just batch them together into one block and say look!

6:26 It works!

6:27 No.

6:27 It tests every single new puzzle piece in isolation and tells us for each one,

6:33 how much they add to the picture.

6:36 Now, if you stored the whole scene globally,

6:40 style consistency would worsen a bit, and camera control, oh my.

6:47 That is a disaster.

6:50 Can we see what that looks like?

6:52 This is a good paper, so the answer is…yes!

6:56 Oh goodness.

6:57 If you do the global scene thing, it starts producing the wrong camera views.

7:03 While the full proposed technique is much closer to what it should show.

7:08 It really shows that these concepts work in practice.

7:12 So this is why they propose remembering the scaffolding of the scene per frame.

7:19 So much better!

7:21 Love it.

7:22 Note that there is so much more in the paper,

7:25 we really just scratched the surface here.

7:27 But, not even this technique is perfect.

7:32 Limitations.

7:32 One.

7:33 Static scenes only.

7:34 No moving stuff.

7:36 Two, it inherits flaws from its training data.

7:39 Namely, if you have a dataset that has photometric inconsistencies,

7:44 it will inherit that.

7:46 What does that mean?

7:48 Well, if you feed it data with different kinds of lighting and exposure,

7:53 it will also appear in its predictions.

7:56 Of course it does, the training data tells it how the world works,

8:01 and it thinks that lighting and exposure can change on a whim.

8:06 Three, the 3D geometry that we get from it

8:10 can contain artifacts and these weird little floaters.

8:15 Hmm..but why?

8:16 The issue is that the generated views

8:18 are not perfectly consistent with each other,

8:20 and when you try to reconstruct 3D from them,

8:25 these small inconsistencies can turn into floaters and noise.

8:30 If you ask me, these are very typical problems for a first,

8:35 or in this case, second version of such a work.

8:38 And it is very typical that all three of these will

8:41 be ironed out just one more paper down the line.

8:45 Remember, this is the First Law of Papers.

8:48 Do not look at where we are,

8:50 look at where we will be two more papers down the line.

8:53 So, finally, we take just one photo,

8:57 and get to create incredible digital worlds that don’t break down.

9:02 We finally have it.

9:04 That is fantastic.

9:06 And we get all of this, model and code for free?

9:11 Yes!

9:11 What a time to be alive!

9:13 A great gift for us Fellow Scholars and tinkerers.

9:17 Thank you so much for this.

Study with Looplines Download Captions Watch on YouTube