NVIDIA's New AI Turns One Photo Into A World That Never Breaks
Two Minute Papers
0:00 I can’t believe that we are getting this for free.
0:03 What is this?
0:04 Well, hold on to your papers Fellow Scholars because you take just one image,
0:12 and it creates a 3D explorable world out of it.
0:16 Super cool.
0:17 They call it Lyra 2.0.
0:19 It sounds almost too good to be true, and it often is.
0:24 I’ll tell you why.
0:26 I grew up in Budapest,
0:28 and now I live in southern Hungary in a different, beautiful city called Pécs.
0:33 And whenever I visit Budapest, I love to walk around the parts where I grew up,
0:39 it is always an incredible feeling.
0:40 And I’m thinking that if we can even use research technology
0:44 to deliver just a fraction of that feeling, that is fantastic.
0:50 Or, if you need just one image, you can take a Street View image and it
0:55 will create a video game world out of it.
0:57 Drop in a robot and have it train there safely and learn how to be a good robot.
1:03 A different variant of this concept is called Cosmos, it’s a bit different.
1:09 It creates simulation data for training robots and self-driving
1:13 cars.I recently tried a self-driving car in San Francisco,
1:18 and it was incredible… even though only
1:22 part of its training comes from simulated data, that part is crucial.
1:27 This is a testament to how important and useful simulations are.
1:33 They unlocks unexpected solutions for tough problems.
1:37 But, not so fast.
1:39 This isn’t so easy because unfortunately…we have a big problem.
1:45 These worlds break down.
1:47 Also, wait a second, DeepMind also did this earlier, didn’t they?
1:52 Genie 3.
1:53 An image goes in, a game comes out.
1:57 It can even be a drawing, painting, whatever you wish.
2:01 So how is this different?
2:04 Is this the same thing?
2:08 Well, no.
2:09 Okay, let me try to explain.
2:11 A bit more than a year ago an amazing AI appeared that claimed to have
2:17 watched 1,000,000 hours of Minecraft videos and thus
2:21 remade the very coarse Minecraft game for us.
2:24 And the interesting thing was this.
2:27 We look at something, look away, look back and…whoops!
2:31 Yes, you saw it right.
2:34 That just happened.
2:35 When we ask, okay little AI, what was there, it says: “I dunno”.
2:43 It did not have object permanence.
2:45 Nearly every human toddler has object permanence.
2:49 So it had very limited memory, so long-term consistency was really hard.
2:54 But then, Genie 3 took one image
2:58 and generated interactive worlds with multi-minute consistency.
3:02 All this progress in just a bit more than a year.
3:10 That is…insane.
3:12 However, this is still not that practical
3:15 because over a few minutes, it still forgets.
3:18 What I want to see is long term coherence.
3:21 So what is the solution?
3:23 How do we get a world from a photo that doesn’t break down?
3:28 Is that even possible?
3:29 Dear Fellow Scholars, this is Two Minute Papers with Dr.
3:33 Károly Zsolnai-Fehér.
3:34 Well, most of these techniques see 2D pixels on a flat screen.
3:39 No 3D geometry just a bunch of numbers for pixel colors.
3:44 Here, the core generator is a diffusion transformer, kind of like OpenAI’s Sora.
3:51 Not new.
3:52 But wait a second… it still somehow always remembers.
3:57 The worlds never break.
3:59 In this one, looking away and looking back
4:02 will always give you back what you saw earlier.
4:06 That is possible for video games with 3D geometry made by artists.
4:10 But…how is that even possible with this kind of system?
4:15 Well, that is the point…almost exactly like that!
4:19 You see, it is using a per-frame 3D geometry cache.
4:26 Okay, so what does that mean?
4:28 Well, it means that it keeps a small 3D memory of the scene.
4:33 In simpler words, it doesn’t remember the whole world as is,
4:37 it just remembers the scaffolding of the world.
4:41 And then, it is able to recreate the rest consistently.
4:45 So when you look away, and look back,
4:48 it doesn’t make up something new from scratch, no.
4:51 Instead, it thinks, wait, what was there a moment ago?
4:56 Got it!
4:57 Now, it does not store the whole scene as is, it has a depth map,
5:02 something they call a downsampled point cloud, and some camera movement info.
5:07 That is fantastic.
5:09 But it turns out…that’s not quite fantastic enough.
5:14 This depth map is not for the whole global scene.
5:17 Because if you try to fuse everything into one giant 3D world,
5:22 it is done in a way that errors accumulate over time.
5:27 Tiny mistakes start piling up, and then over time,
5:30 it gets more and more corrupted.
5:32 It is kind of like doing a photocopy of something.
5:36 And then, a photocopy of the photocopy, and then…you know how that goes.
5:42 It just gets lower and lower quality over each step.
5:47 Not good.
5:48 Okay, so now, what is the solution?
5:52 Well, instead, it keeps a separate little 3D snapshot for each view.
5:57 Then later, when it comes back, it can ask:
6:02 which earlier views saw this place best?
6:06 And it uses those as memory.
6:08 That is an incredible idea.
6:11 So, does that really work?
6:13 The ablation study reveals the answer.
6:16 This is a good paper so it proposes a bunch of puzzle pieces,
6:21 and it doesn’t just batch them together into one block and say look!
6:26 It works!
6:27 No.
6:27 It tests every single new puzzle piece in isolation and tells us for each one,
6:33 how much they add to the picture.
6:36 Now, if you stored the whole scene globally,
6:40 style consistency would worsen a bit, and camera control, oh my.
6:47 That is a disaster.
6:50 Can we see what that looks like?
6:52 This is a good paper, so the answer is…yes!
6:56 Oh goodness.
6:57 If you do the global scene thing, it starts producing the wrong camera views.
7:03 While the full proposed technique is much closer to what it should show.
7:08 It really shows that these concepts work in practice.
7:12 So this is why they propose remembering the scaffolding of the scene per frame.
7:19 So much better!
7:21 Love it.
7:22 Note that there is so much more in the paper,
7:25 we really just scratched the surface here.
7:27 But, not even this technique is perfect.
7:32 Limitations.
7:32 One.
7:33 Static scenes only.
7:34 No moving stuff.
7:36 Two, it inherits flaws from its training data.
7:39 Namely, if you have a dataset that has photometric inconsistencies,
7:44 it will inherit that.
7:46 What does that mean?
7:48 Well, if you feed it data with different kinds of lighting and exposure,
7:53 it will also appear in its predictions.
7:56 Of course it does, the training data tells it how the world works,
8:01 and it thinks that lighting and exposure can change on a whim.
8:06 Three, the 3D geometry that we get from it
8:10 can contain artifacts and these weird little floaters.
8:15 Hmm..but why?
8:16 The issue is that the generated views
8:18 are not perfectly consistent with each other,
8:20 and when you try to reconstruct 3D from them,
8:25 these small inconsistencies can turn into floaters and noise.
8:30 If you ask me, these are very typical problems for a first,
8:35 or in this case, second version of such a work.
8:38 And it is very typical that all three of these will
8:41 be ironed out just one more paper down the line.
8:45 Remember, this is the First Law of Papers.
8:48 Do not look at where we are,
8:50 look at where we will be two more papers down the line.
8:53 So, finally, we take just one photo,
8:57 and get to create incredible digital worlds that don’t break down.
9:02 We finally have it.
9:04 That is fantastic.
9:06 And we get all of this, model and code for free?
9:11 Yes!
9:11 What a time to be alive!
9:13 A great gift for us Fellow Scholars and tinkerers.
9:17 Thank you so much for this.