The Infrastructure of Intelligence: NVIDIA and OpenAI on a Decade of Building Frontier AI

0:01 Ian, you've been up here so much today.

0:03 They should have just put you a chair over in the corner.

0:05 Nice and quiet.

0:06 Nice comfy chair over there.

0:08 So one of the things we wanted to talk about

0:10 today is the concept of building AI infrastructure at scale.

0:14 But what scale means in 26 is different than what

0:18 it meant two years ago, or even a year ago.

0:21 What does it take to build compute at scale today?

0:23 And do you want to start us off?

0:26 Yeah.

0:27 I mean, I think if you today no AI model except for maybe

0:31 the smallest ones fit on a single GPU or even a single server,

0:35 we've now gone from, uh, GPU scale,

0:37 NoSQL to rack scale to row scale to really being a data center infrastructure.

0:43 You have to design a compute token factory from end to end.

0:48 And there are huge efficiencies when you

0:51 do that in terms of coordinating the power,

0:54 scheduling the KB cache, operating and serving literally.

0:58 Um, well, Sachin is going to probably can speak more to the to the thousands

1:04 or tens of thousands or hundreds of thousands

1:06 of queries coming in to all the compute infrastructure.

1:08 It's not just a GPU problem,

1:10 it's the CPU problems with storage problems and networking problems.

1:14 So I think that has been the biggest pivot in the last, say, five years,

1:20 but certainly in the last two,

1:22 as we gone upscale to sort of gigawatt scale AI factors and which is exciting.

1:28 I think from a business standpoint, we have to help the world,

1:33 you know, build a supply chain that can build,

1:37 all this infrastructure and also understand how they can

1:40 run it efficiently to maximize cost or minimize cost,

1:44 maximize the throughput, and get the most out of the infrastructure.

1:47 And stuff's changing so fast.

1:49 It's a it is actually not a, you know, one time strategy.

1:54 It's a continuous sort of Silicon Valley technology innovation, collaboration.

1:58 And I mean, to a large extent, what's being built at this point.

2:03 The scale is unprecedented.

2:04 So are you just making it up as you go along?

2:07 How do you how do you de-risk doing something

2:10 that no one's really done before in this way?

2:13 I mean, I think it's, Really partnerships like the one we have with NVIDIA.

2:18 So, like Ian said, I think the scale continues to grow.

2:23 On the training side, scaling laws have not stopped.

2:26 The scaling laws continue to hold.

2:29 And so what that means is we will still see those jumps every year.

2:34 Every generation on the amount of compute will be

2:37 doing growing at training these next generations of models.

2:40 So as that scale increases, it becomes a whole system optimization problem.

2:44 It's not just chips, uh, GPUs or CPUs, networking memory.

2:48 It's also the power distribution system, the cooling system,

2:52 the whole warm shell with the chips inside.

2:55 It needs to be treated as one unit and that's where it's super important to.

2:59 Work with the whole ecosystem.

3:00 We can't do it alone.

3:01 So we rely on partners like NVIDIA to actually build build that.

3:05 I think as we look at this, I think very consistently we see returns to compute,

3:11 like we can see the leaps in intelligence.

3:15 We are seeing that with Astra coming out.

3:18 It is the first one trained on Abilene on black GPUs at scale.

3:23 that is obviously showing phenomenal results.

3:25 And I don't think that's stopping anytime soon.

3:28 So one example of building compute

3:30 for the future is the massive Ohio Data Center.

3:34 That is a project between OpenAI and NVIDIA and SB energy.

3:37 Um, can you tell me, apart from the fact that it's huge,

3:41 what's what's novel about it, besides the scale.

3:44 When you start on a project that's at literally gigawatt scale,

3:48 you can start from a blank sheet of paper and see where we can optimize.

3:53 So obviously we want to be able to leverage the ecosystem

3:57 and the capabilities we have today and take it to the next level.

4:01 But when you get a chance

4:03 to actually design things like the power infrastructure,

4:05 uh, as I talked about a little bit this morning,

4:08 you know, the there are a wide variety of workloads and each

4:12 have a different level of individual scale and also power within the rack.

4:17 You know, if you're maximizing user token rate,

4:19 you may be more communication and latency

4:22 sensitive than in just driving up flops,

4:25 where if you're doing offline or batch processing or large pre-filled jobs,

4:30 uh, it's more, uh, flop rich and powerful, power hungry.

4:34 That dynamic in this inside of the data

4:37 center actually is an opportunity to optimize.

4:39 You can schedule work on different nodes,

4:42 on different rows inside the data center

4:44 to distribute the compute and the power consumption.

4:47 Um, but you got to do that in a way that it can be,

4:51 uh, you can take advantage of it.

4:53 So if you're a job scheduler,

4:55 knows what the where the jobs are, how much their power they're consuming.

4:59 You can then build a power infrastructure

5:01 that can dynamically load balance the delivery.

5:04 By doing that, you know you're going to only you're going to consume

5:08 your full allocated power instead of just

5:10 sort of provisioning for the whole thing,

5:12 and maybe only use 60 or 70% of it most of the time.

5:16 Um, to do that, you have to have a smart GPU,

5:20 which can be understand its own power and be able to manage it.

5:23 You need to have a rack that actually is able to handle

5:27 transients and EDP spikes and other things to maintain to keep the, uh,

5:32 the, the electrical characteristics consistent and modeled and safe.

5:36 And then also smart, uh,

5:38 intelligent power controllers down the road, down the racks.

5:41 Uh, and all the PDAs that goes back to some,

5:45 uh, math schedulers be able to keep all that operating.

5:48 If you do that, um, you can get upwards

5:51 of 30 or 40% more GPUs in the data center,

5:53 but you got to design that from the beginning.

5:56 So that's one of the exciting things about Portsmouth.

5:59 And designing these next generation data centers is that you can think about

6:04 the problem end to end and design

6:07 in collaboration with with OpenAI in their workflows,

6:10 but also put the right things into the compute trays,

6:14 the PDU specs, even the interconnects

6:17 and the protocols for talking to the the end

6:21 of row and even up to the transformer to manage reliability and manage power.

6:26 Um, I've become a little bit of a power engineer

6:29 in my next click of my career and learning about what does it

6:32 take to actually go from being voltage to low voltage

6:36 to and kV and dealing understanding what EDP and power spikes and transients.

6:41 Uh, capacitors are a good thing.

6:43 They can help normalize power and actually keep

6:46 the whole thing flowing and the compute, you know,

6:50 on an instant without creating a havoc for a lot of things, like a power grid.

6:54 And when you when you look at all of these innovations

6:57 and things that are different in that data center project,

6:59 what do you how do you think it changes the way

7:02 the whole industry will build data centers in the future?

7:04 What do you hope people might be able to take

7:07 away from what what you're doing with your partners?

7:09 Yeah, I mean.

7:10 When it's fully built out, it will be the largest data center ever built.

7:14 Obviously.

7:15 Who knows?

7:15 If no one passes it yet.

7:18 But I think.

7:19 There's so many things there that at eight gigawatts fully built out,

7:23 it's a big chunk of the grid in Ohio itself.

7:26 And so as we think about the future, how do we co-design the workload,

7:31 the training job, for example, how do we design?

7:33 What do we need on the compute side,

7:36 what kind of reliability we need for our training job?

7:39 What kind of networking reliability do we need?

7:41 And then how does that flow through

7:43 to the cooling system design and the power system design.

7:46 All of that is one continuous loop.

7:48 The second thing I'd say is,

7:50 if designed right, this can help stabilize the grid.

7:53 In Ohio, whenever there's spikes, we can go down on our jobs.

7:58 Whenever there's gaps,

7:59 we can actually go up so that we can keep the grid smooth.

8:03 And that kind of stuff can actually help reduce electricity bills.

8:07 That sounds somewhat counterintuitive, given all the concerns about the impact,

8:10 the negative impacts on the grids right now.

8:13 Are you saying that you're doing things

8:15 that are different or that people should?

8:17 I think that is.

8:18 An opportunity to do things that are different

8:20 when we co-design like this with the whole ecosystem.

8:23 Right.

8:23 So today everything's designed in silos.

8:25 The benefit of this greenfield project is from the ground up.

8:29 We can design all of the different layers appropriately,

8:32 such that this can be a very good citizen

8:35 on the grid and actually help the grid modernize,

8:39 stabilize and reduce electricity bills by doing

8:42 adaptive load sharing with the whole grid.

8:45 So such and a lot of people spend their weekends relaxing.

8:50 But it sounds like your team spent the weekend trying out Astra on a Vera Rubin.

8:55 Can you tell us a little bit more about that and how it's going so far?

8:59 Yeah, this was uh, funnily so Ian and I had a chat last week,

9:04 and I was just telling him about how we are about to launch

9:08 a service which is going to use our AI to help infrastructure designers design,

9:14 optimize and operate that infrastructure, including, for example,

9:18 inference of any model on any chip.

9:21 And and Leon's challenge to me was, oh, great, show it to me on Rubin, right.

9:27 We have a precious Rubin rack in our lab.

9:30 Uh, it's, uh, but it's, uh, we'd be super excited to ramp,

9:34 uh, Rubin as quickly as we can.

9:36 And so we took that challenge.

9:37 So the team, uh, took Astra, uh, actually a checkpoint of Astra,

9:42 uh, and first took a Blackwell implementation of Astra inference.

9:46 Astra running on black will check and just

9:49 run it on Rubin with very minor modification.

9:52 It's a testament to NVIDIA's hardware that out

9:55 of the box we got a 3x improvement in throughput.

9:58 We didn't have to change anything.

10:00 We literally took the same code that we had optimized for Blackwell.

10:03 Run it on Rubin got 3x improvement.

10:06 Then we are like, can we actually make

10:09 Astra run agents using Astra as the base model?

10:13 Go optimize this inference off of Astra itself on Ruben.

10:17 It's a recursive loop.

10:18 The model is figuring out how to optimize its own inference,

10:22 serving on Rubin chips and over 72 hours.

10:25 The model produced another 2x improvement on the throughput, right?

10:29 So things that would take weeks, if not months to optimize.

10:34 In the past, we now are getting models that are

10:37 very capable of actually producing this over a weekend.

10:40 You just are throwing more compute at it, more black holes at it effectively,

10:45 but I think the larger takeaway point is good hardware plus good tooling, right?

10:50 The reason this happened so quickly over 72 hours is good.

10:55 Programable fact compilers.

10:56 All of those building blocks that DNN team built allow the model

11:00 to quickly explore how best to optimize inference on these chips,

11:05 and they were able to produce a two X benefit over,

11:08 uh, what was already possible because of the hardware improvements itself.

11:12 I mean, our relationship goes back all the way to, um,

11:17 from the first, GPU supercomputers, all the way back to the Ampere generation.

11:22 And, um, it's one thing about opening I they're not shy about giving feedback.

11:28 And so by building, uh, taking that feedback,

11:32 learning and understanding how we can make better tooling, better AI models,

11:36 better architecture to deliver the best possible performance for inference,

11:40 for training, for pre-training,

11:42 for post and for and not to mention just operating it at scale.

11:46 Um, all of that becomes a body of work that, um,

11:50 engineers know how to think about these kinds of machines,

11:53 thinking about them at scale, but also agents or the HCI and the AGI of ASI,

12:00 of the of the interface now becomes something that we can both be optimizing.

12:06 So I'm curious now, what was the piece of feedback from OpenAI

12:10 that you remember most that affected the NVIDIA roadmap in a certain way?

12:14 Well, it was a big, um,

12:16 risk to go from the hopper generation, which was one box.

12:20 We put eight GPUs in one node, and that was kind of the unit of compute was

12:25 a single server to realizing that we're going to trillion parameter models.

12:29 Let me say we talked about it like it was nothing.

12:32 But back then it was like, oh my God, a trillion parameters.

12:34 Um, the only way to get there was to actually disaggregate,

12:37 to take that hopper box with GPUs and to a few NVLink

12:41 switches and now spread out across the entire rack to build 72.

12:45 It was a, um, uh, existential moment to actually do that.

12:49 All of the engineering, all the rest.

12:51 And that, you know, that future of of going

12:55 to these frontier models, OpenAI and others.

12:58 Uh, but but having OpenAI right there with us to, you know,

13:02 the first time we tried to optimizing it to building it and then building it,

13:06 working at 100,000 beyond GPU scale is a, uh, a journey.

13:10 That is that was I think now, at this point,

13:13 Astro is a testament of all of that work that we've done.

13:17 And while your team was working on Astra, on Vera Rubin,

13:20 the whole rest of the AI space was talking about safety and alignment pauses.

13:25 Given what Sam has said about pausing,

13:28 does that impact these aggressive compute build

13:31 out plans that that you've been talking about?

13:34 If anything, I think it actually, uh,

13:37 double underscores the need for more compute.

13:40 The way we will achieve safety and alignment for these very capable

13:45 models is by actually spending and more compute on building the safety models,

13:51 the alignment models that will make sure

13:54 that the frontier models are actually safe and aligned.

13:58 So I think the the perhaps counterintuitive

14:01 conclusion from that is we'll actually be spending

14:04 more compute because not we will obviously

14:06 continue to train and infer these frontier models.

14:09 But in addition, we will be investing in more compute

14:12 to make sure that these models are safe and aligned.

14:15 So it's perhaps a counterintuitive observation, but if anything,

14:18 we believe that we will need more compute to make

14:20 sure that future models are more safe and more aligned.

14:25 Again, um, OpenAI has talked quite a bit about how

14:29 it's used its models and designing its own jalapeno chip.

14:32 Can you talk a little bit about how NVIDIA uses AI models in chip design,

14:37 and is it NVIDIA's own open source models?

14:39 Is it third party?

14:41 We use them all.

14:42 We absolutely use them all.

14:44 Um, some of the open models, our own Nemo, Tron model.

14:48 Absolutely.

14:48 GPT and and others.

14:50 Um, our engineers have access to all of it.

14:53 Uh, and then we're build,

14:54 build our own tools and our own tooling and our own flows.

14:58 A couple of, um, uh,

14:59 I will say our bug databases is dates back to when I was an intern at Nvidia.

15:04 So it is a wealth of information

15:06 that teaches us about our own products, that actually,

15:09 it's wonderful having an agent actually ask you what's actually

15:12 going on with this bug and be able to summarize it.

15:14 And all the engineers in the status.

15:17 Um, we use it.

15:18 So for understanding issues in our hardware,

15:20 we use it for optimizing the silicon yields.

15:23 Um, we actually even use it for code check ins.

15:27 Um, you can't just check your own.

15:29 No one can just check in code to the Cuda driver.

15:31 You have to have it properly reviewed by, uh,

15:34 domain expert or by a peer to make sure there's no bugs.

15:37 It's best practice, but now we have an agenda

15:40 tool that they can actually pre submit to the tool.

15:42 And it looks for common issues or known bugs,

15:45 because we actually have individual agents that are aware

15:49 and knowledgeable about different aspects of our software stack.

15:52 And the engineers really appreciate that because they can they

15:56 can get it checked before it goes to the the expert.

15:59 What's also interesting is that it's a psychological experiment.

16:01 When we first turned it on, we had like 1 to 2 our agent test,

16:05 and all the developers hated it because it was like slowing the down like,

16:09 oh God, I want to just finally die.

16:12 I wanted to review it.

16:13 Um, it was actually, uh, GPT one amazing thing about GPT is

16:17 that it really has really great token efficiency.

16:20 And so we were able we realized we had to bring down this pre-check test.

16:25 And by using GPT, we can do it with fewer tokens.

16:29 It happens faster.

16:30 And for those of you who are maybe doing this for yourself,

16:34 15 minutes seems to be the good.

16:36 Developers are good with a 15 minute tool on a pre-check.

16:39 If you go any longer than that, they start

16:42 to complain just regardless of how useful it is.

16:45 So thank you.

16:46 Um, Ian, if we could go as we're going through your career,

16:49 we just covered off on intern a little bit later.

16:51 Uh, you created Cuda.

16:52 Um, it's now, you know, about two decades on, uh,

16:57 as software systems and chip systems become more and more complex,

17:00 what kinds of new software do we need in networking and orchestration?

17:03 Or perhaps another area to handle that?

17:06 I think, um, as such, and mentioned,

17:08 Astra is an excellent kernel optimization tool.

17:11 I think we the it AI is another tool in our toolbox,

17:15 along with all the other profilers.

17:17 None of these other tools are going to be replaced.

17:20 They're all just complementary,

17:22 and they're all be taking advantage of of AI and computing in various ways,

17:26 just like the cloud, uh, really accelerated the productivity of those tools, uh,

17:30 in terms of what the platform looks like and what how how to engage developers.

17:36 Um, they now have agents to to pull from.

17:39 The agents can be using the tools directly or the, um,

17:43 the libraries themselves can exist.

17:45 Um, every single domain speaks in a different language.

17:49 Sometimes they're even the same algorithms.

17:51 But if you're developing software for, um,

17:54 EDA, if you're a silicon vendor or, um,

17:57 the fluid simulation or an airplane wing or the operational,

18:02 um, logistics of, uh, shipping or delivering pizzas,

18:05 they all have speak in a language of libraries and algorithms and concepts

18:11 that are known both to the users and the agents and the AI.

18:17 Uh, we have over 8000 applications that are been, uh,

18:22 GPU accelerated, major ISV applications that we help track and curate, um,

18:27 the capability and making those developers productive in the end is what's going

18:31 to connect them to the value of what they're bringing to their users.

18:35 So while I don't, um, individually manage through it anymore, uh,

18:39 one of the joys of my job is to help those industries adopt AI,

18:45 adopt computing, and figure out how to take advantage of all these technologies,

18:49 uh, and computing in general to be more.

18:51 And, and such.

18:52 And obviously, you know, OpenAI is running systems with, again,

18:55 a bunch of different semiconductors.

18:57 How do how do you think about orchestration and software

19:00 needed for systems that are in data centers that are increasingly,

19:04 you know, heterogeneous?

19:05 I think we see heterogeneity everywhere, uh,

19:08 was talking about at the start or even for training,

19:11 as well as for products and inference.

19:15 We'll have obviously GPUs, but also CPUs, networking, storage,

19:19 all of which needs to be stitched together and orchestrated,

19:23 increasingly user expectation around interactivity.

19:26 Uh, but that that needs to be a very tightly controlled loop.

19:31 I think overall where we see this evolving is,

19:35 uh, heterogeneity is a fact of life.

19:38 We try to get away from not had to deal with it for so long,

19:42 but I don't think we can anymore, or as we think about the future.

19:47 It's we're a small team.

19:49 OpenAI is still a very small team, relatively speaking.

19:52 And so our push is being on how do we use our own models,

19:55 how to use our own AI to build out this orchestration software.

19:59 And just like with kernel optimization,

20:01 Astra is actually very good at even end to end optimization.

20:04 You can give it an end to end problem,

20:06 and it can figure out how to sort through it and actually make the end

20:11 to end system work So we feel increasingly

20:13 we will build with our partners like NVIDIA,

20:15 the right tooling, the right telemetry, expose the right knobs,

20:19 and really let agents go at it to figure it out.

20:23 End to end.

20:24 Great.

20:24 Well, that's all the time we have.

20:26 Sachin and Ian, thank you for joining us.

20:29 Thank you.