The Wire Problem Nobody In AI Is Talking About

0:00 Nvidia just committed $4 billion to two photonics

0:03 manufacturers that most people have never heard of.

0:07 And when I read the press release, I obviously went straight to Jensen's exact

0:11 words because I really wanted to understand

0:14 what he was actually saying about this, what he actually thought about this.

0:18 Now, he called it building the next generation of gigawatt scale AI factories.

0:23 Gigawatt scale AI factories built on light.

0:27 Now, if you know what that phrase means,

0:29 you already know why this is one of the most

0:31 significant infrastructure bets Nvidia [music] has ever made.

0:36 And if you do not know what it means yet,

0:38 I mean, well, that is exactly what this video is about.

0:42 I went into this video thinking the story was going to be about chips.

0:45 I mean, that is where every headline points.

0:47 Transistors are shrinking more slowly.

0:50 The 3-nanometer wall is real.

0:52 And I did cover the whole Dennard

0:53 scaling breakdown in my Nvidia Blackwell video,

0:56 which is linked [music] down in my description if you haven't seen it already.

0:59 It's a really interesting I mean, I think so.

1:02 I made the video.

1:03 But it's very interesting and really worth considering watching.

1:06 But the deeper I got into the engineering literature,

1:08 the more I kept hitting a different word.

1:10 And it wasn't chip.

1:11 It wasn't transistor.

1:13 It was interconnect.

1:14 [music] The connection between chips kept coming up as the actual

1:18 binding constraint in the next generation of AI clusters.

1:21 [music] And when I finally understood why, it reframed the entire story.

1:27 [music] So, here's what is happening inside a modern AI training cluster.

1:30 First off, you have tens of thousands of GPUs working together.

1:34 Each one is doing an extraordinary amount of computation on its own.

1:39 But training a frontier model means those GPUs are in constant conversation.

1:44 I mean, passing gradient updates, synchronizing parameters,

1:48 moving activations across the network billions of times per second.

1:53 Now, engineers call this collective communication.

1:56 It is the invisible architecture underneath every

1:59 large language model you have ever used,

2:01 which [music] is kind of wild to think about.

2:03 But it brings up a question.

2:04 What carries all those signals between the GPUs?

2:08 Copper traces.

2:09 Electrical signals moving through the [music] metal.

2:12 And copper, at the speed and density

2:15 that the next generation AI clusters require,

2:18 runs into physics that it just can't, [music] frankly, negotiate with.

2:22 Let's talk about the first problem.

2:23 The first problem is resistance.

2:25 Every electrical signal moving through copper converts

2:27 some [music] of its energy into heat.

2:29 I mean, at a small scale, it's manageable.

2:32 At the scale of a 100,000 GPU training clusters, [music]

2:36 that heat becomes a dominant engineering variable.

2:39 The thing that every hardware decision [music] downstream has to account for.

2:43 Now, the second problem is signal loss.

2:45 You see, so electrical signals lose

2:47 fidelity over distance and at high frequencies.

2:50 So, engineers face a real trade-off.

2:53 Faster or farther, but not both without adding regeneration

2:58 hardware that brings its own cost and its own heat.

3:02 Now, the third problem,

3:03 the one that really stopped me when I was reading these numbers,

3:07 is [music] energy per bit.

3:09 Moving a single bit of data electrically has an energy cost.

3:14 Small, but real.

3:16 When you're moving petabytes of data per second inside a training cluster,

3:19 the energy cost of the wires alone starts to show up,

3:23 frankly, meaningfully on the operating budget.

3:26 Now, the analogy I kept on coming back to was this.

3:29 Moving electrons through copper is kind of like pushing water through a straw.

3:34 You can widen it, you can push harder,

3:37 but friction [music] is a physical property of the medium itself,

3:41 and no engineering decision removes friction from a [music] physical medium.

3:45 Now, photons, what this video is all about, they work differently.

3:49 Particles of light carry no electrical charge.

3:52 They generate no resistive heat as [music] they travel.

3:55 They also do not degrade over distance the way electrical signals do.

3:59 And through a technique that's called wavelength division multiplexing, [music]

4:03 you can send dozens of completely independent data

4:05 streams through a single optical channel all at once.

4:08 Each one riding a different color

4:10 of light with [music] no interference between them.

4:13 I mean, that is why a single strand of fiber

4:15 optic cable can carry the internet traffic of an entire continent.

4:19 [music] That physics has existed for decades in long-haul telecommunications.

4:24 But what's happening right now is that engineers are building

4:26 that same physics down to the scale of a chip,

4:29 down to the inside of a server rack.

4:32 I mean, frankly, down to the connection

4:34 between two GPUs sitting centimeters apart.

4:37 So, before we go any further, here is exactly where we are.

4:40 First of all, the chips are doing their job.

4:42 The wires, copper wires, they're the constraint.

4:44 And replacing those wires with light is

4:46 a product roadmap with real money behind that.

4:49 Keep that in mind because it changes everything that we are about to cover next.

4:53 A conventional GPU processes data electrically.

4:56 Transistors switch on and off billions of times per second.

5:00 And the data, it moves through

5:02 copper interconnects between compute [music] cores,

5:04 between memory, between GPUs in the cluster.

5:08 And basically, every hop over copper costs latency.

5:11 [music] Every hop generates heat.

5:13 And the more you scale the cluster,

5:15 the more both of those costs stack against each other.

5:18 [music] Now, an optical processing unit,

5:20 engineers are starting to call these OPUs,

5:23 restructures this chain at a fundamental level.

5:27 [music]

5:26 Instead of encoding information as electrical voltage, data is encoded as light.

5:32 [music] It's intensity, its phase, or its wavelength.

5:35 So, basically, instead of copper traces,

5:37 signals travel through silicon waveguides,

5:40 which are microscopic channels etched into a photonic chip

5:44 that guides light the way a fiber optic cable does,

5:47 but at scales measured in micrometers.

5:51 Now, here is the part of the MIT research

5:53 that really made me stop and had to reread the paragraph.

5:56 The core mathematical operation in virtually every neural network,

6:00 the operation running inside every transformer, every diffusion model,

6:04 every I mean, every large language model you have ever used,

6:07 is called the multiply-accumulate operation.

6:10 And it's basically matrix multiplication.

6:12 Billions of these per forward [music] pass.

6:14 The intelligence of a model is, I mean, at the hardware level,

6:17 an enormous amount of this one [music] operation repeating.

6:21 Photonic hardware can perform matrix multiplications in the optical domain.

6:24 Basically, [music] light passing through a series of tunable components.

6:28 And it's able to do this, I mean,

6:30 at the speed of light without ever converting to an electrical signal.

6:34 Now, the energy reduction that MIT's research group published is

6:37 over 90% for certain operations compared to electronic [music] equivalents.

6:43 Now, I really want to stay on that for a second here.

6:45 Let's take it up a beat.

6:46 We were really down in the weeds.

6:47 I want to stay on that number for a second because I think

6:49 it [music] is easy to just kind of let it wash over you.

6:53 We spend so much time in this community, I mean, in Tiffin Tech,

6:57 talking about model architecture improvements that deliver 3%,

7:00 5%, I mean, maybe 10% efficiency gains, anyways.

7:04 And those matter.

7:05 I mean, people build careers on them.

7:07 But 90% [music] is a different category on its own.

7:11 That is a different physical substrate doing the same work entirely.

7:16 I do also, though, want to be careful with those because

7:18 those figures come from lab conditions and specific workloads.

7:22 [music] Real-world numbers will land a lower,

7:25 of course, as these systems mature.

7:27 But even at 50% or 60%, consider what that means at a moment when

7:32 AI energy consumption is becoming a grid-level policy concern,

7:37 as we just heard recently, anyways, for national governments.

7:40 The economics of inference change in a way that compounds every deployment.

7:45 Okay.

7:46 Back to where the market is moving, though.

7:48 Now, let's talk a little bit about Lightmatter.

7:49 It's one of my favorite companies.

7:51 I'm sure you've heard me speak about it a lot.

7:52 [music] Now, Lightmatter was actually founded out of MIT,

7:55 and it has built a photonic inference accelerator, which is called Passage.

7:59 And it replaces copper interconnects inside a server with optical ones.

8:03 They have raised over $400 million

8:05 and moved from prototype to customer deployments.

8:08 I mean, they are from a university lab bench to production infrastructure,

8:13 and it happened in under 7 years, which is pretty incredible.

8:17 [music] There are some other labs

8:18 as well that are working on co-packaged optics.

8:20 Another one is AR Labs.

8:22 It is working on co-packaged optics.

8:23 [music] So, basically,

8:24 integrating optical input and output directly into the chip package itself.

8:29 So, the data leaving a GPU travels optically

8:31 from the moment [music] it exits the die.

8:34 Nvidia, Intel, I mean, and several other major hyperscalers are investors.

8:39 And then, recently, with Nvidia,

8:42 it was literally as I was finishing research on this this topic, actually,

8:45 when they announced something that really just kind of puts a bow,

8:49 or a punctuation mark, if you will, around it all.

8:52 With Jensen committing $4 billion across two photonic companies.

8:56 $2 [music] billion into Lumentum, and $2 billion into Coherent.

9:01 Two optics in laser component manufacturers.

9:05 [music] And Nvidia just made them two

9:06 of the largest infrastructure bets in its portfolio.

9:08 [music] Now, Jensen's exact words were, "Together with Lumentum,

9:12 Nvidia is advancing the world's most sophisticated silicon photonics to build

9:17 [music] the next generation of gigawatt scale AI factories." Now,

9:21 what really makes this investment so credible to me,

9:24 and what I think gets missed most in the coverage of this all,

9:28 is that Nvidia has actually been cautious about

9:30 photonics in specific parts of its own [music] architecture.

9:34 Jensen previously noted that sticking with copper on their rack-scale GB200

9:38 systems actually shaved down 20 kW off of a 120 kW system.

9:44 [music] So, this is a company that weighed photonics carefully

9:46 against copper and chose copper where copper made sense.

9:50 The $4 billion commitment tells you, though,

9:53 where and when copper stops making sense.

9:56 And markets read it the same way.

9:58 Lumentum closed up to nearly 12% on the day it was announced.

10:02 Same with Coherent at 15%.

10:05 All right, let's take another breather here.

10:07 So, that is a science and that is the market

10:09 signal landing on top of it in the same week.

10:11 It's kind of huge.

10:13 It's It's wild.

10:14 All right, but let me tell you why I think the combination

10:16 of those two things matters more than either one of them alone.

10:19 Because this is really where the real

10:21 scale of what is happening becomes so clear.

10:25 We tend to think about technology improvements as [music] incremental.

10:28 A chip gets faster 20%.

10:30 Bandwidth then in turn doubles.

10:33 Important, I mean, it's expected.

10:35 We hear about it.

10:35 We move on.

10:36 Photonics though is a phase [music] change.

10:39 And the reason is that the problems it solves are

10:41 compounding constraints that get worse together with every hardware generation.

10:46 Here's exactly what I mean though.

10:49 [music] Every generation of AI models has been larger than the last.

10:52 The compute [music] required to train and run them scales with parameter count,

10:56 data volume, and cluster size.

10:59 [music] Under electrical architectures, so does the heat and energy cost.

11:03 And these are problems that multiply across every dimension simultaneously.

11:07 Where pressure builds with every generation.

11:10 There is a point on that curve where you simply cannot build the next model.

11:14 The physics of moving [music] electrical signals at that density

11:18 and speed will not cooperate with the ambition

11:22 regardless of how good the software is

11:24 or how intelligent the architecture team really [music] is.

11:28 Photonics though, it moves that point.

11:31 It removes the ceiling that copper imposes on how large

11:34 a cluster can be and how efficiently it can operate.

11:38 And that matters tremendously for where AI capability goes from here.

11:43 Now, the map from the AI systems of say 2025 [music]

11:47 to the AI systems of 2028 may have less to do

11:51 with the transformer architecture improvements and more [music] to do

11:55 with whether photonic interconnects reach production scale in time to support

12:01 [music]

12:00 the training that runs the next generation of models and what it will require.

12:04 Now, this is a hardware story, but it's also a capability story.

12:09 An energy story, I guess, if you will.

12:11 Really a national infrastructure story.

12:13 Both [music] Lumentum and Coherent are now committing to expand

12:16 US manufacturing capacity [music] as part of their Nvidia agreements,

12:20 which really tells you this is being treated as a strategic infrastructure play.

12:25 Not just a supply chain decision.

12:27 And the fact that it touches all of those dimensions at once is exactly

12:31 why the largest compute companies on the planet

12:34 are funding it and taking it seriously.

12:37 And also too, why it deserves more attention than it is currently getting.

12:41 Nvidia just invested 4 billion into [music] this.

12:45 So, let me put that into plain terms, if you will.

12:47 The models you will use in 2028,

12:50 the ones that will feel qualitatively different from what exists today,

12:55 may owe a significant part of their capability to the fact

12:59 that the wires [music] finally got out of the way.

13:02 That is what this transition means at scale.

13:04 Now, let me get specific about what this means for [music] you.

13:08 Every major hardware transition in computing history

13:11 has created a new class of engineers who understood the new substrate before

13:15 [music] most people even knew it existed.

13:18 And those engineers shape what came next.

13:21 For example, when computing went multi-core,

13:23 the people who already understood the concurrent

13:25 [music] programming and memory models became disproportionately valuable.

13:29 I mean, when GPUs became AI infrastructure,

13:31 it was once again the people who understood

13:34 CUDA and kernel optimization who wrote the playbook.

13:38 Everyone else eventually had to learn.

13:41 When cloud replaced on-premise infrastructure,

13:43 it was [music] the distributed systems thinkers

13:45 who helped entire industries make the transition.

13:49 Photonics is the next [music] substrate shift.

13:52 And really the window where understanding it early is

13:55 a genuine differentiator [music] that is open right now.

13:58 But I want to be clear about something before I give you specific advice here.

14:01 I'm saying this because I think this is one of the most [music]

14:05 interesting skill gaps to close right now.

14:07 And I want to be very honest about what closing it actually can look like.

14:12 Because well, first of all, it is more accessible than maybe it sounds.

14:15 The role that becomes increasingly valuable over the next few

14:18 years [music] is what I would call the AI systems architect.

14:21 Someone who holds the full stack hardware and software together.

14:24 [music] Now, don't worry, you don't need to be an expert per se in all of this.

14:28 But someone who understands how light moves through silicon wave guides,

14:31 how that affects the memory [music] bandwidth and latency at the cluster level,

14:35 and how those physical properties cascade [music] into how you

14:39 structure a distributed training job or an inference serving system.

14:44 That is the difference between being carried along by a transition

14:47 [music] and understanding it well enough to shape what comes next.

14:51 All right, as we wrap up this video,

14:53 here is a thought I keep coming back to after

14:55 spending the last few weeks really deep in this research topic.

14:59 Now, every time computing has approached a physical limit,

15:02 the story gets written as a dead end.

15:04 Each one really described as the thing that finally stopped progress,

15:08 no matter what story you're talking about.

15:10 We are in such a monumental, such a pivotal moment right now.

15:13 The engineers working on photonic computing are doing it because light

15:17 is genuinely better physics for what AI infrastructure needs to become.

15:22 No resistive heat, no signal degradation, no charge.

15:25 The fundamental speed limit of the universe, which sounds crazy to say,

15:29 carrying dozens of data streams simultaneously on a single physics path.

15:34 These are structural advantages,

15:36 the kind that compound across every generation of hardware that follows.

15:40 The infrastructure industry is really starting to act like it.

15:42 I mean, MIT research is in production, startups are deploying,

15:46 and recently Nvidia put 4 billion dollars behind it.

15:50 This is what I genuinely find interesting and so exciting about this moment.

15:53 We did not hit a wall, we found a better medium to think in.

15:57 So, here is what I want to hear from you in the comments.

15:59 As photonic interconnects change the data movement architecture of AI clusters,

16:03 do you think it forces a fundamental rethink

16:06 of how we write distributed training systems and inference pipelines?

16:10 Or does the software abstraction layer hold and most

16:13 engineers never really need [music] to care about it?

16:16 I mean, clearly I have a strong view on this.

16:17 I think we need to care about it.

16:20 But drop your comments down below.

16:22 I always read every response that is left on these videos and a lot

16:27 of the videos I structure come from what you suggest for me to do.

16:30 I mean, photonics was brought to my attention

16:32 by some comments in my last [music] video, so thank you for that.

16:35 So, let me know what other topics you're interested in.

16:37 I love these deep dive tech videos that lets me do

16:41 a ton of research and then get to share with you my findings.

16:44 [music] And hopefully help you stay ahead

16:47 in tech and better understand what is happening.

16:50 All right, I'll see you in the next video.