State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490

0:00 The following is a conversation all

0:02 about the state-of-the-art in artificial intelligence,

0:04 including some of the exciting technical breakthroughs and developments

0:08 in AI that happened over the past year,

0:11 and some of the interesting things we think might happen this upcoming year.

0:16 At times, it does get super technical,

0:19 but we do try to make sure that it remains

0:22 accessible to folks outside the field without ever dumbing it down.

0:26 It is a great honor and pleasure to be able to do

0:30 this kind of episode with two of my favorite people in the AI community,

0:35 Sebastian Raschka and Nathan Lambert.

0:38 They are both widely respected machine learning researchers

0:42 and engineers who also happen to be great communicators,

0:46 educators, writers, and X posters.

0:49 Sebastian is the author of two books

0:53 I highly recommend for beginners and experts alike.

0:56 First is Build a Large Language Model

0:59 from Scratch and Build a Reasoning Model from Scratch.

1:04 I truly believe in the machine learning world,

1:08 the best way to learn and understand

1:11 something is to build it yourself from scratch.

1:15 Nathan is the post-training lead at the Allen Institute for AI,

1:21 author of the definitive book on Reinforcement Learning from Human Feedback.

1:26 Both of them have great X accounts, great Substacks.

1:30 Sebastian has courses on YouTube, Nathan has a podcast.

1:34 And everyone should absolutely follow all of those.

1:37 those.

1:38 This is the Lex Fridman podcast.

1:40 To support it, please check out our sponsors in the description,

1:43 where you can also find links to contact me,

1:47 ask questions, get feedback, and so on.

1:50 And now, dear friends, here's Sebastian Raschka and Nathan Lambert.

1:57 So I think one useful lens to look

1:59 at all this through is the so-called DeepSeek moment.

2:03 This happened about a year ago in January 2025,

2:07 when the open-weight Chinese company DeepSeek released DeepSeek R1,

2:11 that I think it's fair

2:14 to say surprised everyone with near-state-of-the-art performance,

2:17 with allegedly much less compute for much cheaper.

2:22 And from then to today, the AI competition has gotten insane,

2:28 both on the research and product level.

2:31 It's just been accelerating.

2:32 discuss all of this today,

2:34 and maybe let's start with some spicy questions if we can.

2:38 Who's winning at the international level?

2:40 Would you say it's the set of companies in China

2:43 or the set of companies in the United States?

2:46 And Sebastian, Nathan, it's good to see you guys.

2:50 guys.

2:50 So Sebastian, who do you think is winning?

2:53 Winning is a very broad term.

2:57 I would say you mentioned the DeepSeek moment,

2:59 and I think DeepSeek is winning the hearts of the people

3:02 who work on open-weight models because they share these as open models.

3:06 Winning, I think, has multiple timescales to it.

3:09 We have today, we have next year, we have in 10 years.

3:13 One thing I know for sure is that I don't think nowadays, in 2026,

3:18 that there will be any company that has access

3:22 to technology that no other company has access to.

3:26 That is mainly because researchers are frequently changing jobs and labs.

3:32 They rotate.

3:32 I don't think there will be a clear winner in terms of technology access.

3:36 However, I do think there will be,

3:39 The differentiating factor will be budget and hardware constraints.

3:43 I don't think the ideas will be proprietary,

3:46 but rather the resources needed to implement them.

3:52 I don't see currently a winner-take-all scenario.

3:55 I can't see that.

3:57 At the moment.

3:59 Nathan, what do you think?

4:00 You see the labs put different energy into what they're trying to do,

4:04 and I think to demarcate the point in time when we're recording this, the hype

4:08 over Anthropic's Claude Opus 4.5 model

4:11 has been absolutely insane, which is just...

4:13 I mean, I've used it and built stuff in the last few weeks, and it's...

4:17 it's almost gotten to the point where it feels like

4:19 a bit of a meme in terms of the hype.

4:21 And it's kind of funny because this is very organic,

4:24 and then if we go back a few months ago,

4:26 we can see the release date and the notes,

4:29 as Gemini 3 from Google got released, and it seemed like the marketing and just,

4:34 like, wow factor of that release was super high.

4:37 But then at the end of November,

4:39 Claude Opus 4.5 was released and the hype has been growing,

4:42 but Gemini 3 was before this.

4:43 And it kind of feels like people don't really talk about it as much,

4:46 even though when it came out, everybody was like,

4:48 this is Gemini's moment to retake Google's structural advantages in AI.

4:53 And Gemini 3 is a fantastic model, and I still use it.

4:56 It's just kind of differentiation is lower.

4:59 And I agree with Sebastian;

5:01 what you're saying with all these, the idea space is very fluid,

5:05 but culturally Anthropic is known for betting very hard on code,

5:09 which is the Claude Code thing, is working out for them right now.

5:12 So I think that even if the ideas flow pretty freely,

5:15 so much of this is bottlenecked

5:16 by human effort and the culture of organizations,

5:19 where Anthropic seems to at least be presenting as the least chaotic.

5:23 It's a bit of an advantage, if they can keep doing that for a while.

5:27 But on the other side of things, there's a lot of ominous technology from China

5:31 where there's way more labs than DeepSeek.

5:34 So DeepSeek kicked off a movement within China,

5:37 I say kind of similar to how ChatGPT kicked off

5:40 a movement in the US where everything had a chatbot.

5:43 There's now tons of tech companies in China

5:46 that are releasing very strong frontier open-weight models,

5:48 to the point where I would say that DeepSeek is kind

5:51 of losing its crown as the preeminent open model maker in China,

5:54 and the likes of Z.ai with their GLM models, Minimax's models,

6:00 Kimi Moonshot, especially in the last few months, has shown more brightly.

6:04 The new DeepSeek models are still very strong, but that's kind of a...

6:08 it could look back as a big narrative point

6:10 where in 2025 DeepSeek came and it provided this platform

6:14 for way more Chinese companies that are releasing these fantastic

6:17 models to kind of have this new type of operation.

6:20 So these models from these Chinese companies are open-weights,

6:23 and depending on this trajectory of business

6:25 models that these American companies are doing, they could be at risk.

6:29 But currently, a lot of people are paying for AI software in the US,

6:33 and historically in China and other parts of the world,

6:36 people don't pay a lot for software.

6:38 So some of these models like DeepSeek have

6:40 the love of the people because they are open-weight.

6:42 How long do you think the Chinese companies keep releasing open-weight models?

6:47 I would say for a few years.

6:49 I think that, like in the US, there's not a clear business model for it.

6:53 I have been writing about open models for a while,

6:55 and these Chinese companies have realized it.

6:57 So I get inbound from some of them.

6:59 And they're smart and realize the same constraints:

7:01 a lot of top US tech companies and other IT companies

7:04 won't pay for an API subscription to Chinese companies for security concerns.

7:08 This has been a long-standing habit in tech,

7:11 and the people at these companies then see open weight models as an ability

7:16 to influence and take part of a huge growing AI expenditure market in the US.

7:20 And they're very realistic about this, and it's working for them.

7:24 I think that the government will see that that is building

7:27 a lot of influence internationally in terms of uptake of the technology,

7:31 so there's going to be a lot of incentives to keep it going.

7:34 But building these models and doing the research is very expensive,

7:37 so at some point, I expect consolidation.

7:39 But I don't expect that to be a story of 2026,

7:42 where there will be more open model

7:45 builders throughout 2026 than there were in 2025.

7:47 And a lot of the notable ones will be in China.

7:50 You were going to say something?

7:51 Yes.

7:52 You mentioned DeepSeek losing its crown.

7:54 I do think to some extent, yes, but we also have to consider though,

8:00 they are still, I would say, slightly ahead.

8:02 And the other ones—it's not that DeepSeek got worse,

8:04 it's just that the other ones are using the ideas from DeepSeek.

8:08 For example, you mentioned Kimi—same architecture, they're training it.

8:11 And then again, we have this leapfrogging where they might be at some

8:14 point in time a bit better because they have the more recent model.

8:17 And I think this comes back to the fact that there won't be a clear winner.

8:22 It will just be like that: one person releases something,

8:25 the other one comes in, and the most

8:27 recent model is probably always the best model.

8:30 Yeah.

8:30 We'll also see the Chinese companies have different incentives.

8:33 Like, DeepSeek is very secretive,

8:35 whereas some of these startups are like the MiniMaxs and Z.ais of the world.

8:40 Those two literally have filed IPO paperwork,

8:42 and they're trying to get Western mindshare and do a lot of outreach there.

8:47 So I don't know if these incentives will change the model development,

8:50 because DeepSeek famously is built by a hedge fund,

8:53 Highflyer Capital, and we don't know exactly what they

8:56 use the models for or if they care about this.

8:59 They're secretive in terms of communication;

9:00 they're not secretive in terms of the technical

9:02 reports that describe how their models work.

9:04 They're still open on that front.

9:05 And we should also say, on the Claude Opus 4.5 hype,

9:10 there's the layer of something being the darling of the X echo chamber,

9:17 on the Twitter echo chamber,

9:18 and the actual amount of people that are using the model.

9:22 I think it's probably fair to say that ChatGPT and Gemini are focused

9:26 on the broad user base that just want to solve problems in their daily lives,

9:32 and that user base is gigantic.

9:34 So the hype about the coding may not be representative of the actual use.

9:39 I would say also a lot of the usage patterns are,

9:43 like you said, name recognition,

9:44 brand and stuff, but also muscle memory almost, where,

9:48 you know, ChatGPT has been around for a long time.

9:51 People just got used to using it, and it's almost like a flywheel:

9:54 they recommend it to other users and that stuff.

9:57 One interesting point is also the customization of LLMs.

10:00 For example, ChatGPT has a memory feature, right?

10:03 And so you may have a subscription and you use it for personal stuff,

10:07 but I don't know if you want to use that same thing at work.

10:10 Because it's a boundary between private and work.

10:12 If you're working at a company,

10:13 they might not allow that or you may not want that.

10:16 And I think that's also an interesting

10:18 point where you might have multiple subscriptions.

10:20 One is just clean code.

10:22 It has nothing of your personal images or hobby projects in there.

10:26 It's just like the work thing.

10:28 And then the other one is your personal thing.

10:30 So I think that's also something where there are two different use cases,

10:32 and it doesn't mean you only have to have one.

10:36 I think the future is also multiple ones.

10:39 What model do you think won 2025,

10:40 and what model do you think is going to win '26?

10:43 I think in the context of consumer chatbots,

10:45 it's a question of: are you willing to bet on Gemini over ChatGPT?

10:50 Which I would say, in my gut,

10:52 feels like a bit of a risky bet because OpenAI has been the incumbent,

10:55 and there are so many benefits to that in tech.

10:58 I think the momentum, if you look at 2025,

11:02 was on Gemini's side, but they were starting from such a low point.

11:07 And RIP Bard and these earlier attempts at getting started.

11:13 Huge credit to them for powering through

11:15 the organizational chaos to make that happen.

11:17 But also it's hard to bet against OpenAI

11:19 because they always come off as so chaotic,

11:23 but they're very good at landing things.

11:25 And I think, personally, I have very mixed reviews of GPT-5,

11:29 but it must have saved them so much money with the high-line feature being

11:33 a router where most users are no longer charging their GPU costs as much.

11:38 So I think it's very hard to dissociate the things that I like out

11:43 of models versus the things that are

11:45 going to actually be a general public differentiator.

11:50 What do you think about 2026?

11:51 Who's going to win?

11:53 I'll say something, even though it's risky.

11:54 I think Gemini will continue to make progress on ChatGPT.

11:56 I think Google's scale,

11:58 when both of these are operating at such extreme scales—and Google

12:02 has the ability to separate research and product a bit better,

12:06 whereas you hear so much about OpenAI

12:08 being chaotic operationally and chasing the high-impact thing,

12:11 which is a very startup culture.

12:13 And then on the software and enterprise side,

12:15 I think Anthropic will have continued success,

12:16 as they've again and again been set up for that.

12:19 And obviously Google Cloud has a lot of offerings,

12:23 but I think this kind of Gemini name brand is important for them to build.

12:27 Google Cloud will continue to do well,

12:30 but that's a more complex thing to explain in the ecosystem,

12:35 because that's competing with the likes of Azure

12:37 and AWS rather than on the model provider side.

12:41 So in infrastructure, you think TPU is giving an advantage?

12:46 Largely because the margin on NVIDIA chips is insane,

12:49 and Google can develop everything from top to bottom

12:51 to fit their stack and not have to pay this margin.

12:54 And they've had a head start in building data centers.

12:57 So all of these things that have both high

12:59 lead times and very hard margins on high costs,

13:02 Google has a just kind of historical advantage there.

13:05 And if there's going to be a new paradigm, it's most likely to come from OpenAI

13:09 where their research division again and again

13:12 has shown this ability to land a new research idea or a product.

13:17 Like Deep Research, Sora,

13:18 o1 thinking models—all these definitional things have come from OpenAI,

13:23 and that's got to be one of their top traits as an organization.

13:27 So it's kind of hard to bet against that, but I think a lot of this year

13:31 will be about scale and optimizing what

13:33 could be described as low-hanging fruit in models.

13:37 And clearly there's a trade-off between intelligence and speed.

13:41 This is what ChatGPT-5 was trying to solve behind the scenes.

13:46 It's like, do people actually want intelligence,

13:49 the broad public, or do they want speed?

13:52 I think it's a nice variety, or the option to have a toggle there.

13:56 I mean, for my personal usage, most of the time when I look something up,

14:00 I use ChatGPT to ask a quick question, get the information I wanted fast.

14:04 For most daily tasks, I use the quick model.

14:07 Nowadays, I think the auto mode is pretty good

14:09 where you don't have to specifically say thinking or non-thinking.

14:12 Then again, I also sometimes want the pro mode.

14:15 Very often what I do is, when I have something written,

14:18 I put it into ChatGPT and say, "Hey, do a very thorough check.

14:23 Are all my references correct?

14:24 Are all my thoughts correct?

14:26 Did I make any formatting mistakes and are

14:28 the figure numbers wrong?" Or something like that.

14:31 And I don't need that right away.

14:33 I finish my stuff, maybe have dinner, let it run, come back and go through this.

14:38 I think this is where it's important to have this option.

14:42 I would go crazy if for each query I

14:43 would have to wait 30 minutes or 10 minutes even.

14:46 That's me.

14:48 I'm sitting over here losing my mind

14:50 that you use the router and the non-thinking model.

14:52 I'm like, "How do you live with that?" That's like my reaction.

14:57 I've been heavily on ChatGPT for a while.

15:01 I never touched ChatGPT-5 non-thinking.

15:03 I find its tone and then its propensity

15:05 for errors—it has a higher likelihood of errors.

15:08 Some of this is from back when OpenAI released o3,

15:11 which was the first model to do this deep

15:14 search and find many sources and integrate them for you.

15:17 I became habituated with that.

15:18 So I will only use GPT-5.2 Thinking or Pro

15:21 when I'm finding any sort of information query for work,

15:25 whether that's a paper or some code reference that I found.

15:28 And I will regularly have like five Pro queries going simultaneously,

15:33 each looking for one specific paper or feedback on an equation or something.

15:38 I have a fun example where I needed the answer as fast

15:41 as possible for this podcast before I was going on the trip.

15:46 like a local GPU running at home and I wanted to run a long RL experiment.

15:50 And usually I also unplug things because you never know if you're not at home,

15:54 you don't want things plugged in.

15:56 And I accidentally unplugged the GPU.

15:57 My wife was already in the car and it's like,

16:00 "Oh dang." Then basically I wanted as fast as possible

16:04 a Bash script that runs my different experiments and the evaluation.

16:09 And it's something I know,

16:10 I learned how to use the Bash interface or Bash terminal,

16:14 but in that moment I just needed like 10 seconds, give me the command.

16:18 This is a hilarious situation but yeah, so what did you use?

16:21 So I did the non-thinking fastest model.

16:23 It gave me the Bash command to chain

16:26 different scripts to each other and then the thing

16:29 is like you have the tee thing where you want to route this to a log file.

16:34 Top of my head I was just like in a hurry, I could have thought about it myself.

16:37 By the way I don't know if there's a representative case,

16:39 wife waiting in the car-...

16:40 you have to run, you know, unplug the GPU.

16:42 You have to generate a Bash script.

16:43 This sounds like a movie, like- Mission Impossible.

16:46 I use Gemini for that.

16:47 So I use thinking for all the information stuff and then

16:50 Gemini for fast things or stuff that I could sometimes Google,

16:52 which is like it's good at explaining things and I trust

16:55 that it has this kind of background of knowledge and it's simple.

16:59 And the Gemini app has gotten a lot

17:00 better and- It's good for those sorts of things.

17:02 And then for code and any sort of philosophical discussion,

17:05 I use Claude Opus 4.5.

17:07 Also always with extended thinking.

17:09 Extended thinking and inference time scaling is just

17:11 a way to make the models marginally smarter.

17:14 And I will always err on that side when the progress is

17:18 very high because you don't know when that'll unlock a new use case.

17:21 And then sometimes use Grok for real-time

17:24 information or finding something on AI Twitter

17:26 that I knew I saw and I need to dig up and I just fixated on.

17:31 Although when Grok 4 came out, the Grok 4 SuperGrok Heavy,

17:35 which was like their pro variant was actually

17:37 very good and I was pretty impressed with it,

17:38 and then it just kind of like muscle memory

17:41 lost track of it with having the ChatGPT app open.

17:44 So I use many different things.

17:46 Yeah.

17:46 I actually do use Grok 4 Heavy for debugging.

17:50 For like hardcore debugging that the other ones can't solve,

17:54 I find that it's the best at.

17:56 And...

17:56 it's interesting 'cause you say ChatGPT is the best interface.

18:00 For me, for that same reason,

18:02 but this could be just momentum- Gemini is the better interface for me.

18:07 I think because I fell in love with their best needle in the haystack.

18:11 If I ever put something that has a lot of context but I'm looking

18:15 for very specific kinds of information to make sure it tracks all of it,

18:19 I find at least that Gemini for me has been the best.

18:24 So it's funny with some of these models,

18:26 if they win your heart over- for one particular feature on one particular day,

18:31 for that particular query, that prompt, you're like,

18:35 "This model's better." And so you'll just stick with it

18:38 for a bit until it does something really dumb.

18:41 There's like a threshold effect.

18:43 Some smart thing and then you fall in love with it

18:45 and then it does some dumb thing and you're like, "You know what?

18:47 I'm gonna switch and try Claude or ChatGPT." And all that kind of stuff.

18:51 This is exactly it: you use it until it breaks,

18:53 until you have a problem, and then you change the LLM.

18:57 And I think it's the same as how we use anything,

19:01 like our favorite text editor, operating systems, or the browser.

19:04 I mean, there are many options: Safari, Firefox, Chrome.

19:07 They're relatively similar, but then there are edge cases,

19:11 extensions you want, and then you switch.

19:14 But I don't think anyone types the same

19:18 thing into different browsers and compares them.

19:21 You only do that when something breaks.

19:23 So that's a good point.

19:25 You use it until it breaks, then you explore other options.

19:28 On the long context thing, I was also a Gemini user,

19:31 but the GPT-5.2 release blog had crazy long context scores.

19:34 People were like, "Did they just figure out some algorithmic change?"

19:38 It went from 30% to 70% in this minor model update.

19:42 It's very hard to keep track of all of these things,

19:46 but now I look more favorably at GPT-5.2's long context.

19:49 So it's just like, "How do I actually

19:53 get to testing this?" It's a never-ending battle.

19:57 Well, it's interesting that none of us talked

19:59 about the Chinese models from a usage perspective.

20:02 What does that say?

20:04 Does it mean the Chinese models are not as good,

20:08 or are we just very biased and US-focused?

20:11 I think currently there's a discrepancy between the model and the platform.

20:15 The open models are more known for the open weights, not the platform yet.

20:19 known for the open weights, not their platform yet.

20:21 Many companies will sell you open-model inference at a very low cost.

20:25 With OpenRouter, it's easy to look at multi-model things.

20:29 You can run DeepSeek on Perplexity.

20:31 Sitting here, we're like, "We use OpenAI GPT-5 Pro consistently." We're all

20:36 willing to pay for the marginal intelligence gain.

20:39 These models from the US are better in terms of the outputs.

20:45 I think the question is,

20:47 will they stay better for this year and for years to come?

20:51 As long as they're better, I'm gonna pay for them.

20:55 There's also analysis showing that the way

20:59 the Chinese models are served—you could argue

21:01 this is due to export controls— is that they use fewer GPUs per replica,

21:05 which makes them slower and have different errors.

21:07 If speed and intelligence are in your favor as a user,

21:10 in the US, a lot of users will go for this.

21:13 And I think that will spur these Chinese

21:15 companies to want to compete in other ways,

21:18 whether it's free or substantially lower costs,

21:21 or it'll breed creativity in terms of offerings,

21:24 which is good for the ecosystem.

21:26 But the simple thing is: the US models are currently better, and we use them.

21:30 I tried these other open models, and I'm like, "Fun,

21:32 but I don't go back." models, and I'm like, "Fun, but not gonna...

21:36 I don't go back to it."- We didn't really mention programming.

21:40 That's another use case that a lot of people deeply care about.

21:45 I use basically half-and-half Cursor and Claude Code, because they're...

21:49 I fundamentally different experiences and both are useful.

21:54 What do you guys...

21:55 You program quite a bit, so what do you use?

21:57 What's the current vibe?

21:59 So, I use the Codeium plugin for VS Code.

22:02 You know, it's very convenient.

22:03 It's just like a plugin,

22:04 and then it's a chat interface that has access to your repository.

22:06 I know that Claude Code is, I think, a bit different.

22:10 It is a bit more agentic.

22:11 It touches more things.

22:12 It does the whole project for you.

22:13 I'm not quite there yet where I'm comfortable

22:16 with that because maybe I'm a control freak,

22:18 but I still would like to see a bit what's going on.

22:21 And Codeium is kind of, right now, for me,

22:24 the sweet spot where it is helping me, but it is not taking completely over.

22:29 I should mention, one of the reasons I do use

22:31 Claude Code is to build the skill of programming with English.

22:34 I mean, the experience is fundamentally different.

22:37 You're...

22:38 As opposed to micromanaging the details

22:40 of the process of the generation of the code, and looking at the diff,

22:45 which you can in Cursor if that's the IDE you use, and in changing, altering.

22:52 Looking and reading the code and understanding the code deeply as you progress,

22:56 versus just thinking in this design space

23:00 and just guiding it at this macro level,

23:05 which I think is another way of thinking about the programming process.

23:10 Also, we should say that Claude Code just seems

23:14 to be somehow a better utilization of Claude Opus 4.5.

23:19 It's a good side-by-side for people to do.

23:20 You can have Claude Code open,

23:21 you can have Cursor open, you can have VS Code open,

23:24 and you can select the same models on all of them— ...and ask questions,

23:27 and it's very interesting.

23:28 Claude Code is way better in that domain.

23:32 It's remarkable.

23:33 All right, we should say that both of you are legit on multiple fronts:

23:36 researchers, programmers, educators, Tweeters.

23:43 And on the book front, too.

23:45 So Nathan, at some point soon, hopefully has an RLHF book coming out.

23:50 It's available for preorder, and there's a full digital preprint.

23:54 I'm just making it pretty and better organized for the physical thing,

23:56 which is a lot of why I do it,

23:58 because it's fun to create things that you think are excellent

24:01 in the physical form when so much of our life is digital.

24:05 I should say, going to Perplexity here,

24:07 Sebastian Raschka is a machine learning researcher

24:09 and author known for several influential books.

24:11 A couple of them that I wanted to mention—which is

24:14 a book I highly recommend—Build a Large Language Model from Scratch,

24:18 and the new one, Build a Reasoning Model from Scratch.

24:21 So, I'm really excited about that.

24:24 Building stuff from scratch is one of the most powerful ways of learning.

24:28 Honestly, building an LLM from scratch is a lot of fun.

24:30 It's also a lot to learn.

24:31 And like you said, it's probably the best

24:33 way to learn how something really works,

24:35 'cause you can look at figures, but figures can have mistakes.

24:38 You can look at concepts and explanations, but you might misunderstand them.

24:43 But if there is code, and the code works, you know it's correct.

24:48 I mean, there's no misunderstanding.

24:50 It's precise.

24:50 Otherwise, it wouldn't work.

24:52 And I think that's the beauty behind coding.

24:54 It doesn't lie.

24:56 It's math, basically.

24:57 So, even though with math,

24:59 I think you can have mistakes in a book you would never notice.

25:02 Because you are not running the math when you are reading the book,

25:05 you can't verify this.

25:06 And with code, what's nice is you can verify it.

25:09 Yeah, I agree with you about the Build an LLM from Scratch book.

25:12 It's nice to tune out everything else,

25:14 the internet and so on, and just focus on the book.

25:16 But, you know, I read several history books.

25:21 It's just less lonely somehow.

25:24 It's really more fun.

25:25 Like for example, on the programming front,

25:28 I think it's genuinely more fun to program with an LLM.

25:31 And I think it's genuinely more fun to read with an LLM.

25:36 But you're right.

25:37 That distraction should be minimized.

25:40 So you use the LLM to basically enrich the experience, maybe add more context.

25:48 I just find the rate of aha moments for me is really high with LLMs.

25:55 100%.

25:55 I also want to correct myself: I'm not suggesting not to use LLMs.

25:58 I suggest doing it in multiple passes.

26:01 Like, one pass just offline, focus mode, and then after that...

26:05 I mean, I also take notes, but I,

26:07 I try to resist the urge to immediately look things up.

26:11 I do a second pass.

26:13 It's just more structured this way.

26:15 Sometimes things are answered in the chapter,

26:18 but sometimes also it just helps to let it sink in and think about it.

26:22 Other people have different preferences.

26:24 I highly recommend using LLMs when reading books.

26:26 For me, it's not the first thing to do; it's the second pass.

26:30 My recommendation is the opposite.

26:31 I like to use the LLM at the beginning to lay out

26:36 the full context of what is this world that I'm now stepping into?

26:40 But I try to avoid clicking out of the LLM into the world of Twitter and blogs,

26:47 because then you're down this rabbit hole.

26:50 You're reading somebody's opinion.

26:51 There's a flame war about a particular topic and all of a sudden

26:55 you're in the realm of the internet and Reddit and so on.

27:00 But if you're purely letting the LLM give you the context of why this matters,

27:05 what are the big picture ideas...

27:07 sometimes books are good at doing that, but not always.

27:12 This is why I like the ChatGPT app,

27:14 because it gives the AI a home on your computer where you can focus on it,

27:17 rather than just being another tab in my mess of internet options.

27:21 And I think Claude Code does a good job of making that a joy,

27:26 where it seems very engaging as a product design to be

27:30 an interface that your AI will then go out into the world.

27:34 It's something that is intangible between it and Codex;

27:37 it just feels warm and engaging, where Codex can often be as good from OpenAI,

27:42 but it just, feel a little bit rough around the edges.

27:46 Whereas Claude Code makes it fun to build things from scratch,

27:50 where you just trust that it'll make something.

27:53 Obviously this is good for websites and kind of refreshing tooling

27:57 and stuff like this, which I use it for, or data analysis.

28:01 For my On my blog, we scrape Hugging Face

28:04 so we keep download numbers for every dataset and model.

28:06 over time, so we have them.

28:07 And Claude was just like, "Yeah,

28:09 I've made use of that data, no problem." And I was like,

28:12 "That would've taken me days." And then

28:14 I have enough situational awareness to be like,

28:16 "Okay, these trends obviously make sense." You can check things.

28:18 But that's just a wonderful interface where you

28:20 can have an intermediary and not have to do

28:23 the kind of awful low-level work that you

28:26 would have to do to maintain different web projects.

28:29 All right.

28:30 So we just talked about a bunch of the closed-weight models.

28:33 Let's talk about the open ones.

28:36 Tell me about the landscape of open LLM models.

28:39 Which are interesting?

28:40 Which stand out to you and why?

28:42 We already mentioned DeepSeek R1.

28:45 Do you wanna see how many we can name off the top of our head?

28:47 Yeah, without looking at notes.

28:49 DeepSeek, Kimi, MiniMax, Z.ai, Moonshot.

28:53 We're just going Chinese.

28:57 Let's throw in Mistral AI, Gemma...

29:01 ...gpt-oss, the open weight model by OpenAI.

29:04 Actually, NVIDIA had a really cool one, Nemotron 3.

29:09 There, there's a lot of stuff especially at the end of the year.

29:11 Qwen might be the one—- Oh, yeah.

29:13 Qwen was the obvious name I was gonna say.

29:15 You can get at least 10 Chinese and at least 10 Western.

29:18 I think that OpenAI released their first open model— ...since GPT-2.

29:23 When I was writing about OpenAI's open model release, they were like,

29:27 "Don't forget about GPT-2," which I thought was

29:29 really funny 'cause it's just such a different time.

29:32 But gpt-oss-120b is actually a very strong model and does

29:35 some things that other models don't do very well.

29:39 Selfishly, I'll promote a bunch of Western companies

29:43 in the US and Europe that have these fully open models.

29:46 I work at the Allen Institute for AI,

29:48 where we've been building OLMo, which releases data and code.

29:51 And now we have actual competition for people that are

29:55 trying to release everything so that others can train these models.

29:58 There's the Institute for Foundation Models/LM360,

30:00 which has had their K2 models of various types.

30:04 Apertus is a Swiss research consortium.

30:07 Hugging Face has SmolLM, which is very popular.

30:12 And NVIDIA's Nemotron 3 has started releasing data as well.

30:15 And then Stanford's Martini Community Project,

30:17 which is kind of making it so there's a pipeline for people to open a GitHub

30:21 issue and implement a new idea and then

30:23 have it run in a stable language modeling stack.

30:26 This space, that list was way smaller in 2024— ...so I think it was just AI2.

30:32 So it's a great thing for more people

30:34 to get involved and to understand language models,

30:36 which doesn't really have a Chinese analog.

30:39 While I'm talking, I'll say that the Chinese

30:44 open language models tend to be much bigger,

30:47 and that gives them higher peak performance as MoEs,

30:49 where a lot of these things that we like a lot,

30:52 whether it was Gemma and Nemotron, have tended to be smaller models from the US,

30:57 which is starting to change from the US and Europe.

30:59 Mistral Large 3 came out, which was a giant MoE model,

31:02 very similar to DeepSeek architecture in December.

31:05 And then a startup, RCAI,

31:08 and both Nemotron and NVIDIA have teased MoE models way bigger than 100

31:15 billion parameters- like this 400 billion parameter

31:17 range coming in this Q1 2026 timeline.

31:20 So I think this kind of balance is set

31:23 to change this year in terms of what people

31:25 are using the Chinese versus US open models for, which

31:28 I'm personally going to be very excited to watch.

31:32 First of all, huge props for being able to name so many of these.

31:36 Did you actually name LLaMA?

31:39 No.

31:39 I feel like...

31:41 RIP.

31:41 This was not on purpose.

31:43 RIP LLaMA.

31:45 All right.

31:45 Can you mention some interesting models that stand out?

31:48 You mentioned Qwen 3 is obviously a standout.

31:51 So I would say the year's almost bookended by both DeepSeek V3 and R1.

31:56 And then on the other hand, in December, DeepSeek-V3.2.

31:59 Because what I like about those is they always

32:01 have an interesting architecture tweak that others don't have.

32:05 But otherwise, if you want to go with the familiar but really good performance,

32:09 Qwen 3 and, like Nathan said, also gpt-oss-120b.

32:13 And I think what's interesting about it is it's kind of like the first

32:18 public or open weight model that was really trained with tool use in mind,

32:22 which I do think is kind of a paradigm

32:25 shift where the ecosystem was not quite ready for it.

32:27 By tool use, I mean that the LLM is able

32:30 to do a web search or to call a Python interpreter.

32:33 And I do think it's a standout because it's a huge unlock.

32:37 Because one of the most common complaints about LLMs are,

32:41 for example, hallucinations, right?

32:43 And so, in my opinion, one of the best ways to solve hallucinations is

32:46 to not try to always remember information or make things up.

32:51 For math, why not use a calculator app or Python?

32:54 If I ask the LLM, "Who won the soccer

32:57 World Cup in 1998?" instead of just trying to memorize, it could go do a search.

33:03 I think mostly it's still a Google search.

33:06 So ChatGPT and gpt-oss-120b, they would do a tool call to Google,

33:09 maybe find the FIFA website.

33:11 Find, okay, it was France.

33:13 It would get you that information reliably

33:15 instead of just trying to memorize it.

33:17 So I think it's a huge unlock which right

33:20 now is not fully utilized yet by the open-source, open-weight ecosystem.

33:24 A lot of people don't use tool call modes because I think,

33:28 first, it's a trust thing.

33:29 You don't want to run this on your computer where it has access to tools,

33:32 could wipe your hard drive or whatever.

33:34 So you want to maybe containerize that.

33:36 But I do think that is like a really

33:40 important step for the upcoming years to have this ability.

33:44 So a few quick things.

33:45 First of all, thank you for defining what you mean by tool use.

33:49 I think that's a great thing to do

33:50 in general for the concepts we're talking about.

33:53 Even things as sort of well-established as MoEs.

33:57 You have to say that means mixture of experts,

34:00 and you kind of have to build up an intuition for people what that means,

34:04 how it's actually utilized, what are the different flavors.

34:06 So what does it mean that there's just such an explosion of open models?

34:11 What's your intuition?

34:13 If you're releasing an open model,

34:14 you want people to use it, is the first and foremost thing.

34:17 And then after that comes things like transparency and trust.

34:20 I think when you look at China, the biggest reason is that they want

34:24 people around the world to use these models,

34:26 and I think a lot of people will not.

34:28 If you look outside of the US, a lot of people will not pay for software,

34:31 but they might have computing resources where you

34:32 can put a model on it and run it.

34:34 I think there can also be data that you don't want to send to the cloud.

34:37 So the number one thing is getting people to use models, use AI,

34:41 or use your AI that might not be able

34:43 to do it without having access to the model.

34:46 I guess we should state explicitly,

34:47 so we've been talking about these Chinese models and open weight models.

34:51 Oftentimes, the way they're run is locally.

34:54 So it's not like you're sending your data

34:58 to China or to whoever developed Silicon Valley, or whoever developed the model.

35:04 A lot of American startups make money

35:06 by hosting- ...these models from China and selling them.

35:09 It's called selling tokens,

35:11 which means somebody will call the model to do some piece of work.

35:15 I think the other reason is for US companies like OpenAI.

35:18 They are so GPU deprived.

35:20 They're at the limits of the GPUs.

35:22 Whenever they make a release, they're always talking about like,

35:25 "Our GPUs are hurting." And I think

35:27 during one of these gpt-oss-120b release sessions,

35:30 Sam Altman said, "Oh, we're releasing this because we can use your GPUs.

35:33 We don't have to use our GPUs, and OpenAI can still get distribution out

35:38 of this," which is another very real thing,

35:41 because it doesn't cost them anything.

35:44 And for the user, I think also,

35:45 there are users who just use the model locally how they would use ChatGPT.

35:49 But also for companies I think it's a huge

35:51 unlock to have these models because you can customize them,

35:53 you can train them, you can add post-training, add more data.

35:58 Like, specialize them into, let's say, law, medical models, whatever you have.

36:02 And the appeal, you mentioned Llama,

36:04 the appeal of the open-weight models from China

36:07 is that the open-weight models' licenses are even friendlier.

36:11 I think they are just unrestricted open source licenses

36:13 where if we use something like Llama or Gemma, there are some strings attached.

36:17 I think it's like an upper limit in terms of how many users you have.

36:20 And then if you exceed, I don't know, so and so many million users,

36:23 you have to report your financial situation to, let's say,

36:26 Meta or something like that.

36:28 And I think while it is a free model, there are strings attached,

36:33 and people do like things where strings are not attached.

36:36 So I think that's also one of the reasons, besides performance,

36:39 why the open-weight models from China are so popular,

36:42 because you can just use them.

36:43 There's no catch in that sense.

36:46 The ecosystem has gotten better on that front,

36:48 but mostly downstream of these new providers providing such open licenses.

36:51 That was funny when you pulled up Perplexity and said,

36:53 "Kimi K2 Thinking hosted in the US." Which is just like an exact...

36:56 I've never seen this, but it's an exact example

36:58 of what we're talking about where people are sensitive to this.

37:01 But Kimi K2 Thinking and Kimi K2 is a model that is very popular.

37:05 People say that has very good creative

37:07 writing and also in doing some software things.

37:09 So it's just these little quirks that people

37:11 pick up on with different models that they like.

37:14 What are some interesting ideas that some of these models have

37:18 explored that you can speak to, that are particularly interesting to you?

37:22 Maybe we can go chronologically.

37:23 I mean, there was, of course, DeepSeek.

37:25 DeepSeek R1 that came out in January of 2025, if we just focus on 2025.

37:29 However, this was based on DeepSeek-V3,

37:31 which came out the year before in December 2024.

37:34 There are multiple things on the architecture side.

37:37 What is fascinating is...

37:38 I mean, that's what I do with my from-scratch coding projects.

37:41 You can still start with GPT-2,

37:43 and you can add things to that model to make it into this other model.

37:47 So it's all still kind of like the same lineage.

37:50 It is a very close relationship between those.

37:53 But top of my head, DeepSeek—what was unique there is the Mixture of Experts.

37:57 Not that they were inventing Mixture of Experts—we

38:00 can maybe talk a bit more about what Mixture of Experts means—but just to list

38:04 these things first before we dive into detail.

38:07 Mixture of Experts, but then they also had Multi-head Latent Attention,

38:11 which is a tweak to the attention mechanism, where this was, I would say,

38:17 the main distinguishing factor between these open-weight models.

38:22 Different tweaks to make inference or KV cache size...

38:25 We can also define KV cache in a few moments,

38:29 but to kind of make it more economical to have long context,

38:32 to shrink the KV cache size.

38:34 So what are tweaks that we can do?

38:36 And most of them focused on the attention mechanism.

38:38 There is Multi-head Latent Attention in DeepSeek.

38:41 There is Group Query Attention, which is still very popular.

38:44 It's not invented by any of those models.

38:46 It goes back a few years.

38:47 But that would be the other option.

38:50 Sliding window attention—I think OLMo 3 uses it, if I remember correctly.

38:54 So there are these different tweaks that make the models different.

38:57 Otherwise, I put them all together

39:00 in an article once where I just compared them.

39:03 They are very, surprisingly similar.

39:05 It's just different numbers in terms of how many

39:08 repetitions of the transformer block you have in the center.

39:11 And, like, just little knobs that people tune.

39:14 But what's so nice about it is it works no matter what.

39:17 You can tweak things.

39:19 You can move the normalization layers around to get some performance gains.

39:22 And OLMo is always very good in ablation studies,

39:26 showing what it actually does to the model if you move something around.

39:30 Ablation studies: does it make it better or worse?

39:32 But there are so many, let's say,

39:33 ways you can implement a transformer and make it still work.

39:36 The big ideas that are still prevalent is Mixture of Experts,

39:40 multi-head latent attention, sliding window attention, group query attention.

39:44 And then at the end of the year, we saw a focus on making the attention

39:49 mechanism scale linearly with inference token prediction.

39:52 So there was Qwen2-VL, for example, which added a gated delta net.

39:57 It's kind of inspired by State space models,

40:00 where you have a fixed state that you keep updating.

40:02 But it makes essentially this attention cheaper,

40:06 or it replaces attention with a cheaper operation.

40:08 And it may be useful to step

40:11 back and talk about transformer architecture in general.

40:14 Yeah, so maybe we should start with the GPT-2 architecture.

40:17 The transformer that was derived from the "Attention Is All You Need" paper.

40:21 The "Attention Is All You Need" paper

40:23 had a transformer architecture that had two parts, an encoder and a decoder.

40:28 And GPT went just focusing in on the decoder part.

40:32 It is essentially still a neural network

40:35 and it has this attention mechanism inside.

40:37 And you predict one token at a time.

40:40 You pass it through an embedding layer.

40:43 There's the transformer block.

40:44 The transformer block has attention modules and a fully connected layer.

40:47 And there are some normalization layers in between.

40:50 But it's essentially neural network layers with this attention mechanism.

40:53 So coming from GPT-2 when we move on to gpt-oss-120b,

40:57 there is, for example, the Mixture of Experts layer.

41:00 It's not invented by gpt-oss-120b.

41:02 It's a few years old.

41:04 But it is essentially a tweak to make the model

41:09 larger without consuming more compute in each forward pass.

41:13 So there is this fully connected layer,

41:15 and if listeners are familiar with multi-layer perceptrons,

41:19 you can think of a mini multi-layer perceptron,

41:22 a fully connected neural network layer inside the transformer.

41:25 And it's very expensive, because it's fully connected.

41:27 If you have a thousand inputs and a thousand outputs,

41:30 that's like one million connections.

41:31 And it's a very expensive part in this transformer.

41:34 And the idea is to kind of expand that into multiple feedforward networks.

41:39 So instead of having one, let's say you have 256,

41:43 but it would make it way more expensive,

41:45 because now you have 256, but you don't use all of them at the same time.

41:49 So you now have a router that says, "Okay, based on this input token,

41:52 it would be useful to use this fully connected network." And in that context,

41:57 it's called an expert.

41:58 So a Mixture of Experts means you have multiple experts.

42:01 And depending on what your input is, let's say it's more math-heavy,

42:05 it would use different experts, compared to, let's say,

42:09 translating input text from English to Spanish.

42:11 It would maybe consult different experts.

42:13 It's not quite clear, I mean, not as clear-cut to say, "Okay,

42:16 this is only an expert for math and for Spanish." It's a bit more fuzzy.

42:20 But the idea is essentially that you pack more knowledge into the network,

42:25 but not all the knowledge is used all the time.

42:27 That would be very wasteful.

42:29 So, during the token generation, you are more selective.

42:32 There's a router that selects which tokens should go to which expert.

42:36 It adds more complexity.

42:38 It's harder to train.

42:39 There's a lot that can go wrong, like collapse and everything.

42:42 So I think that's why OLMo 3 still uses dense...

42:45 I mean, you have OLMo models with Mixture of Experts,

42:48 but dense models, where dense means...

42:50 So also, it's jargon.

42:52 There's a distinction between dense and sparse.

42:55 So Mixture of Experts is considered sparse, because we have a lot of experts,

42:59 but only a few of them are active.

43:01 So that's called sparse.

43:01 And then dense would be the opposite,

43:03 where you only have one fully connected module, and it's always utilized.

43:08 So maybe this is a good place to also talk about KV cache.

43:11 But actually, before that, even zooming out, like fundamentally,

43:14 how many new ideas have been implemented from GPT-2 to today?

43:22 Like, how different really are these architectures?

43:25 Take the Mixture of Experts.

43:27 The attention mechanism in gpt-oss-120b,

43:29 that would be the Group Query Attention mechanism.

43:31 So it's a slight tweak from Multi-Head Attention to Group Query Attention.

43:35 So that we have too...

43:36 I think they replaced LayerNorm by RMSNorm,

43:39 but it's just like a different normalization there and not a big change.

43:43 It's just like a tweak.

43:45 The nonlinear activation function— people familiar with deep neural networks,

43:49 I mean, it's the same as changing sigmoid with ReLU.

43:52 It's not changing the network fundamentally.

43:55 It's just a little tweak.

43:56 And that's about it, I would say.

43:59 It's not really fundamentally that different.

44:01 It's still the same architecture.

44:03 So you can go from one into the other by just adding these changes basically.

44:10 It fundamentally is still the same architecture.

44:12 Yep.

44:12 For example, you mentioned my book earlier.

44:14 That's a GPT-2 model in the book because it's simple and it's very small,

44:18 so 124 million parameters approximately.

44:20 But in the bonus materials, I do have OLMo from scratch,

44:25 Gemini 3 from scratch, and other types of from-scratch models.

44:28 And I always start it with my GPT-2 model and just tweak the—well,

44:31 add different components and you get from one to the other.

44:34 It's kind of like a lineage in a sense.

44:38 Can you build up an intuition for people?

44:40 Because when you zoom out, you look at it,

44:43 there's so much rapid advancement in the AI world.

44:46 And at the same time, fundamentally the architectures have not changed.

44:51 So where is all the turbulence, the turmoil of the advancement happening?

44:58 Where are the gains to be had?

45:01 So there are different stages where you

45:03 develop the network or train the network.

45:05 You have the pre-training.

45:06 Now back in the day, it was just pre-training with GPT-2.

45:09 Now you have pre-training, mid-training, and post-training.

45:12 So I think right now we are in the post-training focus stage.

45:17 Pre-training still gives you advantages if you scale it up with better,

45:23 higher quality data.

45:24 But then we have capability unlocks that were not there with GPT-2,

45:28 for For example, ChatGPT is basically a GPT-3 model.

45:32 And GPT-3 is the same as GPT-2 in terms of architecture.

45:36 What was new was adding supervised

45:39 fine-tuning and reinforcement learning with human feedback.

45:41 So it's more on the algorithmic side than the architecture.

45:45 I would say that the systems also change a lot.

45:47 If you listen to NVIDIA's announcements,

45:48 they talk about things like, "You now do FP8,

45:51 you can now do FP4." What's happening is these labs are figuring

45:55 out how to utilize more compute to put it into one model,

45:58 which lets them train faster and put more data in.

46:01 And then you can find better configurations faster by doing this.

46:05 So you can look at, essentially, tokens per second per GPU as a metric

46:09 that you look at when you're doing large-scale training.

46:12 You can go from 10k to 13k by turning on FP8 training,

46:16 which means you're using less memory per parameter in the model.

46:20 By saving less information, you do less communication and train faster.

46:24 So all of these system things underpin

46:27 way faster experimentation on data and algorithms.

46:35 It's a loop that keeps going where it's hard to describe

46:38 when you look at architectures and they're exactly the same,

46:40 but the code base used to train

46:42 these models is vastly different- -and you could probably...

46:45 the GPUs are different but you probably train gpt-oss-20b way

46:49 faster in wall-clock time than GPT-2 was trained at the time.

46:54 Yeah.

46:54 Like you said, they had, for example,

46:56 in Mixture of Experts this FP4 optimization where you get more throughput.

47:00 But I do think, for speed this is true,

47:04 but it doesn't give the model new capabilities.

47:07 It's just: how much can we make the computation

47:11 coarser without suffering in terms of model performance degradation?

47:15 But I do think- I mean, there are alternatives popping up to the transformer.

47:20 Text diffusion models, a completely different paradigm.

47:23 And there is also...

47:24 I mean, although text diffusion models might use transformer architectures,

47:27 it's not an autoregressive transformer.

47:30 And also Mamba models.

47:32 It's a state space model.

47:34 But they do have trade-offs, and nothing has yet replaced

47:39 the autoregressive transformer as the state-of-the-art model.

47:42 For state-of-the-art, you would still go with that, but there are now

47:46 alternatives for the cheaper end—alternatives

47:49 that are kind of making compromises.

47:52 It's not just one architecture anymore.

47:54 There are little ones coming up.

47:57 But if we talk about the state-of-the-art,

47:59 it's pretty much still the transformer architecture,

48:02 autoregressive, derived from GPT-2 essentially.

48:06 I guess the big question here is,

48:07 we talked quite a bit about the architecture behind the pre-training.

48:11 Are the scaling laws holding strong across pre-training,

48:16 post-training, inference, context size, data, and synthetic data?

48:21 I'd like to start with the technical definition

48:22 of a scaling law- -which informs all of this.

48:24 The scaling law is the power law relationship between...

48:27 You can think of the x-axis,

48:28 so kind of what you are scaling as a combination of compute and data,

48:32 which are kind of similar,

48:34 and then the y-axis is like the held-out prediction accuracy over next tokens.

48:38 We talked about models being autoregressive.

48:39 It's like if you keep a set of text that the model has not seen,

48:45 how accurate will it get when you train?

48:47 And the idea of scaling laws came when people

48:50 figured out that that was a very predictable relationship.

48:53 And I think that that technical term is continuing,

48:57 and then the question is, what do users get out of it?

49:01 Then there are more types of scaling where,

49:03 OpenAI's o1 was famous for introducing inference time scaling.

49:06 And I think less famously for also

49:08 showing that you can scale reinforcement learning training

49:11 and get kind of this log x-axis

49:14 and then a linear increase in performance on y-axis.

49:16 So there's kind of these three axes now where

49:19 the traditional scaling laws are talked about for pre-training,

49:21 which is how big your model is and how big your dataset is,

49:25 and then scaling reinforcement learning, which is like how long can you do

49:28 this trial and error learning that we'll talk about.

49:30 We'll define more of this, and then this inference time compute,

49:33 which is just letting the model generate more tokens on a specific problem.

49:36 So I'm kind of bullish, but they're all really still working,

49:40 but the low-hanging fruit has mostly been taken,

49:43 especially in the last year on reinforcement learning with verifiable rewards,

49:46 which is this RLVR, and then inference time scaling,

49:50 which is just why these models feel so different to use,

49:53 where previously you would get that first token immediately.

49:55 And now they'll go off for seconds, minutes, or even hours,

49:59 generating these hidden thoughts before giving

50:01 you the first word of your answer.

50:03 And that's all about this inference time scaling,

50:05 which is such a wonderful kind of step

50:08 function in terms of how the models change abilities.

50:11 They kind of enabled this tool use stuff and enabled

50:13 this much better software engineering that we were talking about.

50:17 And this, when we say enabled, is almost entirely downstream of the fact

50:21 that this reinforcement learning with verifiable

50:23 rewards training just kind of let the models pick up these skills very easily.

50:27 So let the models learn, so if you look at the reasoning process

50:32 when the models are generating a lot of tokens,

50:34 what it'll often be doing is: it tries a tool, it looks at what it gets back.

50:37 It tries another API, it sees what it gets back and if it solves the problem.

50:41 So the models, when you're training them, very quickly learn to do this.

50:45 And then at the end of the day,

50:47 that gives this kind of general foundation where the model

50:49 can use CLI commands very nicely in your repo

50:52 and handle Git for you and move things around

50:55 and organize things or search to find more information,

50:57 which if we were sitting in these chairs a year ago

51:00 is something that we didn't really think of the models doing.

51:03 So this is just kind of something that has

51:05 happened this year and has totally transformed how

51:07 has totally transformed how we think of using

51:10 AI which evolution and just unlocks so much value.

51:13 But it's like, just so- pr- unlocks so much value.

51:18 But it's- it's like, it's not clear what the next avenue will

51:21 be in terms of unlocking stuff like this.

51:23 I think there's...

51:24 we'll get to continual learning later,

51:25 but there's a lot of buzz around certain areas of AI,

51:28 but no one knows when the next step function will really come.

51:32 So you've actually said quite a lot of things there,

51:35 and said profound things quickly.

51:37 It would be nice to unpack them a little bit.

51:40 You say you're bullish basically on every version of scaling.

51:43 So can we just even start at the beginning?

51:47 Pre-training, are we kind of implying that the low-

51:51 hanging fruit on pre-training scaling has been picked?

51:55 Has pre-training hit a plateau,

51:58 or is even pre-training still something you're bullish on?

52:01 Pre-training has gotten extremely expensive.

52:03 I think to scale up pre-training,

52:05 it's also implying that you're gonna serve a very large model to the users.

52:10 So I think that it's been loosely established the likes of GPT-4

52:14 and similar models were around one trillion parameters at the biggest size.

52:18 There's a lot of rumors that they've actually

52:20 gotten smaller as training has gotten more efficient.

52:23 You want to make the model smaller because

52:25 then your costs of serving go down proportionately.

52:28 These models, the cost of training them is really low relative

52:31 to the cost of serving them to hundreds of millions of users.

52:34 I think DeepSeek had this famous number of about

52:36 five million dollars for pre-training at cloud market rates.

52:40 In OLMo 3, section 2.4 in the paper, we just detailed how long we had the GPU

52:46 clusters sitting around for training which includes engineering issues,

52:50 multiple seeds, and it was like about two million dollars to rent

52:53 the cluster to deal with all the headaches of training a model.

52:56 So these models are pretty— like,

52:59 a lot of people could get one to 10 million dollars to train a model,

53:02 but the recurring costs of serving millions

53:05 of users is really billions of dollars of compute.

53:08 I think that you can look at a thousand

53:11 GPU rental you can pay 100 grand a day for.

53:14 And these companies could have millions of GPUs.

53:16 Like you can look at how much these things cost to sit around.

53:19 So that's kind of a big thing, and then it's like,

53:23 if scaling is actually giving you a better model,

53:25 is it gonna be financially worth it?

53:27 And I think we'll slowly push it out as AI solves more compelling tasks,

53:31 so like the likes of Claude Opus 4.5, making Claude Code just work for things.

53:36 I— I launched this project called the ATOM project,

53:39 which is American Truly Open Models in July,

53:42 and that was like a true vibe coded website,

53:45 and like, I have a job to make plots and stuff.

53:49 And then I came back to refresh it in the last few weeks

53:51 and it's like Claude Opus 4.5 versus whatever model at the time was like,

53:55 just crushed all the issues that it had from building in June and July and like,

53:59 it might be a bigger model.

54:01 There's a lot of things that go into this, but there's still progress coming.

54:04 So what you're speaking to is the nuance of the y-axis

54:07 of the scaling laws—the way it's experienced versus on a benchmark,

54:11 the actual intelligence might be different.

54:13 But still, your intuition about pre-training,

54:16 if you scale the size of compute, will the models get better?

54:21 Not whether it's financially viable but just from the law aspect of it,

54:26 do you think the models will get smarter?

54:28 Yeah.

54:29 And I think that there's...

54:30 And this sometimes comes off as almost like disillusionment from people,

54:34 leadership at AI companies saying this, but they're like,

54:37 "It's held for 13 orders of magnitude of compute,

54:40 why would it ever end?" So I think fundamentally it is pretty unlikely to stop,

54:44 it's just eventually we're not even gonna be able to test

54:47 the bigger scales because of all the problems that come with more compute.

54:50 I think that there's a lot of talk on how

54:54 2026 is a year when very large Blackwell compute clusters,

54:58 like gigawatt-scale facilities at hyperscalers, are coming online.

55:03 These were all contracts for power and data centers

55:06 that were signed and sought out in 2022 and 2023.

55:11 So before or right after ChatGPT.

55:13 It took this two-to-three-year lead time to build

55:16 these bigger clusters to train the models.

55:18 While there's obviously immense interest in building

55:20 even more data centers than that.

55:21 So that is the crux that people are saying: these new clusters are coming.

55:25 The labs are gonna have more compute for training.

55:28 They're going to utilize this, but it's not a given.

55:31 I've seen so much progress that I expect it,

55:34 and I expect a little bit bigger models, and I expect...

55:39 I would say it's more like we'll see a $2,000 subscription this year.

55:42 We've seen $200 subscriptions.

55:43 That could 10X again, and these are the kind of things that could come,

55:47 and they're all downstream of this bigger model

55:50 that offers just a little bit more cutting edge.

55:53 So, you know, it's reported that xAI

55:55 is gonna hit that one-gigawatt scale early '26,

55:59 and a full two gigawatts by year end.

56:03 How do you think they'll utilize that in the context of scaling laws?

56:09 Is a lot of that inference?

56:10 Is a lot of that training?

56:13 It ends up being all of the above.

56:15 So I think that all of your decisions

56:17 when you're training a model come back to pre-training.

56:20 So if you're going to scale RL on a model,

56:22 you still need to decide on your architecture that enables this.

56:25 We were talking about other architectures

56:27 and using different types of attention, or a mixture of experts models.

56:31 The sparse nature of MoE models makes it much more efficient to do generation,

56:37 which becomes a big part of post-training,

56:40 and you need to have your architecture ready

56:42 so that you can actually scale up this compute.

56:45 I still think most of the compute is going in at pre-training.

56:48 Because you can still make a model better,

56:51 you still want to go and revisit this.

56:53 You still want the best base model you can.

56:55 And in a few years that'll saturate and the RL compute will just go longer.

57:00 Are there people who disagree with you and say pre-training is dead?

57:06 It's all about scaling inference, scaling post-training,

57:09 scaling context, continual learning, scaling data, synthetic data?

57:15 People vibe that way and describe it in that way,

57:17 but I think it's not the practice that is happening.

57:19 It's just the general vibe of people saying

57:21 this thing is dead-- The excitement is elsewhere.

57:23 So the low-hanging fruit- ...in RL is elsewhere.

57:26 For example, we released our model in November...

57:28 Every company has deadlines.

57:30 Our deadline was November 20th, and for that, our run was five days,

57:34 which compared to 2024 is a very long time to just

57:37 be doing post- training at a model of 30 billion parameters.

57:40 It's not a big model.

57:41 And then in December, we had another release,

57:43 where we let the RL run for another three and a half weeks,

57:47 and the model got notably better, so we released it.

57:50 And that's a to just allocate to something

57:53 that is going to be your peak- ...for the year.

57:56 So it's like-- The reasoning is-- There's these types

57:59 of decisions when training a model where they just...

58:01 They can't leave it forever.

58:03 You have to keep pulling in the improvements from researchers.

58:07 So you redo pre-training, you'll do this post-training for a month,

58:11 but then you need to give it to your users.

58:14 You need to do safety testing.

58:15 So it's just...

58:16 I think there's a lot in place

58:18 that reinforces this cycle of updating the models.

58:21 Things improve.

58:22 You get a new compute cluster that lets you do something more stably or faster.

58:27 It's like you hear a lot about Blackwell having rollout issues, where at AI2,

58:32 most of the models we're pre-training are on 1,000 to 2,000 GPUs.

58:35 But when pre-training on 10,000 or 100,000 GPUs,

58:38 you hit very different failures.

58:40 GPUs break in weird ways, and on a 100,000 GPU run,

58:44 you're pretty much guaranteed to have one GPU that is down.

58:48 Your training code must handle that redundancy,

58:50 which is a very different problem.

58:51 Whereas what we're doing, like playing with post-training on a cluster,

58:55 or for people learning ML, what they're battling to train these biggest

59:00 models is just- ...mass distributed scale, and it's very different.

59:05 But that's somewhat different than- That's a systems

59:10 problem- ...in order to enable scaling laws, especially at pre-training.

59:15 You need all these GPUs at once.

59:17 When we shift to RL, it actually lends itself to heterogeneous compute

59:21 because you have many copies of the model.

59:24 To do a primer for language model reinforcement learning,

59:28 what you're doing is having two sets of GPUs.

59:31 One you can call the actor, and one you call the learner.

59:34 The learner is where your actual reinforcement learning updates happen.

59:38 These are traditionally policy gradient algorithms.

59:42 Proximal Policy Optimization, PPO, and Group Relative Policy Optimization,

59:46 GRPO, are the two popular classes.

59:50 And on the other side you have actors which are generating completions,

59:54 and these completions are what you're going to grade.

59:57 Reinforcement learning is all about optimizing reward.

1:00:00 In practice, you can have a lot of different actors

1:00:03 in different parts of the world doing different types of problems,

1:00:06 and then you send it back to this highly networked compute

1:00:10 cluster to do this actual learning where you take the gradients.

1:00:14 You need to have a tightly meshed network to do different

1:00:18 types of parallelism and spread out your model for efficient training.

1:00:22 Every different type of training and serving has these considerations to scale.

1:00:28 We talked about pre-training and RL,

1:00:30 and then inference time scaling- how do you serve

1:00:33 a model that's thinking for an hour to 100 million users?

1:00:35 I don't know about that, but I know that's a hard problem.

1:00:39 In order to give people this intelligence, there's all these systems problems,

1:00:42 and we need more compute and you need more stable compute to do it."-

1:00:46 But you're bullish on all of these kinds of scaling is what I'm hearing.

1:00:49 On the inference, on the reasoning, even on the pre-training?

1:00:54 Yeah, so that's a big can of worms here, but there are basically two...

1:00:58 The knobs are the training and the inference scaling where you can get gains.

1:01:02 In a world where we had, let's say,

1:01:05 infinite compute resources, you want to do all of them.

1:01:08 So you have training, you have inference scaling,

1:01:10 and training is like a hierarchy:

1:01:12 it's pre-training, mid-training, post-training.

1:01:13 Changing the model size, more training data,

1:01:16 training a bigger model gives you more knowledge in the model.

1:01:20 Then the model, let's say, has a better base model.

1:01:23 Back in the day, or still, we call it a foundation model, and it unlocks...

1:01:28 But you don't, let's say, have the model be able to solve

1:01:32 your most complex tasks during pre-training or after pre-training.

1:01:36 You still have these other unlock phases

1:01:38 where you have mid-training or, for example,

1:01:41 post-training with RL that unlocks capabilities that the model

1:01:43 has in terms of knowledge in the pre-training.

1:01:45 And I think, sure, if you do more pre-training,

1:01:50 you get a better base model that you can unlock later.

1:01:53 But like Nathan said, it just becomes too expensive.

1:01:55 We don't have infinite compute, so you have to decide,

1:01:58 do I want to spend that compute more on making the model larger?

1:02:01 It's like a trade-off.

1:02:02 In an ideal world, you want to do all of them.

1:02:05 And I think in that sense, scaling is still pretty much alive.

1:02:08 You would still get a better model,

1:02:09 but like we saw with Claude Opus 4.5, it's just not worth it.

1:02:12 Because you can unlock more performance

1:02:16 with other techniques at that current moment,

1:02:19 especially if you look at inference scaling.

1:02:21 That's one of the biggest gains this year with o1,

1:02:24 where it took a smaller model further than

1:02:29 pre-training a larger model like Claude Opus 4.5.

1:02:31 So I wouldn't say pre-training scaling is dead,

1:02:33 it's just that there are other more attractive ways to scale right now.

1:02:37 But at some point, you will still

1:02:39 want to make some progress on the pre-training.

1:02:42 The thing also to consider is where you want to spend your money.

1:02:46 If you spend it more on the pre-training, it's like a fixed cost.

1:02:50 You train the model, and then it has this capability forever.

1:02:53 You can always use it.

1:02:56 With inference scaling, you don't spend money during training,

1:02:58 you spend money later per query, and then it's also like math.

1:03:02 How long is my model gonna be on the market if I replace it in half a year?

1:03:06 Maybe it's not worth spending $5 million,

1:03:08 $10 million, $100 million on training it longer.

1:03:12 Maybe I will just do more inference scaling and get performance there.

1:03:17 It maybe costs me $2 million in terms of user queries.

1:03:19 It becomes a question of how many users you have and doing the math,

1:03:23 and I think that's also where it's interesting where ChatGPT is in a position.

1:03:26 I think they have a lot of users where they need to go a bit cheaper,

1:03:29 where they have that GPT-5 model that is a bit smaller.

1:03:32 Other companies that have...

1:03:34 Let's say, if your customers have other trade-offs.

1:03:38 For example, there was also the Math Olympiad or some

1:03:41 of these math problems where ChatGPT or they had a proprietary model,

1:03:46 and I'm pretty sure it's just like a model

1:03:49 that has been fine-tuned a little bit more, but most of it was during inference

1:03:53 scaling to achieve peak performance in certain tasks.

1:03:56 need that all the time.

1:03:57 But yeah, long story short, I do think all of these pre-training,

1:04:02 mid-training, post-training, inference scaling,

1:04:04 they are all still things you want to do.

1:04:06 It's just finding—at the moment, in this year,

1:04:08 it's finding the right ratio that gives

1:04:10 you the best bang for the buck, basically.

1:04:13 I think this might be a good place to define pre-training,

1:04:16 mid-training, and post-training.

1:04:18 So, pre-training is the classic training one next token prediction at a time.

1:04:21 You have a big corpus of data.

1:04:23 And Nathan probably also has very interesting insights there because of OLMo 3.

1:04:27 A big portion of the paper focuses on the right data mix.

1:04:30 So, pre-training is essentially just, you know, training cross entropy loss,

1:04:34 training on next token prediction on a vast corpus of internet data,

1:04:39 books, papers and so forth.

1:04:41 It has changed a little bit over the years

1:04:43 in the sense people used to throw in everything they can.

1:04:46 Now, it's not just raw data.

1:04:48 It's also synthetic data where people, let's say, rephrase certain things.

1:04:54 So synthetic data doesn't necessarily mean purely AI-made data.

1:04:58 It's also taking something from an article, a Wikipedia article,

1:05:02 and then rephrasing it as a Q&A question or summarizing it,

1:05:07 rewording it, and making better data that way.

1:05:12 Because I think of it also like with humans.

1:05:15 If someone, let's say, reads a book compared to a messy—no offense,

1:05:19 but like—Reddit post or something like that, I do think

1:05:24 you learn—- There's going to be a post about this, Sebastian.

1:05:28 Some Reddit data is very coveted and excellent for training.

1:05:31 You just have to filter it.

1:05:33 And I think that's the idea.

1:05:35 I think it's like if someone took that and rephrased it in a, let's say,

1:05:40 more concise and structured way,

1:05:42 I think it's higher quality data that gets the LLM there faster.

1:05:46 You get the same LLM out of it at the end,

1:05:49 but it trains faster because if the grammar and the punctuation are correct,

1:05:54 it already learns the correct way,

1:05:57 versus getting information from a messy source

1:05:59 and then learning later how to correct that.

1:06:02 So, I think that is how pre-training evolved and why scaling still works.

1:06:09 It's not just about the amount of data,

1:06:13 it's also the tricks to make that data better for you, in a sense.

1:06:17 And then mid-training is...

1:06:18 I mean, it used to be called pre-training.

1:06:21 I think it's called mid-training because it was awkward

1:06:23 to have pre-training and post-training but nothing in the middle, right?

1:06:26 It sounds a bit weird.

1:06:27 You have pre-training and post-training, but what's the actual training?

1:06:29 So, the mid-training is usually similar to pre-training,

1:06:33 but it's a bit more specialized.

1:06:35 It's the same algorithm, but what you do is you focus,

1:06:39 for example, on long-context documents.

1:06:42 The reason you don't do that during pre-training is

1:06:46 because you don't have that many long context documents.

1:06:49 We have a specific phase.

1:06:50 And one problem of LLMs is still that it's a neural network.

1:06:54 It has the problem of catastrophic forgetting.

1:06:56 So, you teach it something, it forgets other things.

1:06:58 And you wanna...

1:06:59 I mean, it's not 100% forgetting,

1:07:01 but it's like "no free lunch." It's the same with humans.

1:07:04 If you ask me some math I learned 10 years ago,

1:07:07 I would have to look at it again.

1:07:09 Nathan was actually saying that he's consuming so

1:07:11 much content that there's a catastrophic forgetting issue.

1:07:14 Yeah, I'm trying to learn so much about AI,

1:07:16 and it's like I was learning about pre-training parallelism.

1:07:18 I'm like, "I lost something and I don't know

1:07:21 what it was."- I don't want to anthropomorphize LLMs,

1:07:23 but it's the same kind of thing in how humans learn.

1:07:27 I mean, quantity is not always better because you have to be selective.

1:07:32 And mid-training is being selective in terms of quality content at the end.

1:07:36 So the last thing the LLM has seen is the quality stuff.

1:07:39 And then post-training is all the fine-tuning, supervised fine-tuning, DPO,

1:07:45 Reinforcement Learning with Verifiable Rewards (RLVR),

1:07:49 with human feedback, and so forth.

1:07:51 So the refinement stages.

1:07:52 And it's also interesting, it's a cost thing.

1:07:54 You spend a lot of money on pre-training right now.

1:07:57 RL a bit less.

1:07:59 With RL, you don't really teach it knowledge.

1:08:02 It's more like unlocking the knowledge; it's more like a skill learning,

1:08:05 like how to solve problems with the knowledge that it has from pre-training.

1:08:09 There are actually three papers this year,

1:08:11 or last year, 2025, on RL for pre-training.

1:08:14 But I don't think anyone does that in production.

1:08:17 Toy examples for now.

1:08:18 Toy examples, right?

1:08:19 But to generalize, RL post-training is more like the skill unlock,

1:08:23 where pre-training is like soaking up the knowledge.

1:08:27 A few things that could be helpful.

1:08:29 A lot of people think of synthetic data as being bad for training the models.

1:08:34 You mentioned how DeepSeek got almost...

1:08:37 OCR, which is Optical Character Recognition.

1:08:39 A lot of labs did it.

1:08:41 Ai2 had one, Meta had multiple.

1:08:44 And the reason each of these labs has

1:08:47 these is because there are vast amounts of PDFs

1:08:49 and other digital documents on the web that aren't

1:08:52 in formats that are encoded with text easily.

1:08:54 So you use these Almost-OCR, DeepSeek OCR, or what we called our Almost-OCR,

1:08:59 to extract trillions of tokens of candidate data for pre-training.

1:09:04 Pre-training dataset size is measured in trillions of tokens.

1:09:08 Smaller models from researchers can be something like five to 10 trillion.

1:09:11 researchers can be something like five to 10 trillion.

1:09:15 Um, Qwen is documented going up to 50 trillion,

1:09:17 and there are rumors that these closed labs can go to 100 trillion tokens.

1:09:21 Getting this potential data to put in—they have a very big funnel,

1:09:24 and the data you actually train on is a small percentage of this.

1:09:29 This character recognition data would be described

1:09:32 as synthetic data for pre-training in a lab.

1:09:34 And then there's also the fact that ChatGPT now gives wonderful answers,

1:09:38 and you can train on those best answers, and that's synthetic data.

1:09:41 It's very different than early ChatGPT with lots of hallucination data.

1:09:46 when people became grounded in synthetic data.

1:09:49 One interesting question is, if I recall correctly,

1:09:51 OLMo 3 was trained with less data than

1:09:53 specifically some other open-weight models, maybe even OLMo 2.

1:09:56 But you still got better performance,

1:09:58 and that might be one example of how the data helped.

1:10:01 It's mostly down to data quality.

1:10:02 I think if we had more compute, we would train for longer.

1:10:05 I think we'd ultimately see that as something we would want to do.

1:10:09 And especially with big models, you need more compute,

1:10:11 because we talked about having more parameters and we talked about knowledge.

1:10:15 Essentially, there's a ratio where big models can absorb more from data,

1:10:19 and then you get more benefit out of this.

1:10:22 Any logarithmic graph in your mind is like a small

1:10:25 model will level off sooner if you're measuring tons of tokens,

1:10:28 and bigger models need more.

1:10:30 But mostly, we aren't training that big of models right now at AI2,

1:10:34 and getting the highest quality data we can is the natural starting point.

1:10:38 Is there something to be said about the topic of data quality?

1:10:41 Is there some low-hanging fruit there still where the quality could be improved?

1:10:46 It's like turning the crank.

1:10:47 Historically, in the open,

1:10:49 there's been a canonical best pre-training dataset that has moved around

1:10:54 between who has the most recent one or the best recent effort.

1:10:56 Like AI2's Dolma was very early with the first OLMo,

1:10:59 and Hugging Face had FineWeb.

1:11:01 And there's a DCLM project,

1:11:02 which has been kind of like a, which stands for Data Comp Language Model.

1:11:07 There's been Data Comp for other machine learning projects,

1:11:10 and they had a very strong dataset.

1:11:12 And a lot of it is the internet is becoming fairly closed off,

1:11:17 so we have Common Crawl,

1:11:18 which is hundreds of trillions of tokens, and you filter it.

1:11:21 It looks like scientific work where you're training

1:11:24 classifiers and making decisions based on how you

1:11:27 prune down this dataset into the highest quality

1:11:30 stuff and the stuff that suits your tasks.

1:11:33 Previously, language models were tested a lot

1:11:35 more on knowledge and conversational things,

1:11:37 but now they're expected to do math and code.

1:11:40 To train a reasoning model, you need to remix your whole dataset.

1:11:43 And there's a lot of wonderful scientific methods here where you can,

1:11:47 you can take your gigantic dataset,

1:11:49 sample really tiny things from different sources,

1:11:52 such as GitHub, Stack Exchange, Reddit, Wikipedia.

1:11:56 You can sample small things from them,

1:11:57 and train small models on each of these mixes

1:12:00 and measure their performance on your evaluations.

1:12:02 You can just do basic linear regression, and it's like,

1:12:04 "Here's your optimal dataset." But if your evaluations change,

1:12:07 your dataset changes a lot.

1:12:08 So a lot of OLMo 3 was new sources for reasoning to be better at math and code,

1:12:13 and then you do this mixing procedure and it gives you the answer.

1:12:17 I think that's happened at labs this year;

1:12:19 there's new hot things, whether it's coding environments or web navigation,

1:12:23 and you need to bring in new data,

1:12:24 change your whole pre-training so that your post-training can work better.

1:12:28 And that's like the constant evolution and the redetermining

1:12:31 of what they care about for their models.

1:12:35 Are there fun anecdotes of what sources of data

1:12:38 are particularly high quality that we wouldn't expect?

1:12:41 You mentioned Reddit sometimes can be a source.

1:12:45 Reddit was very useful.

1:12:47 I think PDFs is definitely one.

1:12:51 Oh, especially arXiv.

1:12:52 Yeah, so AI2 has run Semantic Scholar for a long time,

1:12:56 which is what you can say is a competitor

1:12:59 to Google Scholar with a lot more features.

1:13:01 And to do this, AI2 has found and scraped a lot of PDFs for openly

1:13:06 accessible papers that might not be behind

1:13:09 the closed walled garden of a certain publisher.

1:13:11 So, truly open scientific PDFs.

1:13:13 And if you sit on all of these and you process it, you can get value out of it.

1:13:17 And I think that like,

1:13:19 a lot of that style of work has been done by the frontier labs did much earlier.

1:13:24 You just need to have a pretty

1:13:26 skilled researcher that understands how things change models;

1:13:30 they bring it in, clean it, and it's a lot of labor.

1:13:33 When frontier labs scale researchers, a lot more goes into data.

1:13:38 If you join a frontier lab and you want to have impact,

1:13:41 the best way to do it is just find new data that's better.

1:13:45 And then, the fancy, glamorous algorithmic things like figuring out how to make

1:13:50 o1 is like the sexiest thought of a scientist.

1:13:52 It's like, "Oh, I figured out how to scale RL." There's a group

1:13:55 that did that, but most of the contribution is like—- On the dataset-

1:13:58 ..."I'm gonna make the data better,"

1:14:00 or, "I'm gonna make the infrastructure better

1:14:01 so everyone on my team can run experiments 5% faster."- At the same time,

1:14:05 I think it's also one of the closest guarded secrets,

1:14:07 what your training data is, for legal reasons.

1:14:09 And so there's also, I think,

1:14:10 a lot of work that goes into hiding what your training data was essentially.

1:14:14 Like training the model to not give

1:14:17 away the sources because you have legal reasons.

1:14:19 The other thing, to be complete,

1:14:20 is that some people are trying to train on only licensed data,

1:14:23 whereas Common Crawl is a scrape of the whole internet.

1:14:26 So if I host multiple websites, I'm happy to have them train language models,

1:14:32 but I'm not explicitly licensing what governs it.

1:14:35 And therefore, Common Crawl is largely unlicensed,

1:14:38 which means that your consent really hasn't

1:14:41 been provided for how to use the data.

1:14:43 There's another idea where you can train language

1:14:44 models only on data that has been licensed explicitly,

1:14:47 so that the kind of governing contract is provided,

1:14:50 and I'm not sure if Apertus is the copyright thing or the license thing.

1:14:53 I know that the reason that they did it was for an EU compliance thing,

1:14:56 where they wanted to make sure that their model fit one of those checks.

1:15:05 On that note, there's also the distinction in licensing.

1:15:09 Some people just purchase the license.

1:15:12 Let's say they buy an Amazon Kindle book,

1:15:15 or a Manning book, and then use that in training.

1:15:18 That is a gray zone 'cause you paid

1:15:19 for the content and you might want to train on it.

1:15:22 But then there are also restrictions where even that shouldn't be allowed.

1:15:25 And so that is where it gets a bit fuzzy.

1:15:28 And yeah, I think that is right now still a hot topic.

1:15:33 Big companies like OpenAI approached private companies

1:15:36 for their proprietary data and private companies,

1:15:39 they become more and more, let's say,

1:15:42 protective of their data because they know, "Okay,

1:15:44 this is going to be my moat in a few

1:15:47 years." And I do think that's like the interesting question,

1:15:50 where if LLMs become more commoditized,

1:15:53 and I think a lot of people learn about LLMs,

1:15:56 there will be a lot more people able to train LLMs.

1:15:58 Of course, there are infrastructure challenges.

1:16:00 But if you think of big industries like pharmaceutical industries,

1:16:04 law, finance industries, I do think they, at some point,

1:16:07 will hire people from other frontier labs

1:16:10 to build their in-house models on their proprietary data,

1:16:13 which will be then, again,

1:16:14 another unlock with pre-training that is currently not there.

1:16:17 Because even if you wanted to, you can't get that data.

1:16:21 You can't get access to clinical trials most

1:16:23 of the time and these types of things.

1:16:25 So, I do think scaling, in that sense, might be still pretty much alive

1:16:28 if you also look in domain-specific applications,

1:16:31 because we are still right now, in this year,

1:16:33 just looking at general purpose LLMs on, on ChatGPT, Anthropic and so forth.

1:16:37 They are just general purpose, they're not even, I think,

1:16:40 scratching the surface of what an LLM can do if

1:16:43 it is really specifically trained and designed for a specific task.

1:16:47 I think on the data thing—this is one of the things that happened in 2025,

1:16:50 and we totally forget it—is Anthropic lost

1:16:52 in court and owed $1.5 billion to authors.

1:16:55 Anthropic, I think, bought thousands of books and scanned them

1:16:59 and was cleared legally for that because they bought the books,

1:17:03 and that is kind of going through the system.

1:17:04 And then the other side, they also torrented some books,

1:17:07 and I think this torrenting was the path where the court said

1:17:10 that they were then culpable to pay these billions of dollars to authors,

1:17:13 which is just such a mind-boggling lawsuit that kind of just came and went.

1:17:17 That is so much money-...

1:17:20 from the VC ecosystem.

1:17:22 These are court cases that will define the future of human civilization,

1:17:25 because it's clear that data drives a lot

1:17:27 of this, and there's this very complicated human tension of...

1:17:30 I mean, you can empathize.

1:17:33 You're both authors.

1:17:34 And there's some degree to which, I mean,

1:17:36 you put your heart and soul and your sweat

1:17:39 and tears into the writing that you do.

1:17:42 It feels a little bit like theft for somebody

1:17:46 to train your data without giving you credit.

1:17:49 And there are, like Nathan said, also two layers to it.

1:17:51 Someone might buy the book and then train on it,

1:17:54 which could be argued fair or not fair,

1:17:56 but then there are the straight-up companies who use

1:18:00 pirated books where it's not even compensating the author.

1:18:03 That is, I think, where people got

1:18:04 a bit angry about it specifically, I would say.

1:18:06 Yeah, but there has to be some kind of compensation scheme.

1:18:09 This is like moving towards-...

1:18:11 towards something like Spotify streaming did-...

1:18:13 originally for music.

1:18:14 You know, what does that-...

1:18:15 compensation look like?

1:18:16 You have to define those kinds of models.

1:18:17 You have to think through all of that.

1:18:19 One other thing I think people are generally curious about,

1:18:22 I'd love to get your thoughts, as LLMs are used more and more.

1:18:26 If you look at even arXiv, but GitHub,

1:18:29 more and more of the data is generated by LLMs.

1:18:32 What do you do in that kind of world?

1:18:36 How big of a problem is that?

1:18:39 Largest problem's the infrastructure and systems,

1:18:41 but from an AI point of view, it's kind of inevitable.

1:18:45 So it's basically LLM-generated data

1:18:47 that's curated by humans essentially, right?

1:18:49 Yes, and I think that a lot

1:18:51 of open source contributors are legitimately burning out.

1:18:53 If you have a popular open source repo,

1:18:55 somebody's like, "Oh, I want to do open source AI.

1:18:57 It's good for my career," and they just

1:19:00 vibe- -code something and they throw it in.

1:19:02 You might get more of this-- I have a--- than I do.

1:19:05 Yeah, so I have actually a case study here.

1:19:09 I have a repository called MLxtend that I

1:19:11 developed as a student around 10 years ago,

1:19:14 and it is a reasonably popular library still for certain algorithms,

1:19:18 I think especially like frequent data mining stuff.

1:19:22 And there were recently two or three people who submitted

1:19:25 a lot of PRs in a very short amount of time.

1:19:28 I do think LLMs have been involved in submitting these PRs.

1:19:31 Me, as the maintainer, there are two things.

1:19:33 First, I'm a bit overwhelmed.

1:19:35 I don't have time to read through it because,

1:19:37 especially as an older library, that is not a priority for me.

1:19:40 At the same time, I kind of also appreciate it because

1:19:43 I think something people forget is it's not just using the LLM.

1:19:46 There's still a human layer that verifies something,

1:19:48 and that is in a sense also how data is labeled, right?

1:19:53 One of the most expensive things is getting

1:19:56 labeled data for RL from human feedback phases.

1:19:59 And this is kind of like that, where it goes through phases,

1:20:03 and then you actually get higher quality data out of it.

1:20:06 And so I don't mind it in a sense.

1:20:08 It can feel overwhelming, but I do think there is also value in it.

1:20:12 It feels like there's a fundamental

1:20:14 difference between raw LLM-generated data and LLM-generated

1:20:16 data with a human in the loop that does some kind of verification,

1:20:21 even if that verification is a small percent of the lines of code.

1:20:26 I think this goes with anything where people think also sometimes, "Oh, yeah.

1:20:31 I can just use an LLM to learn about XYZ," which is true.

1:20:34 You can, but there might be a person who is

1:20:37 an expert who might have used an LLM to write specific code.

1:20:41 There is kind of like this human work that went into it to make it nice,

1:20:45 throwing out the not-so-nice parts to kind of pre-digest it for you,

1:20:50 and that saves you time.

1:20:52 I think that's the value-add,

1:20:53 where you have someone filtering things or even using the LLMs correctly.

1:20:59 This is still labor that you get for free.

1:21:02 For example, if you read a Substack article,

1:21:04 I could maybe ask an LLM to give me opinions

1:21:07 on that, but I wouldn't even know what to ask.

1:21:10 And I think there is still value in reading that article

1:21:13 compared to me going to the LLM because you are the expert.

1:21:17 You select what knowledge is actually spot on, should be included,

1:21:20 and you give me this very...

1:21:23 this executive summary.

1:21:24 And this is a huge value-add because now I don't have

1:21:28 to waste three to five hours to go through this myself,

1:21:31 maybe get some incorrect information and so on.

1:21:34 And so I think that's also where the future

1:21:37 still is for writers even though there are LLMs that...

1:21:41 Can kind of save you time.

1:21:44 It's kinda fascinating to watch.

1:21:45 I'm sure you guys do this, but for me,

1:21:48 I look at the difference between a summary and the original content.

1:21:53 Even if it's a page-long summary of a page-long content,

1:21:57 it's interesting to see how the LLM-based summary takes the edge off.

1:22:03 What is the signal it removes from the thing?

1:22:07 The voice is what I talk about a lot.

1:22:09 Voice?

1:22:10 Well, voice...

1:22:10 I would love to hear what you mean by voice,

1:22:13 but sometimes there's literally insights.

1:22:16 By removing an insight, you're changing the meaning of the thing.

1:22:21 So I'm continuously disappointed how bad LLMs

1:22:24 are at really getting to the core insights, which is what a great summary does.

1:22:31 Yet even if I have these extremely elaborate prompts

1:22:36 where I'm really trying to dig for the insights,

1:22:40 it's still not quite there, which...

1:22:42 I mean, that's a whole deep philosophical question about what is human knowledge

1:22:46 and wisdom and what does it mean to be insightful and so on.

1:22:49 But when you talk about the voice, what do you mean?

1:22:52 When I write, I think a lot of what I'm trying to do

1:22:55 is take what you think as a researcher, which is very raw.

1:22:59 A researcher is trying to encapsulate an idea at the frontier

1:23:02 of their understanding and they're trying to put what is a feeling into words.

1:23:07 I try to do this in my writing,

1:23:11 which makes it come across as raw but also high-information

1:23:14 in a way that some people will get it and some won't.

1:23:17 And that's the nature of research.

1:23:19 And language models don't do this well.

1:23:21 They're all trained with Reinforcement Learning from Human Feedback,

1:23:25 which takes feedback from many people

1:23:27 and averages how the model behaves from this.

1:23:30 And I think it's going to be hard for a model

1:23:34 to be very incisive when there's that sort of filter.

1:23:37 This is a wonderful fundamental problem for researchers in RLHF.

1:23:43 This provides so much utility in making the models better,

1:23:47 but also the problem formulation is kind of...

1:23:51 there's this knot in it that you can't get past.

1:23:55 These language models don't have this prior

1:23:57 in their deep expression they're trying to get at.

1:24:00 I don't think it's impossible.

1:24:01 There are stories of models that really shock people.

1:24:04 Like, I think of...

1:24:05 I would love to have tried Bing Sydney.

1:24:08 Did that have more voice?

1:24:10 Because it would so often go off the rails,

1:24:13 which is historically obviously a scary way—like telling a reporter to leave

1:24:17 his wife—is a crazy model to potentially put in general adoption.

1:24:21 But that's kind of like a trade-off;

1:24:23 is this RLHF process in some ways adding limitations?

1:24:28 That's a terrifying place to be as one of these frontier labs and companies,

1:24:33 because millions of people are using them.

1:24:36 There was a lot of backlash last year with GPT-4o getting removed,

1:24:39 and I've personally never used the model,

1:24:41 but I've talked to people at OpenAI where they get emails from users

1:24:47 that might be detecting subtle differences

1:24:50 in the deployments in the middle of the night.

1:24:52 And they email them like, "My friend is different." And they find

1:24:56 these employees' emails and send them things because they

1:24:58 are so attached to this set of model

1:25:02 weights and configuration that is deployed to the users.

1:25:05 We see this with TikTok.

1:25:06 You open it...

1:25:07 I don't use TikTok, but supposedly in five minutes the algorithm gets you.

1:25:11 It's locked in.

1:25:12 And those are language models doing recommendations.

1:25:15 I think there are ways you can do this.

1:25:18 Within five minutes of chatting with it, the model just gets you.

1:25:21 And that is something that people aren't really ready for.

1:25:26 Like, don't give that to kids.

1:25:28 At least until we know what's happening.

1:25:30 But there's also going to be this mechanism...

1:25:32 What's going to happen with these LLMs as they're used more and more...

1:25:36 Unfortunately, the nature of the human

1:25:37 condition is such that people commit suicide.

1:25:39 And so what journalists will do is they

1:25:42 will report extensively on the people who commit suicide.

1:25:45 And they would very likely link it to the LLMs

1:25:48 because they have that data about the conversations.

1:25:50 If you're really struggling in your life, if you're depressed,

1:25:54 if you're thinking about suicide,

1:25:55 you're going to probably talk to LLMs about it.

1:25:58 And so what journalists will do is say, "Well,

1:26:01 the suicide was committed because of the LLM."

1:26:03 And that's going to lead to the companies,

1:26:06 because of legal issues and so on, more and more taking the edge off of the LLM.

1:26:13 So it's going to be as generic as possible.

1:26:15 It's so difficult to operate in this space because you don't

1:26:19 want an LLM to cause harm to humans at that level,

1:26:23 but also this is the nature of the human experience,

1:26:27 is to have a rich conversation,

1:26:29 a fulfilling conversation, one that challenges you from which you grow.

1:26:33 You need that edge.

1:26:35 And that's something extremely difficult for AI researchers

1:26:39 on the RLHF front to actually have to solve,

1:26:44 because you're dealing with the human condition.

1:26:47 A lot of researchers at these companies are so well-motivated,

1:26:50 and definitely Anthropic and OpenAI culturally want to do good for the world.

1:26:56 And it's such a—I'm like, "Ooh,

1:26:58 I don't wanna work on this," because on the one hand,

1:27:01 a lot of people see AI as a health ally,

1:27:04 as somebody they can talk to about their health confidentially,

1:27:08 but then it bleeds all the way into this, like talking about mental health,

1:27:14 where it's heartbreaking that this will be

1:27:17 the thing where somebody goes over the edge, but other people might be saved.

1:27:21 And I'm like, "I don't..." As a researcher, it's like,

1:27:25 I don't want to train image generation

1:27:26 models and release them openly because I don't

1:27:29 want to enable somebody to have a tool

1:27:31 on their laptop that can harm other people.

1:27:34 I don't have the infrastructure in my company to do that safely.

1:27:37 But there's a lot of areas like this where it

1:27:40 just needs people that will approach it with complexity and conviction.

1:27:44 It's just such a hard problem.

1:27:48 But also, we as a society, as users of these technologies,

1:27:50 need to make sure that we're having the complicated conversation about it versus

1:27:54 just fearmongering— that Big Tech is causing

1:27:57 harm to humans or stealing your data.

1:27:59 It's more complicated than that.

1:28:01 And you're right.

1:28:03 There's a very large number of people inside these companies,

1:28:05 many of whom I know, who deeply care about helping people.

1:28:09 They are considering the full human experience of people from across the world,

1:28:13 not just Silicon Valley.

1:28:14 People across the United States and the world, what their needs are.

1:28:18 It's really difficult to design this one system that is able

1:28:23 to help all these different kinds of people across different age groups,

1:28:26 cultures, and mental states.

1:28:31 I wish that the timing of AI was different relative

1:28:34 to the relationship of Big Tech to the average person.

1:28:37 Big Tech's reputation was so low, and with how AI is so expensive,

1:28:40 it's inevitably going to be a Big Tech thing.

1:28:42 It takes so many resources, and people say the US is,

1:28:46 "betting the economy on AI" with this build-out.

1:28:48 To have these be intertwined at the same

1:28:51 time makes for such a hard communication environment.

1:28:54 It would be good for me to go talk to more people

1:28:56 in the world who hate Big Tech and see AI as a continuation of this.

1:29:03 And one of the things you recommend,

1:29:05 one of the antidotes that you talk about, is to find agency in this system.

1:29:11 As opposed to sitting back in a powerless way and consuming

1:29:16 the AI slop as it rapidly takes over the internet.

1:29:20 Find agency by using AI to build stuff, build apps, build...

1:29:25 One, that actually helps you build intuition, but two,

1:29:29 it's empowering because you can understand how it works,

1:29:32 what the weaknesses are.

1:29:34 It gives your voice power to say, "This is a bad use of the technology,

1:29:38 and this is a good use." And you're more plugged into the system then,

1:29:44 so you can understand it better and you can steer it better as a consumer.

1:29:48 I think that's a good point you brought up about agency.

1:29:51 Instead of ignoring it and saying, "Okay,

1:29:52 I'm not going to use it," I think it's probably long-term healthier to say,

1:29:57 "Okay, it's out there.

1:29:58 I can't put it back." when they came out.

1:30:01 How do I make best use of it, and how does it help me to up-level myself?

1:30:06 The one thing I worry about here, though, is,

1:30:08 if you just fully use it for something you love to do,

1:30:10 the thing you love to do is no longer there.

1:30:13 And that could potentially, I feel, lead to burnout.

1:30:16 For example, if I use an LLM to do all my coding for me, now there's no coding.

1:30:21 I'm just managing something that is coding for me.

1:30:23 Two years later, let's say, if I just do that eight hours a day,

1:30:27 having something code for me, do I feel fulfilled still?

1:30:36 Is this hurting me in terms of being excited about my job,

1:30:39 excited about what I'm doing?

1:30:40 Am I still proud to build something?

1:30:43 On that topic of enjoyment, it's quite interesting.

1:30:45 We should just throw this in there,

1:30:47 that there's this recent survey of about 791 professional developers,

1:30:52 meaning 10-plus years of experience.

1:30:56 That's a long time.

1:30:58 As a junior developer?

1:31:01 Yeah, in this day and age.

1:31:03 So, there's also many fronts that are surprising.

1:31:06 They break it down by junior and senior developers.

1:31:10 But, I mean, it just shows that both junior

1:31:14 and senior developers use AI-generated code in code they ship.

1:31:22 So this is not just for fun or intermediate learning things.

1:31:26 This is code they ship.

1:31:28 25%—like, most of them use around 50% or more.

1:31:32 And what's interesting is,

1:31:33 for the category of over 50% of your code that you ship is AI-generated,

1:31:38 senior developers are much more likely to do so.

1:31:42 But you don't want AI to take away the thing you love.

1:31:46 I think this speaks to my experience, these results I'm about to say.

1:31:49 Together, about 80% of people find it either somewhat more enjoyable

1:31:54 or significantly more enjoyable to use AI as part of the work.

1:31:59 I think it depends on the task.

1:32:01 From my personal usage, for example,

1:32:03 I have a website where I sometimes tweak things.

1:32:07 I personally don't enjoy this.

1:32:09 So in that sense, if the AI can help me

1:32:13 to implement something on my website, I'm all for it.

1:32:15 It's great.

1:32:16 But then, at the same time,

1:32:17 when I solve a complex problem— well, if there's a bug,

1:32:21 and I hunt this bug, and I find it, it's the best feeling in the world.

1:32:26 You feel great.

1:32:27 But now, if you don't even think about the bug,

1:32:31 you just go directly to the LLM, well,

1:32:33 you never have this kind of feeling, right?

1:32:35 But then there could be the middle ground where, well,

1:32:39 you try yourself, you can't find it, you use the LLM,

1:32:42 and then you don't get frustrated because it helps

1:32:44 you and you move on to something that you enjoy.

1:32:46 And so, looking at these statistics, I think what is not factored in is

1:32:51 that it's averaging over all the different scenarios.

1:32:54 We don't know if it's for the core task or if

1:32:59 it's for something mundane that people would not have enjoyed otherwise.

1:33:02 So, in a sense, AI is really great

1:33:04 for doing mundane things that take a lot of work.

1:33:06 For example, my wife the other day—she has

1:33:09 a podcast for book discussions, a book club,

1:33:13 and she was transferring the show notes from Spotify to YouTube,

1:33:19 and then the links somehow broke.

1:33:21 And she had in some episodes, because it is so many books, like 100 links,

1:33:25 and it would have been really painful to go in there and fix each link manually.

1:33:29 So I suggested, "Hey,

1:33:30 let's try ChatGPT." We copied the text into ChatGPT, and it fixed them.

1:33:35 Instead of two hours going from link to link fixing

1:33:39 that, it made that type of work much more seamless.

1:33:43 I think everyone has a use case where AI is

1:33:47 useful for something that would be really boring, really mundane.

1:33:51 For me personally, since we're talking about coding,

1:33:56 and you mentioned debugging...

1:33:58 the source of enjoyment for me, more on the Cursor side than Claude Code,

1:34:02 is that I have a friend, I have a pair programmer.

1:34:08 It's less lonely.

1:34:10 You made debugging sound like this great joy.

1:34:15 No, I would say debugging is like a drink

1:34:18 of water after you've been going through a desert for days.

1:34:22 You skip the whole desert part where you're suffering.

1:34:27 Sometimes it's nice to have a friend who can't really find the bug,

1:34:31 but can give you some intuition about the code,

1:34:35 and together you're going through the desert and finding that drink of water.

1:34:40 At least for me, maybe it speaks

1:34:43 to the loneliness of the programming experience.

1:34:45 That is a source of joy.

1:34:48 It's maybe also related to delayed gratification.

1:34:51 I'm a person who, even as a kid,

1:34:54 I liked the idea of Christmas presents—having them,

1:34:57 getting them—better than actually receiving the presents.

1:35:01 I would look forward to the day I get the presents,

1:35:04 but then it's over and I'm disappointed.

1:35:06 And maybe it's the same with food.

1:35:08 I think food tastes better when you're really hungry.

1:35:12 You're right, with debugging, it is not always great.

1:35:17 It's often frustrating, but then if you can solve it, then it's great.

1:35:23 But there's a sweet Goldilocks zone;

1:35:24 if it's too hard, then it's just wasting your time.

1:35:28 But I think another challenge is how will people learn?

1:35:33 We looked at the chart and saw that more senior

1:35:37 developers are shipping more AI-generated code than the junior ones.

1:35:42 It's very interesting,

1:35:42 because intuitively you would think it's the junior developers

1:35:45 because they don't know how to do the thing yet,

1:35:49 and so they use AI to do that thing.

1:35:51 It could mean the AI is not good enough yet to solve that task,

1:35:55 but it could also mean experts are more effective at using it.

1:35:59 They know how to use it better,

1:36:02 review the code, and then they trust the code more.

1:36:06 One issue for society in the future will be:

1:36:08 how do you become an expert if you never try to do the thing yourself?

1:36:14 One way I always learned is by trying things myself.

1:36:18 If you look at math textbooks and the solutions,

1:36:21 you learn something, but you learn actually better if you try first.

1:36:27 Then you appreciate the solution differently because you

1:36:29 know how to put it into your mental framework.

1:36:32 If LLMs are here all the time,

1:36:35 would you actually go to the length of struggling?

1:36:38 Would you be willing to struggle?

1:36:40 Struggle is not nice, right?

1:36:42 But if you use the LLM to do everything,

1:36:44 at some point you will never really take the next step,

1:36:47 and then you will maybe not get that unlock

1:36:50 that you would get as an expert using an LLM.

1:36:52 So, I think there's like a Goldilocks sweet spot where maybe the trick

1:36:58 here is you make dedicated offline time where you study two hours a day,

1:37:01 and the rest of the day use LLMs.

1:37:03 But I think it's important also for people to still invest in themselves,

1:37:07 in my opinion, to not just LLM everything.

1:37:11 Yeah, as a civilization, we each individually have to find that Goldilocks zone.

1:37:16 And in the programming context as developers.

1:37:18 Now, we've had this fascinating conversation

1:37:20 that started with pre-training and mid-training.

1:37:24 Let's get to post-training.

1:37:25 A lot of fun stuff in post-training.

1:37:28 So, what are some of the interesting ideas in post-training?

1:37:32 The biggest one from 2025 is

1:37:34 learning this reinforcement learning with verifiable rewards.

1:37:37 You can scale up the training there,

1:37:39 which means doing a lot of this kind of iterative generate-grade loop,

1:37:43 and that lets the models learn both interesting

1:37:47 behaviors on the tool use and software side.

1:37:49 This could be searching, running commands on their own and seeing the outputs,

1:37:53 and then also that training enables this inference time scaling very nicely.

1:37:57 And it just turned out that this paradigm was very nicely linked,

1:38:02 where this kind of RL training enables inference time scaling.

1:38:05 But inference time scaling could have been found in different ways.

1:38:07 So, it was kind of this perfect storm where the models change a lot,

1:38:10 and the way that they're trained is a major factor in doing so.

1:38:15 And this has changed how people approach post-training dramatically.

1:38:20 Can you describe RLVR, popularized by DeepSeek R1?

1:38:23 Can you describe how it works?

1:38:26 Yeah.

1:38:26 Fun fact: I was on the team that came up with the term RLVR,

1:38:29 which is from our Tulu 3 work before DeepSeek.

1:38:33 We don't take a lot of credit for being the people to popularize the scaling RL,

1:38:37 but as fun as what academics get,

1:38:40 as an aside, is the ability to name and influence the discourse,

1:38:44 because the closed labs can only say so much.

1:38:47 That one of the things you can do as an academic

1:38:49 is you might not have the compute to train the model,

1:38:51 but you can frame things in a way that ends up being described

1:38:55 as a community coming together around this RLVR term, which is very fun.

1:39:00 And then DeepSeek are the people that did the training breakthrough,

1:39:03 which is, they scaled the reinforcement learning.

1:39:06 You have the model generate answers and then

1:39:09 grade the completion if it was right,

1:39:11 and then that accuracy is your reward for reinforcement learning.

1:39:16 So reinforcement learning is classically an agent that acts in an environment,

1:39:20 and the environment gives it a state and a reward back,

1:39:24 and you try to maximize this reward.

1:39:26 In the case of language models,

1:39:28 the reward is normally accuracy on a set of verifiable tasks,

1:39:31 whether it's math problems or coding tasks.

1:39:34 And it starts to get blurry with things like factual domains.

1:39:38 That is also, in some ways, verifiable, or constraints on your instruction,

1:39:44 like respond only with words that start with A." All

1:39:47 of these things are verifiable in some way,

1:39:50 and the core idea of this is you find a lot more of these problems

1:39:55 that are verifiable and you let the model

1:39:56 try it many times while taking these RL steps, these RL gradient updates.

1:40:02 The infrastructure evolved from reinforcement learning from human feedback,

1:40:06 where in that era the score they were trying

1:40:10 to optimize was a learned reward model of human preferences.

1:40:13 So you kind of changed the problem domains

1:40:15 and that let the optimization go on to much bigger scales,

1:40:19 which kind of kickstarted a major change in what

1:40:22 the models can do and how people use them.

1:40:25 What kind of domains is RLVR amenable to?

1:40:28 Math and code are the famous ones,

1:40:30 and then there's a lot of work on what is called the rubrics,

1:40:34 which is related to a word people might have heard: LLM-as-a-judge.

1:40:38 For each problem, I'll have a set of problems in my dataset.

1:40:42 I will then have an LLM and ask it,

1:40:45 "What would a good answer to this problem look like?" And then you could

1:40:49 try the problem over and over again and assign a score based on this rubric.

1:40:54 That's not necessarily verifiable like math and code domains,

1:40:57 but this rubrics idea and other scientific problems that might

1:41:00 be a little bit more vague is where the attention is,

1:41:04 where they're trying to push this set of methods into these kind

1:41:07 of more open-ended domains so the models can learn a lot more.

1:41:11 I think that's called reinforcement learning with AI feedback, right?

1:41:14 That's the older term for it coined in Anthropic's Constitutional AI paper.

1:41:18 It's like a lot of these things come in cycles.

1:41:21 Also, just one step back for RLVR.

1:41:24 I think the interesting thing here is that you

1:41:27 ask the LLM a, let's say, math question, and then you know the correct answer,

1:41:31 and you let the LLM, as you said, figure it out.

1:41:35 How it does it—you don't constrain it much.

1:41:37 There are some constraints like "use the same language,

1:41:40 don't switch between Spanish and English."

1:41:42 But let's say you're pretty much hands-off.

1:41:45 You only give the question and the answer,

1:41:47 and then the LLM has the task to arrive at the right answer,

1:41:51 but the beautiful thing here is what happens in practice:

1:41:55 the LLM will do a step-by-step description,

1:41:57 like as a student or as a mathematician would derive the solution.

1:42:02 It will use those steps, and that helps the model to improve its own accuracy.

1:42:07 And then, like you said, the inference scaling.

1:42:11 Inference scaling loosely means spending more compute during inference,

1:42:16 and here the inference scaling is that the model would use more tokens.

1:42:22 In the DeepSeek R1 paper,

1:42:24 they showed the longer they train the model, the longer the responses are.

1:42:28 They grow over time.

1:42:29 They use more tokens, so it becomes more expensive.

1:42:32 It becomes expensive for simple tasks, but these explanations help accuracy.

1:42:36 There are also papers showing what the model explains does not

1:42:40 necessarily have to be correct or maybe it's unrelated to the answer,

1:42:44 but for some reason, it still helps the model that it is explaining.

1:42:48 And I think it's also—again, I don't want to anthropomorphize these LLMs,

1:42:52 but it's kind of like how we humans operate.

1:42:55 If there's a complex math problem in a math class, class,

1:42:59 you usually have a note paper and you do it step by step.

1:43:02 You cross out things.

1:43:03 And the model also self-corrects, and that was,

1:43:05 I think, the aha moment in the DeepSeek R1 paper.

1:43:08 They called it the aha moment because the model

1:43:10 itself recognized it made a mistake and then said, "Ah, I did something wrong,

1:43:13 let me try again." And I think that's just so cool that this falls out

1:43:18 of just giving it the correct answer and having it figure out how to do it,

1:43:22 that it kind of does in a sense what a human would do.

1:43:26 Although LLMs don't think like humans,

1:43:28 it's kind of like an interesting coincidence and it...

1:43:31 And the other nice side effect is it's

1:43:33 great for us humans often to see these steps.

1:43:36 It builds trust, but also we us humans to see these steps.

1:43:38 It builds trust, but also we learn and can double check things.

1:43:40 There's a lot in here.

1:43:40 I think some of the debate...

1:43:42 There's been a lot of debate this year on if the language models like these...

1:43:45 I think the aha moments are kind of fake

1:43:48 because in pre-training you essentially have seen the whole internet.

1:43:51 so you have definitely seen people explaining their work,

1:43:54 even verbally, like a transcript of a math lecture.

1:43:57 "You try this, oh, I messed this up."

1:43:58 And what RLVR is very good at doing is amplifying

1:44:02 these behaviors because they're very useful in enabling

1:44:04 the model to think longer and to check its work.

1:44:07 And I agree that it is very beautiful that this training kind of...

1:44:11 The model learns to amplify this in a way

1:44:13 that is so useful for the final answers being better.

1:44:17 I can give you also a hands-on example.

1:44:18 I was training the Qwen 3 base model with RLVR on MATH-500.

1:44:22 The base model had an accuracy of about 15%.

1:44:26 Just 50 steps, like in a few minutes with RLVR,

1:44:30 the model went from 15% to 50% accuracy.

1:44:33 And the model...

1:44:34 You can't tell me it's learning anything fundamentally about math

1:44:38 in- The Qwen example is weird because there've been two papers this year,

1:44:41 one of which I was on, about data contamination in Qwen

1:44:44 and specifically that they train on a lot of this special

1:44:47 mid-training phase that we should take a minute on, because it's

1:44:50 weird because they train on problems that are almost identical to MATH.

1:44:53 Exactly.

1:44:54 And so you can see that basically the RL,

1:44:57 it's not teaching the model any new knowledge about math.

1:44:59 You can't do that in 50 steps.

1:45:01 So the knowledge is already there,

1:45:02 in the pre-training, you're just unlocking it.

1:45:04 I still disagree with the premise because there's a lot

1:45:06 of weird complexities that you can't prove because one

1:45:10 of the things that points to weirdness is that if

1:45:12 you take the Qwen 3 so-called base model and you...

1:45:15 You could Google like "math dataset, Hugging Face",

1:45:18 and you could take a problem and what you do if you put it into Qwen 3 base...

1:45:23 All these math problems have words,

1:45:24 so it'd be like "Alice has five apples and takes one...

1:45:27 and gives three to whoever," and there are these word problems.

1:45:30 With these Qwen-based models, why people are suspicious of them is if you

1:45:34 change the numbers but keep the words- Qwen will produce,

1:45:38 without tools, will produce a very

1:45:41 high accuracy decimal representation of the answer, which means there's some...

1:45:45 At some time, it was shown problems that were almost identical to the test set,

1:45:50 and it was using tools to get a very high precision answer,

1:45:53 but a language model without tools will never actually have this.

1:45:57 So it's kind of been this big debate in the research community:

1:46:01 how much of these reinforcement learning papers that are

1:46:04 training on Qwen and measuring specifically on this math benchmark,

1:46:07 where there's been multiple papers talking about contamination,

1:46:10 is like, how much can you believe them?

1:46:12 And I think this is what caused the reputation of RLVR being about formatting,

1:46:15 because you can get these gains so quickly,

1:46:18 therefore it must already be in the model.

1:46:20 But there's a lot of complexity here that we...

1:46:22 It's not really like controlled experimentation, so we don't really know.

1:46:27 But if it weren't true, I would say distillation wouldn't work, right?

1:46:30 I mean, distillation can work to some extent, but the thing is that is, I think,

1:46:35 the biggest problem,

1:46:36 and I research this contamination because we don't know what's in the data.

1:46:38 Unless you have a new dataset, it is really impossible.

1:46:42 And the same, you mentioned the math dataset,

1:46:44 where you have a question and then answer and an explanation is given,

1:46:48 but then also even something simpler like MMLU,

1:46:51 which is a multiple-choice benchmark.

1:46:53 If you just change the format slightly, like, I don't know,

1:46:57 if you use a dot instead of a parenthesis

1:47:00 or something like that, the model accuracy will vastly differ.

1:47:05 I think that that could be like a model issue rather than a general issue.

1:47:09 It's not even malicious by the developers of the LLM, like, "Hey,

1:47:11 we want to cheat at that benchmark." It has seen something at some point.

1:47:14 I think the only fair way to evaluate an LLM is to have

1:47:18 a new benchmark that is after the cutoff date when the LLM was deployed.

1:47:22 Can we lay out what would be the recipe

1:47:25 of all the things that go into post-training?

1:47:28 And you mentioned RLVR was a really exciting, effective thing.

1:47:32 Maybe we should elaborate.

1:47:34 RLHF still has a really important component to play.

1:47:37 What kind of other ideas are there on post-training?

1:47:40 I think you can kind of take this in order.

1:47:41 I think you could view it as what made o1,

1:47:45 which is this first reasoning model, possible, or what will the latest model be?

1:47:49 And they actually...

1:47:51 You're going to have similar interventions

1:47:53 at these, where you start with mid-training,

1:47:56 and the thing that is rumored to enable

1:47:59 o1 and similar models is really careful data curation,

1:48:02 where you're providing a broad set of what is called reasoning traces,

1:48:07 which is just the model generating words

1:48:10 in a forward process that is reflecting,

1:48:13 like breaking down a problem into intermediate steps and trying to solve them.

1:48:16 So at mid-training, you need to have data that is similar

1:48:20 to this to make it so that when you move into post-training,

1:48:23 primarily with these verifiable rewards, it can learn.

1:48:27 And then what is happening today is you're figuring out which problems

1:48:32 to give the model and how out which problems to give the model

1:48:35 and how long you can train it for and how much inference

1:48:37 you can enable the model to use when solving these verifiable problems.

1:48:41 So as models get better,

1:48:43 certain problems models get better, certain problems are no longer...

1:48:47 The model will solve them 100% of the time,

1:48:48 and therefore there's very little signal in this.

1:48:51 If we look at the GRPO equation, this one is famous for this because essentially

1:48:55 the reward given to the agent is based on how

1:48:59 good a given action—an action is a completion—is

1:49:02 relative to the other answers to that same problem.

1:49:05 So if all the problems get the same answer,

1:49:07 there's no signal in these types of algorithms.

1:49:08 So what they're doing is they're finding harder problems,

1:49:11 which is why you hear about things like scientific domains,

1:49:14 where it's so hard to get anything right.

1:49:17 If you have a lab or something,

1:49:19 it just generates so many tokens or much harder software problems.

1:49:22 So the frontier models are all pushing into these harder domains when they

1:49:26 can train on more problems and the model will learn more skills at once.

1:49:30 The RLHF link to this is that RLHF has been

1:49:33 and still is kind of like the finishing touch on the models,

1:49:36 where it makes the models more useful

1:49:38 by improving the organization or style or tone.

1:49:41 There are different things that resonate with different audiences,

1:49:43 like some people like a really quirky model

1:49:45 and RLHF could be good at enabling that personality,

1:49:49 and some people hate this markdown bulleted list thing that the models do,

1:49:54 but it's actually really good for quickly parsing information.

1:49:57 In RLHF, this human feedback stage is really great for putting

1:50:02 this into the model at the end of the day.

1:50:05 It's what made ChatGPT so magical for people.

1:50:07 And that use has actually remained fairly stable.

1:50:10 This formatting can also help the models

1:50:14 get better at math problems, for example.

1:50:16 So it's like the border between style and formatting,

1:50:20 and like the method that you use to answer a problem is

1:50:24 actually all very closely linked in terms of when you're training these models,

1:50:29 which is why RLHF can still make a model better at math,

1:50:32 but these verifiable domains are a much more direct process

1:50:35 to doing this because it makes more sense with the problem formulation,

1:50:39 which is why it ends up all forming together.

1:50:42 But to summarize, it's like mid-training is give

1:50:44 the model the skills it needs to then learn.

1:50:47 RL with verifiable rewards is letting the model try a lot of times,

1:50:52 so put a lot of compute into trial-and-error learning across hard problems.

1:50:55 And then RLHF would be like finishing the model,

1:50:57 making it easy to use and kind of just rounding the model out.

1:51:02 Can you comment on the amount of compute required for RLVR?

1:51:06 It's only gone up and up.

1:51:08 I think Ilya Sutskever was famous for saying they

1:51:10 use a similar amount of compute for pre-training and post-training.

1:51:12 Back to the scaling discussion,

1:51:15 they involve very different hardware for scaling.

1:51:17 Pre-training is very compute-bound, which is like this FLOPs discussion,

1:51:20 which is just how many matrix multiplications can you get through at once.

1:51:24 And because with RL you're generating these answers,

1:51:26 you're trying the model in real-world environments,

1:51:29 it ends up being much more memory-bound because

1:51:31 you're generating long sequences and the attention mechanisms

1:51:35 have this behavior where you get a quadratic

1:51:38 increase in memory as you're getting to longer sequences.

1:51:42 So the compute becomes very different.

1:51:43 In pre-training we would talk about a model—if we

1:51:46 go back to like the Biden administration executive order,

1:51:48 it's like 10 to the 25th FLOPs to train a model.

1:51:51 If you're using FLOPs in post-training,

1:51:53 it's a lot weirder because the reality is just like:

1:51:56 how many hours are you allocating?

1:51:58 How many GPUs for?

1:51:59 And I think in terms of time, the RL compute is getting much closer because

1:52:04 you just can't put it all into one system.

1:52:06 Pre-training is so computationally dense where all the GPUs

1:52:09 are talking to each other and it's extremely efficient,

1:52:11 whereas RL has all these moving parts and can take

1:52:13 a long time to generate a sequence of 100,000 tokens.

1:52:17 If you think about GPT-5.2 Pro taking an hour, it's like,

1:52:20 what if your training run has a sample for an hour

1:52:23 and you have to make sure that's handled efficiently?

1:52:25 So I think in GPU hours or just wall-clock hours,

1:52:29 the RL runs are probably approaching the same number of days as pre-training,

1:52:32 but they probably aren't using as many GPUs at the same time.

1:52:36 There are rules of thumb where in labs you don't want

1:52:40 your pre-training runs to last more

1:52:41 than a month because they fail catastrophically.

1:52:43 And if you are planning a huge cluster to be

1:52:46 held for two months and then it fails on day 50,

1:52:49 the opportunity costs are just so big.

1:52:52 So people don't want to put all their eggs in one basket.

1:52:56 GPT-4 was the ultimate YOLO run, and nobody ever wanted to do it before,

1:53:01 where it took three months to train and everybody was shocked that it worked.

1:53:04 I think people are a little bit more cautious and incremental now.

1:53:07 So RLVR is more, let's say,

1:53:10 unlimited in how much you can train or still get benefit, where RLHF,

1:53:14 because it's a preference tuning,

1:53:15 you reach a certain point where it doesn't really

1:53:17 make sense to spend more RL budget on that.

1:53:20 So just a step back with preference tuning:

1:53:22 there are multiple people that can give multiple, let's say,

1:53:26 explanations for the same thing and they can both be correct,

1:53:29 but at some point you learn a certain style

1:53:31 and it doesn't make sense to iterate on it.

1:53:34 My favorite example is: if relatives ask me what laptop they should buy,

1:53:38 I give them an explanation or ask,

1:53:41 "What is your use case?" They, for example, prioritize battery life and storage.

1:53:46 Other people like us, for example, we would prioritize RAM and compute.

1:53:50 Both answers are correct, but different people require different answers.

1:53:55 With preference tuning, you're trying to average somehow.

1:53:58 You are asking the data labelers to give you, not the right,

1:54:02 but the preferred answer and then you train on that.

1:54:04 But at some point you learn that average preferred answer.

1:54:07 And there's no reason to keep training longer on it because it's just a style,

1:54:13 whereas with RLVR, you let the model

1:54:16 solve more and more complex, difficult problems.

1:54:19 So I think it makes more sense to allocate more budget long-term to RLVR.

1:54:25 Also, right now we are in an RLVR 1.0 blend where

1:54:31 it's still that simple thing where we have a question and answer,

1:54:34 but we don't do anything with the stuff in between.

1:54:38 There were multiple research papers, also by Google for example,

1:54:42 on process reward models that also give

1:54:44 scores for the explanation—how correct is the explanation.

1:54:47 And I think that will be the next thing, let's say RLVR 2.0 for this year,

1:54:53 focusing in between question and answer, like how to leverage that information,

1:54:58 the explanation, to help it get better accuracy.

1:55:01 So that's one angle.

1:55:03 And there was a DeepSeek Math-V2 paper where

1:55:07 they also had interesting inference scaling there where,

1:55:11 first, they had developed models that grade themselves, a separate model.

1:55:16 And I think that will be one aspect.

1:55:18 And the other, like Nathan mentioned,

1:55:20 it will be for RLVR branching into other domains.

1:55:24 The place where people are excited are value functions, which is pretty similar.

1:55:28 So process reward models are kind of like...

1:55:30 Process reward models assign how good something is

1:55:34 at each intermediate step in a reasoning process,

1:55:37 where value functions apply value to every token the language model generates.

1:55:41 Both of these have been largely unproven

1:55:44 in the language modeling and reasoning model era.

1:55:48 People are more optimistic about value functions for whatever reason now.

1:55:52 I think process reward models were tried a lot more in this pre-o1,

1:55:57 pre-reasoning model era, and a lot of people had a lot of headaches with them.

1:56:00 So I think a lot of it is human nature...

1:56:03 Value models have a very deep history in reinforcement learning.

1:56:06 They're one of the first things core to deep reinforcement learning existing,

1:56:10 is training value models.

1:56:12 So right now people are excited about trying value models,

1:56:16 but there's very little proof.

1:56:18 And there are negative examples in trying to scale up process reward models.

1:56:22 These things don't always hold in the future.

1:56:24 We came to this discussion by talking about scaling.

1:56:27 The simple way to summarize what you're saying

1:56:29 is you don't want to do too much RLHF, where the signal doesn't scale.

1:56:33 People have worked on RLHF for language models for years,

1:56:36 especially with intense interest after ChatGPT.

1:56:39 And the first release of a reasoning model trained with RLVR, OpenAI's o1,

1:56:44 had a scaling plot where if you increase training compute logarithmically,

1:56:47 you get a linear increase in evaluations.

1:56:50 This has been reproduced multiple times.

1:56:52 DeepSeek had a plot like this.

1:56:54 But there's no scaling law for RLHF where if you log-increase the compute,

1:56:58 you get performance.

1:56:59 In fact, the seminal scaling paper for RLHF

1:57:02 is scaling laws for reward model over-optimization.

1:57:05 So that's a big line to draw with RLVR and the methods we have now.

1:57:10 In the future, they will follow this scaling paradigm:

1:57:13 where you can let the best runs run for an extra 10x and you get performance,

1:57:18 but you can't do this with RLHF.

1:57:20 And that is just going to be field-defining in how people approach them.

1:57:24 While I'm a shill for people to academically do RLHF,

1:57:28 to do the best RLHF you might not need the extra 10 or 100x of compute,

1:57:34 but to do the best RLVR you do.

1:57:38 I think there's a seminal paper from a Meta internship.

1:57:42 It's called something like "The Art of Scaling Reinforcement Learning

1:57:46 with Language Models." What they describe as a framework is Scale-RL.

1:57:50 Their incremental experiment was like 10,000 V100 hours,

1:57:54 which is like thousands or tens of thousands of dollars per experiment.

1:57:58 They do a lot of them, and This cost is not accessible to the average academic,

1:58:04 which is a hard equilibrium where it's trying

1:58:08 to figure out how to learn from each community.

1:58:11 I was wondering if we could take a bit

1:58:13 of a tangent and talk about education and learning.

1:58:16 If you're someone listening to this who's

1:58:20 a smart person interested in programming and AI,

1:58:23 I presume building something from scratch is a good beginning.

1:58:28 So can you take me through what you would recommend people do?

1:58:32 I would personally start, like you said,

1:58:34 implementing a simple model from scratch that you can run on your computer.

1:58:38 The goal is not, when you build a model from scratch,

1:58:41 to have something for every day use.

1:58:43 It's not going to be your personal

1:58:46 assistant replacing an existing open-weight model or ChatGPT.

1:58:49 It's to see what exactly goes into the LLM, what comes out,

1:58:53 and how the pre-training works on your own computer, preferably.

1:58:59 Then you learn about pre-training,

1:59:01 supervised fine-tuning, and the attention mechanism.

1:59:03 You get a solid understanding of how things work,

1:59:06 but at some point you reach a limit, because small models can only do so much.

1:59:11 The problem with learning about LLMs at scale is

1:59:14 that it's exponentially more complex to make a larger model,

1:59:18 because the model isn't just larger—you have

1:59:21 to shard your parameters across multiple GPUs.

1:59:24 Even for the KV cache, there are multiple ways to implement it.

1:59:27 One is just to understand how it works, just to grow the cache.

1:59:31 You grow it step-by-step by, let's say, concatenating lists,

1:59:35 but then that wouldn't be optimal on GPUs.

1:59:39 You would pre-allocate a tensor and then fill it in.

1:59:42 But that adds another 20 or 30 lines of code.

1:59:45 And for each thing, you add so much code.

1:59:47 The goal with the book is basically to understand how the LLM works.

1:59:51 It's not going to be a production-level LLM,

1:59:53 but once you have that, you can understand the production-level LLM.

1:59:56 So you're trying to always build an LLM that's going to fit on one GPU?

2:00:00 Yes.

2:00:00 Most of them do.

2:00:02 I have some bonus materials on some MoE models.

2:00:04 One or two of them may require multiple GPUs,

2:00:08 but the goal is to have it on one GPU.

2:00:10 And the beautiful thing is, you can self-verify.

2:00:13 It's almost like RLVR.

2:00:14 When you code these from scratch,

2:00:17 you can take an existing model from the Hugging Face Transformers library.

2:00:21 The library is great, but if you want to learn about LLMs,

2:00:26 it's not the best place to start because the code

2:00:28 is so complex to fit so many use cases.

2:00:32 Because people use it in production,

2:00:34 it has to be really sophisticated, really intertwined, and hard to read.

2:00:38 It's not linear.

2:00:39 It started as a fine-tuning library,

2:00:40 and then it grew to be the standard representation of every model architecture.

2:00:45 Hugging Face is the default place to get a model,

2:00:48 and Transformers is the software.

2:00:49 It enables it so people can easily load a model and do something basic with it.

2:00:57 And all frontier labs that have open-weight

2:00:59 models have a Transformers version of it, like from DeepSeek to gpt-oss-120b.

2:01:03 That's the canonical weight format you can load.

2:01:06 But even even Transformers, the library, is not used in production.

2:01:10 People use SGLang or vLLM, and it adds another layer of complexity.

2:01:15 We should say that the Transformers library has like 400 models.

2:01:19 So it's the one library that tries to implement a lot of LLMs,

2:01:22 and so you have a huge codebase, basically.

2:01:25 It's huge.

2:01:26 It's like, I don't know, maybe millions—- That's crazy.

2:01:30 hundreds of thousands of lines of code.

2:01:33 Understanding the part you want to understand

2:01:35 is finding the needle in the haystack.

2:01:36 But what's beautiful is you have a working implementation,

2:01:39 so you can work backwards.

2:01:40 What I would recommend doing, or what I also do,

2:01:44 is if I want to understand, for example, how OLMo is implemented,

2:01:47 I would look at the weights in the model hub, the config file,

2:01:50 and then you can see, "Oh, they used so many layers.

2:01:53 They use, let's say,

2:01:55 Group Query Attention or Multi-Head Attention in that case."

2:01:58 Then you see all the components in a human-readable, 100-line config file.

2:02:01 And then you start, let's say, with your GPT-2 model and add these things.

2:02:05 The cool thing here is you can then load

2:02:08 the pretrained weights and see if they work in your model.

2:02:12 You want to match the same output that you get with a Transformer model,

2:02:15 and then you can use that, basically

2:02:18 as a verifiable reward to make your architecture correct.

2:02:21 Sometimes it takes me a day.

2:02:23 With OLMo 3, the challenge was RoPE for the position embeddings.

2:02:26 They had a YaRN extension and there was some custom scaling there,

2:02:32 and I couldn't quite match these things.

2:02:35 In this struggle, you kind of understand things.

2:02:38 At the end, you know you have it correct because you can unit test it.

2:02:42 You can check against the reference implementation.

2:02:44 I think that's one of the best ways to learn, really.

2:02:48 To basically reverse-engineer something.

2:02:51 I think that is something everyone interested

2:02:53 in getting into AI today should do.

2:02:56 That's why I liked your book.

2:02:58 I came to language models from the RL and robotics field.

2:03:01 I had never taken the time to just learn all the fundamentals.

2:03:06 This transformer architecture is so fundamental,

2:03:09 just as deep learning was in the past, and people need to do this.

2:03:14 I think where a lot of people get overwhelmed is,

2:03:19 "How do I apply this to have impact or find

2:03:21 a career path?" Because language models

2:03:23 make this fundamental stuff so accessible,

2:03:26 and people with motivation will learn it.

2:03:29 Then it's like, "How do I get cycles on goal to contribute to research?"

2:03:34 I'm actually fairly optimistic because the field moves so fast that a lot

2:03:38 of times the best people don't fully solve a problem because there's

2:03:42 a bigger problem to solve that's very low-hanging fruit, so they move on.

2:03:46 I think that a lot of what I was trying to do in this RLHF book is

2:03:51 take post-training techniques and describe how people think about

2:03:54 them influencing the model and what people are doing.

2:03:57 Then it's remarkable how many things I

2:04:01 just think people stop studying or don't pursue.

2:04:05 I think people trying to go narrow after doing the fundamentals is good,

2:04:08 and then reading the relevant papers and being engaged in the ecosystem.

2:04:14 It's like you actually...

2:04:16 actually...

2:04:16 The proximity that random people online have

2:04:19 to the leading researchers—no one knows who all the...

2:04:23 The anonymous accounts on X and ML are very popular,

2:04:26 and no one knows who all these people are.

2:04:28 It could just be random people that study this stuff deeply,

2:04:31 especially with the AI tools.

2:04:32 To just be like, "I don't understand this, keep

2:04:34 digging into it," is a very useful thing.

2:04:36 But there's a lot of research areas that maybe

2:04:39 have three papers that you need to read,

2:04:42 and then one of the authors will probably email you back.

2:04:45 But you have to put in a lot

2:04:47 of effort into these emails to understand the field.

2:04:50 I think it would be for a newcomer easily weeks of work

2:04:53 to feel like they can truly grasp what is a very narrow area,

2:04:57 but I think going narrow after you have the fundamentals will be

2:05:00 very useful to people because I've become very interested in character training,

2:05:05 which is how you make the model funny or sarcastic or serious,

2:05:11 and what do you do to the data to do this?

2:05:14 A student at Oxford reached out to me and was like,

2:05:16 "Hey, I'm interested in this," and I advised him.

2:05:18 And that paper now exists.

2:05:20 There's like two or three people in the world that were very interested in this.

2:05:25 He's a PhD student, which gives him an advantage, but for me,

2:05:28 that was a topic I was waiting for someone to be like,

2:05:31 "Hey, I have time to spend cycles on this." I'm sure

2:05:33 there's a lot more very narrow things where you're just like,

2:05:36 "It doesn't make sense that there was no answer to this." I

2:05:38 think it's just there's so much information coming that people are like,

2:05:42 "I can't grab onto any of these," but if you just stick in an area,

2:05:46 I think there's a lot of interesting things to learn.

2:05:48 Yeah, I think you can't try to do it all

2:05:51 because it would be very overwhelming and you would burn out.

2:05:53 For me, for example, I haven't kept up with computer vision in a long time;

2:05:57 I just focused on LLMs.

2:05:58 But coming back to your book,

2:06:00 I think this is a really great book and a really good

2:06:03 bang for the buck because if you want to learn about RLHF,

2:06:06 I wouldn't go out there and read RLHF papers because

2:06:08 you would be spending two years—- Some of them contradict.

2:06:11 I've just edited the book, and there's no chapter where I had to be like,

2:06:15 "X papers say one thing and Y papers say another,

2:06:18 and we'll see what comes out to be

2:06:21 true."- Just to go through the table of contents,

2:06:23 what are some ideas we might have missed in the bigger picture of post-training?

2:06:26 First of all, you did the problem setup,

2:06:28 training overview, what are preferences, preference data,

2:06:31 and the optimization tools, reward modeling, regularization,

2:06:35 instruction tuning, rejection sampling, and reinforcement learning.

2:06:42 Then, Constitutional AI and AI feedback, reasoning,

2:06:46 and inference-time scaling to use in function calling,

2:06:49 synthetic data and distillation,

2:06:51 evaluation, and then an open questions section, over-optimization,

2:06:54 style and information, and then product UX, character and post-training.

2:06:59 What are some ideas worth mentioning that connect

2:07:03 both the educational and the research components?

2:07:06 You mentioned character training, which is pretty interesting.

2:07:08 Character training is interesting because there's so little on it.

2:07:10 We talked about how people engage with these models.

2:07:13 We feel good using them because they're positive,

2:07:16 but that can go too far; it can be too positive.

2:07:19 And it's like, essentially, it's:

2:07:20 How do you change your data and decision-making

2:07:23 to make it exactly what you want?

2:07:26 And like, OpenAI has this thing called a model spec,

2:07:29 which is essentially their internal guideline

2:07:31 for what they want the model to do, and they publish this to developers.

2:07:35 So, essentially, you can know what is a failure

2:07:38 of OpenAI's training—where they have the intentions and they

2:07:41 haven't met them yet— versus what is something that they

2:07:43 actually wanted to do and that you don't like.

2:07:46 And that transparency is very nice,

2:07:47 but all the methods for curating these documents and how

2:07:50 easy it is to follow them is not very well known.

2:07:53 I think the way the book is designed is that the RL

2:07:56 chapter is obviously what people want

2:07:57 because everybody hears about it with RLVR,

2:07:59 and it's the same algorithms and the same math,

2:08:01 but you can use it in very different documents.

2:08:05 So I think the core of RLHF is like how messy preferences are.

2:08:09 It's essentially a rehash of a paper I wrote years ago,

2:08:13 but this is essentially the chapter that'll tell

2:08:15 you why RLHF is never ever fully solvable because,

2:08:21 the way that even RL is set up, it assumes that preferences can be quantified

2:08:29 and that multiple preferences can be reduced to single values.

2:08:33 And I think it relates in the economics

2:08:35 literature to the Von Neumann-Morgenstern utility theorem,

2:08:38 and that is the chapter where all of that philosophical, economic,

2:08:43 and psychological context tells you what gets compressed into doing RLHF.

2:08:47 So it's like you have all of this and then later in the book it's like:

2:08:50 You use this RL map to make the number go up.

2:08:52 And I think that's why it'll be very

2:08:54 rewarding for people to do research on, because

2:08:57 quantifying preferences is something that humans have designed

2:09:01 a problem in order to make preferences studyable.

2:09:04 But there's kind of fundamental debates, like, an example is in a language model

2:09:09 response you have different things you care about, like accuracy or style.

2:09:12 And when you're collecting the data, they all get compressed into: "I like

2:09:16 this more than another." And that is happening,

2:09:19 and there's a lot of research in other areas

2:09:22 of the world that go into how you should actually do this.

2:09:26 I think social choice theory is the subfield

2:09:30 of economics around how you should aggregate preferences.

2:09:33 And I went to a workshop that published a white paper

2:09:37 on: "How can you think about using social choice theory for RLHF?" So

2:09:41 I mostly would want people that get excited about the math

2:09:44 to come and find things where they could stumble into this broader context.

2:09:48 I think there's a fun thing:

2:09:49 I just keep a list of all the tech reports of reasoning models I like.

2:09:54 So in Chapter 14, where there's a short summary of RLVR,

2:09:57 there's just a gigantic table where I

2:09:59 list every single reasoning model that I like.

2:10:03 I think in education, a lot of it needs to be like, at this point,

2:10:07 what I like, because the language models are so good at the math.

2:10:11 For example, the famous paper, Direct Preference Optimization,

2:10:13 which is a much simpler way of solving the problem than RL.

2:10:17 The derivations in the appendix skip steps of math.

2:10:21 And for this book, I redid the derivations and I'm like,

2:10:24 "What the heck is this log trick that they use

2:10:26 to change the math?" But doing it with language models, they're like,

2:10:29 "This is the log trick." And I'm like,

2:10:31 "I don't know if I like this, that the math is so commoditized." I think

2:10:35 some of the struggle in reading this appendix-

2:10:38 ...and following the math is good for learning.

2:10:44 Yeah, we're returning to this often on the topic of education.

2:10:47 You both have brought up the word "struggle" quite a bit.

2:10:51 So there is value.

2:10:52 If you're not struggling as part of this process,

2:10:55 you're not fully following the proper process for learning.

2:10:59 proper process for learning, I suppose.

2:11:02 Some providers are working on models

2:11:04 for education designed to not give- actually, I haven't used them,

2:11:07 but I'd guess they're designed to not give all the information at once.

2:11:12 And make people work for it.

2:11:13 Training models to do this would be a wonderful contribution.

2:11:16 Where, like all of the stuff in the book, you had to reevaluate every decision.

2:11:19 decision for it- It's a great example.

2:11:21 There's a chance we work on it at Ai2, which I thought would be so fun.

2:11:26 It makes sense.

2:11:27 I did something like that the other day for video games.

2:11:30 Sometimes for pastime I play video games,

2:11:32 like I like- Video games with puzzles, like Zelda and Metroid.

2:11:36 And there's this new game where I really got stuck and was okay with it.

2:11:41 I don't want to struggle for two days, so I used an LLM.

2:11:45 But then you say, "Hey, please don't add spoilers.

2:11:47 Just, you know, I'm here and there.

2:11:49 What do I have to do next?" You can do the same thing for math where you say,

2:11:53 "Okay, I'm stuck at this point.

2:11:55 Don't give me the full solution,

2:11:57 but what is something I could try?" Where you carefully probe it.

2:12:01 But the problem here is I think it requires discipline.

2:12:05 Many people enjoy math,

2:12:06 but there are also a lot of people who need to do it for their homework,

2:12:11 and then it's like a shortcut.

2:12:13 We could develop an educational LLM, but other LLMs are still there,

2:12:17 and there's still a temptation to use the other LLMs.

2:12:20 I think many people in college understand the stuff they're passionate

2:12:23 about- about- ...they're self-aware and they understand it shouldn't be easy.

2:12:27 I think we just have to develop a good taste- ...talk about research taste,

2:12:33 school taste about stuff that you should

2:12:36 be struggling on- ...and stuff you shouldn't be.

2:12:38 It's tricky, because you don't have

2:12:40 good long-term vision sometimes you don't have

2:12:43 good long-term vision about what would be actually useful to you in your career.

2:12:48 But you have to develop that taste, yeah.

2:12:52 I was talking to my fiancee or friends about this, there's this brief

2:12:56 10-year window where all of the homework and all the exams could be digital.

2:13:00 Before that, everybody had to do all the exams

2:13:02 in blue books because there was no other way.

2:13:04 And now after AI, everyone's going to need to be

2:13:06 in blue books and oral exams because everyone could cheat so easily.

2:13:09 It's like this brief generation that had

2:13:11 a different education system where everything could be digital,

2:13:15 but you still couldn't cheat.

2:13:16 And now it's just going back.

2:13:18 It's just very funny.

2:13:21 You mention character training.

2:13:22 Just zooming out on a more general topic,

2:13:24 for that project how much compute was required?

2:13:28 And in general, to contribute as a researcher,

2:13:31 are there places where not too much compute is

2:13:35 required where you can actually contribute as an individual researcher?

2:13:39 For the character training thing, I think this research is built on fine-tuning

2:13:43 about 7 billion parameter models with LoRA,

2:13:46 which is essentially only fine-tuning a small

2:13:48 subset of the weights of the model.

2:13:51 I don't know exactly how many GPU hours that would take.

2:13:55 But it's doable.

2:13:56 Not doable for every academic.

2:13:57 The situation for some academics is so dire that the only

2:14:00 work you can do is doing inference where you have

2:14:02 closed models or open models and you get completions from them

2:14:05 and you can look at them and understand the models.

2:14:07 And that's very well-suited to evaluation,

2:14:09 where you want to be the best at creating representative

2:14:14 problems that the models fail on or show certain abilities,

2:14:17 which I think that you can break through with this.

2:14:21 I think that the top-end goal for a researcher working on evaluation,

2:14:25 if you want to have career momentum,

2:14:27 is that Frontier Labs pick up your evaluation.

2:14:30 You don't need to have every project do this.

2:14:32 But if you go from a small university with no compute and find something

2:14:36 that Claude struggles with, and then the next

2:14:39 Claude model has it in the blog post, there's your career rocket ship.

2:14:42 I think that's hard,

2:14:44 but if you want to scope the maximum possible impact with minimum compute,

2:14:48 it's something like that, which is just get very narrow

2:14:51 and it takes learning of where the models are going.

2:14:54 So you need to build a tool that tests where Claude 4.5 will fail.

2:14:59 If I'm going to start a research project,

2:15:02 I need to think where the models in eight months are going to be struggling.

2:15:06 But what about developing totally novel ideas?

2:15:08 This is a trade-off.

2:15:09 I think that if you're doing a PhD,

2:15:11 you could also be like, "It's too risky to work in language models.

2:15:15 I'm going way longer term," which is like what is— what is

2:15:19 the thing that's going to define language model development in 10 years?

2:15:22 I end up being a person that's pretty practical.

2:15:25 I mean, I went to my PhD where it was like, "I got into Berkeley.

2:15:28 Worst case, I get a master's,

2:15:29 and then I go work in tech." I'm very practical about it,

2:15:32 so I'm like the life afforded to people

2:15:36 to work at these AI companies, the amount of...

2:15:38 OpenAI's average compensation is over a million

2:15:40 dollars in stock a year per employee.

2:15:43 For any normal person in the US,

2:15:46 to get into this AI lab is transformative for your life.

2:15:49 So I'm pretty practical about it.

2:15:50 there's still a lot of upward mobility

2:15:52 working in language models if you're focused.

2:15:54 And look at these jobs.

2:15:55 But from a research perspective,

2:15:57 the transformative impact in these academic awards...

2:16:01 to be the next Yann LeCun is from not

2:16:04 working on— not caring about language model development very much.

2:16:07 It's a big financial sacrifice in that case.

2:16:09 So I work with some awesome students, and they're like,

2:16:12 "Should I go work at an AI lab?" And I'm like,

2:16:14 "You're getting a PhD at a top school.

2:16:16 Are you gonna leave to go to a lab?" I don't know.

2:16:19 If you go work at a top lab, I don't blame you.

2:16:22 Don't go work at some random startup that might go to zero.

2:16:24 But if you're going to OpenAI, I'm like, "It could be worth leaving a PhD

2:16:29 for."- Let's more rigorously think through this.

2:16:32 So where would you give a recommendation

2:16:34 for people to do a research contribution?

2:16:36 So the options are academia: get a PhD.

2:16:40 Spend five years publishing.

2:16:44 Compute resources are constrained.

2:16:46 There's— there's research labs that are more

2:16:50 focused on open-weight models, and working there.

2:16:57 Or closed frontier research labs.

2:17:01 So OpenAI, Anthropic, xAI, and so on.

2:17:04 The two gradients are: the more closed,

2:17:06 the more money you tend to get, but you also get less credit.

2:17:10 In terms of building a portfolio of things that you've done,

2:17:17 it's very clear what you have done as an academic.

2:17:20 Versus if you are going to trade this fairly

2:17:25 reasonable progression for being a cog in the machine,

2:17:28 which could also be very fun.

2:17:30 So I think it's very different career paths.

2:17:33 But the opportunity cost for being a researcher is

2:17:36 very high because PhD students are paid essentially nothing.