Can humans make AI any better?

0:00 In 1971, the US government agency ARPA launched

0:04 a program to push the frontiers of speech recognition.

0:07 ARPA set an ambitious 5-year goal,

0:10 a system that could recognize a thousand different words with 90% accuracy.

0:15 5 years later, a team from Carnegie Melon demonstrated Harpy,

0:19 an AI system capable of recognizing a,01 different words with 95% accuracy.

0:25 The ARPA program was a success, but over the next decade,

0:29 the core of Harpy's intelligence, an enormous knowledge graph,

0:33 would be replaced by something completely different.

0:36 Each of the over 14,000 nodes

0:38 in Harpy's knowledge graph represents a single phone.

0:42 One of the 98 basic sounds the team used to break apart spoken American English.

0:47 The graph itself captures all the different ways these phones

0:50 can be strung together to create sentences considered valid by Harpy.

0:55 In this part of the graph, we can see the pathway for the phrases,

0:59 tell me about China, tell me about Nixon, and give me the headlines.

1:04 Each phone in Harpy's graph includes an expected

1:07 frequency curve for the sound of the phone.

1:10 These curves are tuned for the current speaker as audio comes in.

1:15 For example, if I say tell me about China,

1:18 a signal processing algorithm chops the waveform into blocks

1:22 and the frequency content of each block is computed.

1:25 From here, the frequency content of our first block is

1:28 compared to the phones at the start of our graph.

1:31 G in give and T in tell.

1:34 T is a better match.

1:35 So, we progress down this part of the tree.

1:38 We then move to the second block in our waveform.

1:41 This is the L intel and find its closest match.

1:45 Block by block, we progress through our knowledge graph

1:49 using a frequency matching score to guide our search,

1:52 resulting in a final sentence.

1:55 Note that the Harpy team also used a somewhat

1:57 more sophisticated search method called beam search to help

2:00 find the best overall path through the graph instead

2:03 of just greedily choosing the best match at each step.

2:08 The way Harpy's knowledge graph is constructed is painstaking and fascinating.

2:12 First, a language design expert specifies a grammar.

2:17 Harpy could not accept arbitrary sequences of words.

2:20 This would have significantly degraded performance.

2:24 Instead, a formal grammar was used to capture valid

2:27 sentence structures for Harpy's intended use case of document retrieval.

2:31 This small example Harpy grammar tells us that the word tell can

2:35 be followed by me or us and then by all or about.

2:40 From here, each word in Harpy's vocabulary is broken into individual phones.

2:45 These breakdowns are themselves little graphs.

2:48 Here's the pronunciation graph the Harpy team used for the word tell.

2:52 Note that the graph branches due

2:54 to the possibility of pronouncing tell in different ways.

2:58 tell can be pronounced with an extra vowel sound in the middle.

3:02 Replacing each word in our word graph with its phone breakdown,

3:05 we get a larger graph.

3:06 But we aren't done yet.

3:09 When we speak, we often change the way

3:11 words sound depending on the words around them.

3:14 This is known as a juncture.

3:16 When saying a phrase like about China,

3:18 the T in about is often dropped, leaving about China.

3:23 And in transitions like me all, a subtle extra Y sound known as a glide

3:29 is added by some speakers to transition between vowels.

3:33 The Harpy team accounted for this by creating rules to manipulate

3:37 the phone graph to allow for various junctures between phones.

3:42 Now, although Harpy achieved Arper's ambitious goal

3:45 of 90% accuracy on a thousandword vocabulary,

3:49 further scaling Harpy's performance proved difficult.

3:53 And over the next decade,

3:54 Harpy's knowledge graph was replaced by hidden markoff models.

3:59 These models can still be understood

4:00 as implementing a graph structure between phone nodes,

4:04 but the edges of the graph are now probabilities that are learned from data.

4:09 No language expert specified grammar

4:11 and no linguistic expert specified juncture rules.

4:15 This was a controversial shift at the time.

4:18 Building our knowledge into AI systems

4:20 was considered critical by many researchers.

4:24 However, by the late 1980s and early 1990s,

4:27 virtually all speech recognition systems had moved to hidden markoff models.

4:32 And these systems were able to scale to much

4:34 larger vocabularies of 5,000 and then 20,000 words.

4:39 Decades later, in 2019, the computer scientist Richard Sutton published a highly

4:44 cited essay that he called the bitter lesson.

4:47 In his essay, Sutton points out that Harpy's replacement with hidden

4:50 Markoff model based approaches is part of a broader trend.

4:55 A trend that Sutton calls the biggest lesson from 70 years of AI research.

5:01 Specifically, that general methods that leverage computation are

5:04 ultimately the most effective and by a large margin.

5:09 and that trying to build human knowledge into our systems

5:11 as the Harpy team did helps initially but then becomes highly counterproductive.

5:18 The timing of Sutton's essay could not have been better.

5:21 OpenAI had just released GPT2 a few weeks prior

5:24 [music] and a new paradigm was just beginning to emerge.

5:29 A general architecture, the transformer,

5:31 focused on a simple learning objective, next token prediction,

5:35 and trained with massive amounts of compute,

5:38 could produce shockingly intelligent language models.

5:42 For myself and many others, it seemed like we had learned the bitter lesson.

5:47 We had found a method of creating

5:48 AI systems that leveraged massive amounts of compute

5:52 and appeared to rely only minimally on human

5:55 assumptions about how AI systems should work.

5:58 But then in 2025, something really surprising happened.

6:03 Richard Sutton went on a podcast.

6:06 In the first 10 minutes of Sutton's interview with Darkash Patel,

6:09 it became clear that Sutton had a completely different

6:12 take on large language models and the bitter lesson.

6:15 So, I mean, it's interesting because you wrote

6:17 this essay in 2019 titled The Bitter Lesson,

6:20 and this is the most influential essay perhaps in the history of AI,

6:24 but people have used that as a justification

6:30 for scaling up LLMs because in their view,

6:34 this is the one scalable way we have found

6:37 to pour ungodly amounts of compute into learning about the world.

6:41 And so it's interesting that your perspective is

6:43 that the LLMs are actually not bitter lesson.

6:47 It's an interesting question whether uh large language models

6:50 are are uh a case of the bitter lesson.

6:57 Yeah.

6:55 Because they are clearly um a a way of using massive computation things

7:02 that will scale with computation up to up to the limits of the internet.

7:09 Yeah.

7:08 uh but they're also a way of putting in lots of um human knowledge and uh

7:16 so so this is an interesting question

7:19 um it's a sociological or industry question uh

7:24 will they reach the limits of of of the data and and be superseded by things

7:32 that that are can get more data just

7:36 from experience rather than from uh from people.

7:41 Uh in some ways it's a classic case

7:44 of the of the of the bitter lesson with the more

7:47 the more human knowledge we put into the large language

7:49 models the better they can do and so it feels good.

7:53 Um and yet uh one well I in particular expect there to be systems that can

8:01 learn from experience which could well perform

8:04 much much better and be much more scalable.

8:07 In which uh case it will be another

8:09 instance of the bitter lesson that the things

8:12 that that used human knowledge were eventually superseded by things

8:17 that just um trained from uh experience and computation.

8:23 This is such a remarkable moment.

8:26 Sutton is basically telling us that much of the field

8:28 has interpreted his essay in exactly the wrong way.

8:31 that large language models are an example of the bitter lesson,

8:35 but a negative example, one that like Harpy relies far too much on human

8:39 knowledge since LLMs are trained on human generated text.

8:45 So, is Sutton right?

8:47 Will LLMs hit a performance barrier due to their reliance on human

8:51 knowledge and need to be replaced with very different types of AI systems?

8:56 Sutton ends the bitter lesson with these lines.

8:59 We want AI agents that can discover like we can,

9:03 not which contain what we have discovered.

9:07 Building in our discoveries only makes it harder

9:09 to see how the discovering process can be done.

9:14 So what would an AI system look like that can discover like we can?

9:19 The primary way large language models

9:20 are trained today is through supervised learning.

9:23 Given some training text, for example, the first line of Harry Potter,

9:28 the text is broken into little pieces known as tokens,

9:31 and the model is trained to predict the next

9:33 token given all the tokens that come before.

9:36 So given the input Mr.

9:37 and Mrs.

9:38 Dersley of, the model is trained to predict the token for the word number.

9:43 And given the input Mr.

9:44 and Mrs.

9:44 Dersley of number, the model is trained to predict

9:47 the token for the word for and so on.

9:51 Token by token, we're teaching the model what to say.

9:54 And Sutton's criticism here is that like Harpy,

9:57 this process relies too much on human knowledge

10:00 since we're training our model to imitate humans.

10:04 So, how else might we train AI systems?

10:08 Sutton is generally known as the father of reinforcement learning.

10:12 One of the most compelling modern examples

10:14 of AI systems trained using reinforcement learning

10:16 instead of supervised learning are Google Deep

10:19 Minds Alph Go and Alph Go Zero agents.

10:22 Now, Alph Go and Alph Go Zero are game playing agents,

10:26 not large language models, but there are some really interesting parallels here.

10:32 Now, before we see exactly how reinforcement learning allows

10:35 us to push beyond the limits of human knowledge,

10:38 if you're someone who's obsessed with this stuff

10:40 and looking for your next career move, check out this video sponsor, Tufalabs.

10:44 Tufalabs recently won the ARC AGI3 preview competition,

10:49 where AI agents must learn to play and win completely

10:52 novel games that the agents developers have never seen before.

10:56 Tufa Labs is an independent AI lab based in Zurich and puts

11:00 out really interesting research on the frontiers

11:02 of LLMs and reinforcement learning.

11:05 In this recent paper, the team achieved very impressive performance on the MIT

11:09 integration B challenge by creating a self-improvement loop

11:14 where an LLM creates its own math practice

11:16 problems and learns to solve them via reinforcement learning.

11:20 Tufa Labs is serious about compute and is

11:23 currently expanding their infrastructure with two Nvidia

11:27 NVL72 GB300 racks which is a significant amount

11:31 of compute given the relatively small team size.

11:34 TUFALabs is fully self-funded by applying ML to quantitative finance.

11:39 This funding structure is similar to deepseats.

11:42 If this sounds interesting, you can apply at tufalabs.ai/join.

11:47 Now back to the bidder lesson.

11:50 When building Alph Go,

11:51 the first AI system that reached superhuman performance on the board game Go,

11:56 the DeepMind team first trained a supervised policy network.

12:00 In the jargon of reinforcement learning,

12:02 an agent's policy determines what action an agent will take in a given state.

12:07 In the game of Go, the current state is the board position,

12:11 and the action is where the agent will place its next stone on the board.

12:15 The DeepMind team used a deep neural network to learn this policy.

12:20 This part of DeepMind's solution is

12:22 strikingly similar to large language model training.

12:25 Given some input text, large language models return a probability

12:29 for each token the model could say next.

12:32 And given an input board position,

12:34 Alph Go's policy network returns a probability for each

12:37 board position where Alph Go could place a stone.

12:41 Note that Alph Go used a convolutional

12:43 architecture while LLMs generally use transformers.

12:47 But this distinction isn't really significant for our comparison here.

12:52 Very similar to LLM training, Alph Go's policy network was first trained

12:56 using supervised learning on human generated examples,

13:00 learning to match expert human players moves in recorded games.

13:04 The resulting supervised learning trained policy

13:07 network is a fairly competent Go player.

13:10 Evaluating the model in a tournament against other GO agents resulted

13:14 in an ELO rating of 1517 putting the network at mid amateur level.

13:21 Now in the reinforcement learning paradigm agents learn not

13:24 from direct supervision but instead through interacting with their environments.

13:29 In the case of Alph Go, this just means taking the very natural

13:33 approach of learning from actually playing the game.

13:37 The DeepMind team took their trained supervised policy network

13:40 and had it play against various versions of itself.

13:44 After each game, the moves taken by the winning model

13:47 were used as positive training examples to train the policy network.

13:52 And the moves taken by the losing model were used as negative examples.

13:56 This training process is very similar to learning from human expert moves.

14:01 The key difference here is that these moves are generated

14:03 by two policy networks actually playing go and the underlying

14:07 signal the model learns from is not the opinion

14:10 of a human expert but instead the outcome of a real game.

14:14 This approach is known as a policy gradient method and is a key

14:18 technique in reinforcement learning initially developed

14:21 by Sutton and others in the 1990s.

14:24 Now, it turns out that the reinforcement learning trained policy

14:27 network alone was not enough to deliver superhuman Go performance.

14:33 The DeepMind team needed one more big idea from reinforcement learning.

14:39 It turns out there's another way to set up our learning problem here,

14:42 and it's actually an older and arguably more central

14:45 reinforcement learning idea than the policy gradient method we've seen.

14:50 Instead of training a network to predict the move that should come next,

14:54 what if we trained a network to measure the quality of a given board position

14:59 or more concretely to estimate the probability

15:01 of winning starting from a given board position.

15:05 This approach is broadly known in reinforcement

15:07 learning as value function estimation where

15:10 the term value refers to expected future rewards in this case winning the game.

15:16 Value functions play a central role

15:18 in Sutton's canonical text on reinforcement learning.

15:22 The most important component of almost all reinforcement learning

15:25 algorithms we consider is a method for efficiently estimating values.

15:30 The central role of value estimation is arguably the most important thing

15:34 that has been learned about reinforcement learning over the last six decades.

15:39 To make use of this reinforcement learning idea,

15:42 the Google DeepMind team trained a second

15:44 neural network that they called a value network.

15:48 This network took in the same board position inputs as our policy network.

15:52 But instead of predicting what move should come next,

15:55 the value network predicts the probability of Alph

15:58 Go winning the game from that position.

16:01 The DeepMind team trained their value network on board

16:04 positions from games between different versions of Alph Go.

16:07 again avoiding learning from human gameplay.

16:10 Finally, the DeepMind team brought together their policy and value network

16:14 with a method called Monte Carlo tree search to create a formidable Go agent.

16:20 The policy network allows Alph Go to narrow

16:22 down the number of next possible moves,

16:25 while the value network provides a strong estimate of the probability of winning

16:30 if Alph Go continues playing down a given branch of the tree.

16:35 Alph Go famously went on to defeat Lisa Dole in 2016,

16:39 the second ranked Go player in the world at the time.

16:42 The following year, the DeepMind team revealed Alph Go Zero,

16:46 an even stronger player than Alph Go

16:48 that remarkably did not learn from any human games.

16:52 Instead, relying only on reinforcement learning from real gameplay.

16:57 By learning from real interactions with their environments,

17:00 Alph Go and Alph Go Zero dramatically outperformed

17:03 agents trained to imitate human gameplay using supervised learning,

17:08 discovering their own ways to play the game.

17:11 Alph Go's playing style has been described by experts

17:14 as playing against an alien and is from an alternate dimension.

17:19 These are the types of AI systems that Sutton is referring to when he says,

17:23 "Oh, well, I in particular expect there to be systems that can learn

17:27 from experience and which could well perform

17:29 much much better and be much more scalable."

17:33 This really makes me wonder if our current generation of language

17:36 models are constrained in the same way as the AlphaGo

17:39 policy network that was trained with supervised learning was unable

17:43 to discover on their own limited to imitating human language and intelligence.

17:49 Now, it's important to note here that reinforcement learning does

17:52 already play an important role in large language model training.

17:56 After LLMs are trained on next token prediction using supervised learning,

18:00 reinforcement learning through human feedback or RLHF is used

18:04 to align models to human preferences using reinforcement learning techniques.

18:09 And a great deal of recent LLM progress has been driven by reasoning

18:13 models that make use of reinforcement learning with verifiable rewards or RLVR

18:20 where LLMs are trained with reinforcement learning to find their own paths

18:23 to solving problems with known answers such as math problems and coding.

18:29 Pre-training on human generated text and then using reinforcement learning

18:33 to allow models to discover on their own is an exciting direction,

18:37 but it remains to be seen how far this approach can take us.

18:42 In 2025, David Silver, the lead researcher of the AlphaGo project,

18:47 teamed up with Richard Sutton to write

18:48 an essay called Welcome to the Era of Experience.

18:53 Silver and Sutton argue that LLMs are currently limited by human knowledge,

18:57 giving an interesting thought experiment.

19:00 If we trained an LLM on human knowledge from 5,000 years ago,

19:04 it would reason about the physical world in terms of animism.

19:08 If we trained on human knowledge from a thousand years ago,

19:10 it would reason in theistic terms, 300 years ago in terms of Newtonian physics,

19:16 and 50 years ago in terms of quantum physics.

19:20 Moving from one paradigm to the next

19:22 requires actually interacting with the physical

19:25 world as Silver and Sutton say in order to overturn facious methods of thought.

19:32 From here, Sutton and Silver argue that we're on the threshold of a new era

19:35 of AI where agents will learn from real

19:39 world reward signals instead of human knowledge,

19:43 discovering new ways to optimize measures like cost,

19:46 health metrics, climate metrics, profits, sales, and energy consumption.

19:52 Sutton and Silver give the example

19:54 of DeepMind's alpha proof agent which combines

19:56 LLMs and reinforcement learning to discover its

19:59 own very impressive methods of mathematical reasoning.

20:04 So, have we learned the bitter lesson or are we just repeating it?

20:08 When we look back on LLMs one day,

20:10 will they feel like harpy so clearly limited by human knowledge?

20:16 And if so, is reinforcement learning the answer?

20:20 And can LLM serve as the scaffolding that unlocks this next frontier?

20:24 Or do we need to try something totally different?

20:27 My take here is that the bitter

20:29 lesson in Sutton and Silver's reinforcement learning

20:32 perspectives form a really helpful lens onto

20:36 the limitations of our current generation of AI.

20:40 I'm more skeptical about a reinforcement

20:42 learning renaissance being around the corner.

20:45 mostly because the domains we've seen reinforcement [music]

20:47 learning perform really well in like playing games, mathematical proofs,

20:51 and coding, still feel very removed to me from many

20:54 of the real world problems that we care about.

20:58 Either way, it will be fascinating to see what happens next.

21:05 The Welsh Labs team and I have written a whole new book on AI.

21:09 It's beautifully illustrated and is a great way to dig

21:12 deeper into the topics we cover in these videos.

21:15 This is the book I've always wanted to write.

21:18 We've really leaned into the visuals.

21:20 The book has hundreds of figures.

21:23 I especially like these full page spreads.

21:26 This one shows how lost landscapes are computed.

21:30 On the next page, we jump into this super

21:32 highquality overhead contour plot view of our landscape.

21:36 and we show how we might expect our model

21:38 to work its way through valleys to reach its global minimum,

21:41 but it instead creates what looks like a wormhole on our lost landscape.

21:46 We're putting a huge amount of effort into each

21:48 chapter to create these kinds of visuals and deep explanations,

21:52 trying to give the most visceral feel we can for how this stuff really works.

21:58 Each chapter includes supporting Python code that walks

22:00 [music] through the key results from that chapter.

22:03 And there's also a supporting GitHub repo

22:05 as well that's a bit more comprehensive.

22:08 At the end of each chapter, you'll also find exercises.

22:12 We've put a ton of thought into these.

22:15 Here's an exercise from the chapter on back

22:17 propagation where you're given a small complete

22:19 neural network and asked to move some

22:22 data through the network using a few equations.

22:25 Then you're asked to compute the network's gradients and use your computed

22:28 gradients to fill in steps in the model's real learning process.

22:33 These exercises are designed to get you as hands-on as possible

22:36 with modern AI and solutions are in the back of the book.

22:40 Most of the exercises are written or programming,

22:42 but my favorite is probably this spread that [music]

22:45 gives you instructions for building your own perceptron machine.

22:49 The book starts with a fresh take on the fundamentals,

22:51 the perceptron, gradient descent,

22:53 [music] back propagation, deep models, and alexet.

22:57 and then uses this foundation to dive

22:59 into cutting edge topics including neural scaling laws,

23:02 mechanistic interpretability,

23:04 and AI image and video generation models like Sora.

23:08 Each chapter goes along with a Welch Labs

23:10 video that came out over the last 18 months.

23:14 I really think that the book is the best

23:16 way to get deeper into each video's topic.

23:19 The book is great for self-study, AI courses,

23:22 or just looks great on your coffee table.