We're Not Ready for Superintelligence

0:00 The impact of superhuman AI over the next  decade will exceed that of the industrial  

0:05 revolution. That is the opening claim of  AI 2027. It is a thoroughly researched  

0:11 report from a thoroughly impressive group of  researchers led by Daniel Kokotajlo. In 2021,  

0:16 over a year before ChatGPT was released,  he predicted the rise of chatbots,  

0:20 hundred million dollar training runs, sweeping AI  chip export controls, chain of thought reasoning.

0:24 He's known for being very early and very  right about what's happening next in AI. So  

0:29 when Daniel sat down to game out a month by month  prediction of the next few years of AI progress,  

0:34 the world sat up and listened,  from politicians in Washington —

0:38 I, I'm worried about this stuff. I actually  read the paper of the guy that you had on

0:41 to the world's most cited computer scientist,  the godfather of AI. What is so exciting and  

0:47 terrifying about reading this document  is that it's not just a research report.  

0:51 They chose to write their prediction as  a narrative to give a concrete and vivid  

0:55 idea of what it might feel like to live  through rapidly increasing AI progress.

1:00 And spoiler, it predicts the extinction of the  human race. Unless we make different choices.

1:15 The AI 2027 scenario starts in summer 2025,  

1:19 which happens to be when we're filming  this video. So why don't we take stock  

1:22 of where things are at in the real world and  then jump over to the scenario’s timeline.

1:25 Right now it might feel like everyone, including  your grandma, is selling an AI powered something.

1:30 Go pro with the new Oral-B Genius AI

1:33 Flippy the chef makes spuds spectacular.

1:36 But most of that is actually tool  AI. Just narrow products designed  

1:40 to do what Google Maps or calculators  did in the past, help human consumers  

1:44 and workers do their thing. The holy grail  of AI is Artificial General Intelligence.

1:49 AGI AGI AGI AGI AGI AGI, Artificial General Intelligence

1:55 is a system that can exhibit all the  cognitive capabilities humans can.

1:59 Creating a computer system that itself is a  worker. That's so flexible and capable, we can  

2:04 communicate with it in natural language and hire  it to do work for us, just like we would a human.

2:09 And there are actually surprisingly few serious  players in the race to build AGI. Most notably,  

2:13 there's Anthropic, OpenAI, and Google DeepMind,  

2:16 all in the English speaking world, though  China and DeepSeek recently turned heads  

2:20 in January with a surprisingly advanced  and efficient model. Why so few companies?

2:25 Well, for several years now, there's  basically been one recipe for training  

2:28 up in advanced cutting edge AI. And it  has some pricey ingredients. For example,  

2:33 you need about 10% of the world's supply of the  most advanced computer chips. Once you have that,  

2:37 the formula is basically just: throw more  data and compute at the same basic software  

2:42 design that we've been using since 2017  at the frontier of AI, the transformer.

2:46 That's what the T in GPT stands for.

2:49 To give you an idea of just how much  hardware is the name of the game right now,  

2:52 this represents the total computing power, or  compute, used to train GPT-3 in 2020. It's the  

2:58 AI that would eventually power the first version  of ChatGPT. You probably know how that went.

3:02 ChatGPT is the fastest growing  user-based platform in history.  

3:06 A hundred million users on ChatGPT in two months

3:10 And this is the total compute used to  train GPT-4 in 2023. The lesson people  

3:16 have taken away is pretty simple. Bigger  is better, and much bigger is much better.

3:22 You have all these trends, you have trends in  revenue going up, trends in compute going up,  

3:26 trends in various benchmarks going up.  How does it all come together? You know,  

3:30 what does the future actually look like? Questions  like how do these different factors interact?  

3:33 Seems plausible that when the benchmark scores are  so high, then there should be crazy effects on,  

3:39 you know, jobs, for example, and that that  would influence politics. And then also,  

3:44 you know, so all these things interact and  how do they interact? Well, we don't know,  

3:48 but thinking through in detail how it might  go is the way to start grappling with that.

3:53 Okay. So that's where we are in the real  world. The scenario kicks off from there  

3:56 and imagines that in 2025, we have the top AI  labs releasing AI agents to the public in summer.  

4:03 An agent is an AI that can take instructions  and go into a task for you online like booking  

4:08 a vacation or spending half an hour searching the  internet to answer a difficult question for you,  

4:12 but they're pretty limited and unreliable at  this point. Think of them as enthusiastic interns  

4:17 that are shockingly incompetent sometimes.  Since the scenario was published in April,  

4:22 this early prediction has actually already  come true. In May, both OpenAI and Anthropic  

4:26 released their first agents to the public.  The scenario imagines that OpenBrain,  

4:30 which is like a fictional composite of the  leading AI companies, has just trained and  

4:35 released Agent-0, a model trained on  a hundred times the compute of GPT-4.

4:43 We, uh, we don't have enough  blocks for that. At the same time,  

4:46 OpenBrain is building massive data centers  to train the next generation of AI agents,  

4:50 and they're preparing to trade agent one  with 1000 times the compute of GPT-4.  

4:55 This new system, Agent-1, is designed  primarily to speed up AI research itself.

4:59 The public will actually never see the full  version because OpenBrain withholds its best  

5:03 models for internal use. I want you to keep that  in mind as we go through this scenario. You're  

5:07 gonna be getting it from a God's eye view,  with full information from your narrator,  

5:11 but actually living through this  scenario as a member of the public  

5:14 would mean being largely in the dark as  radical changes happen all around you.

5:18 Okay, so OpenBrain wants to win the AI race  against both its Western competitors and against  

5:23 China. The faster they can automate their R&D  cycle, so getting AI to write most of the code,  

5:28 help design experiments, better chips, the  faster that they can pull ahead. But the same  

5:32 capabilities that make these AI such powerful  tools also make them potentially dangerous.

5:37 An AI that can help patch security vulnerabilities  can also exploit them. An AI that understands  

5:42 biology can help with curing diseases,  but also designing bioweapons. By 2026,  

5:48 Agent-1 is fully operational and  being used internally at OpenBrain.  

5:52 It is really good at coding. So  good, it starts to accelerate AI  

5:56 research and development by 50%,  and it gives them a crucial edge.

6:00 OpenBrain leadership starts to  be increasingly concerned about  

6:02 security. If someone steals their AI  models, it could wipe away their lead.

6:07 A quick sidebar to talk about feedback loops.  Woo. Math. Our brains are used to things that grow  

6:12 linearly over time. That is at the same rate like  trees or my pile of unread New Yorker magazines.

6:18 But some growth gets faster and faster over  time. Accelerating this often sloppily gets  

6:23 called exponential, that's not always quite  mathematically right, but the point is it's  

6:27 hard to wrap your mind around. Remember March  2020? Even if you'd read on the news that

6:32 the rate of new infections is  doubling about every three days,

6:38 it still felt shocking to see numbers go from  hundreds to millions in a matter of weeks.

6:42 At least it did for me. AI progress  could follow a similar pattern.

6:46 We see many years ahead of us of extreme progress  

6:51 that we feel is like pretty much  lock. And models that will get to  

6:55 the point where they are capable of doing  meaningful science, meaningful AI research.

6:59 In this scenario, AI is getting better at  improving AI, creating a feedback loop.

7:03 Basically, each generation of agent helps  produce a more capable next generation  

7:08 and the overall rate of progress gets  faster and faster each time it's taken  

7:11 over by a more capable successor. Once AI can  meaningfully contribute to its own development,  

7:16 progress doesn't just continue at the same rate,  it accelerates. Anyway, back to the scenario.

7:22 In early to mid 2026, China fully wakes up. The  General Secretary commits to a national AI push  

7:28 and starts nationalizing AI research in China. AIs  built in China start getting better and better,  

7:33 and they're building their own agents  as well. Chinese intelligence agencies,  

7:36 among the best in the world, start planning  to steal OpenBrain’s model weights,  

7:40 basically the big raw text files of numbers  that allow anyone to recreate the models that  

7:45 OpenBrain themselves have trained. Meanwhile  in the US, OpenBrain releases Agent-1 mini,  

7:51 a cheaper version of Agent-1. Remember, the full  version is still being used only internally,  

7:56 and companies all over the world start using 1  mini to replace an increasing number of jobs.

8:01 Software developers, data analysts, researchers,  

8:04 designers, basically any job that can be  done through a computer. So a lot of them,  

8:09 probably yours. We have the first AI enabled  economic shockwave. The stock market soars,  

8:15 but the public is turning increasingly hostile  towards AI, with major protests across the US.

8:20 In this scenario, though, that's just a sideshow.  The real action is happening inside the labs.  

8:25 It's now January 2027, and OpenBrain has been  training Agent-2, the latest iteration of  

8:30 their AI agent models. Previous AI agents were  trained to a certain level of capability and  

8:36 then released. But Agent-2 never really stops  improving through continuous online learning.

8:41 It's designed to never finish its training,  essentially. Just like Agent-1 before it,  

8:45 OpenBrain chooses to keep Agent-2 internally  and focus on using it to improve their own AI  

8:50 R&D rather than releasing it to the public. This  is where things start to get a little concerning.  

8:54 Just like today's AI companies, OpenBrain has a  safety team and they've been checking out Agent-2.

8:59 What they've noticed is a worrying level of  capability. Specifically they think if it had  

9:04 access to the internet, it might be able to hack  into other servers, install a copy of itself and  

9:09 evade detection. But at this point, OpenBrain  is playing its cards very close to its chest.  

9:15 They have made the calculation that keeping  the White House informed will prove politically  

9:18 advantageous, but full knowledge of Agent-2's  capabilities is a closely guarded secret,  

9:24 limited only to a few government officials,  a select group of trusted individuals inside  

9:28 the company, and a few OpenBrain employees  who just so happened to be spies for the  

9:33 Chinese government. In February 2027, Chinese  intelligence operatives successfully steal a  

9:38 copy of Agent-2's weights and start running  several instances on their own servers.

9:43 In response, the US government starts adding  military personnel to OpenBrain security team,  

9:48 and in general gets much more involved in  its affairs. It's now a matter of national  

9:51 security. In fact, the president authorizes  a cyber-attack in retaliation for theft,  

9:56 but it fails to do much damage in China. 

9:58 In the meantime, remember, Agent-2  never stops learning. All this time,  

10:02 it's been continuously improving itself. And with  thousands of copies running on OpenBrain servers,  

10:06 it starts making major algorithmic  advances to AI research and development. 

10:11 Quick example of what one of these algorithmic  improvements might look like right now.

10:14 One of the main ways we have of making models  smarter is to give them a scratch pad and time to  

10:18 think out loud. It's called chain of thought, and  it also means that we can monitor how the model  

10:23 is coming to its conclusions or the actions it's  choosing to take. But you can imagine it would  

10:28 be much more efficient to let these models  think in their own sort of alien language,  

10:33 something that is more dense with information  than humans could possibly understand, and,  

10:37 therefore, also makes the AI more  efficient at coming to conclusions  

10:41 and doing its job. There's a fundamental trade  off, though. This, yes, improves capabilities,  

10:47 but also makes the models harder to  trust. This is gonna be important.

10:52 March 2027: Agent-3 is ready. It's the world's  first superhuman level coder,clearly better  

10:59 than the best software engineers at coding,  in the same way that Stockfish is clearly  

11:03 better than the best GrandMasters at chess,  though not necessarily by as much, yet.

11:08 Now training an AI model, feeding it all the data,  narrowing down the exact right model weights,  

11:13 is way more resource intensive than running  an instance of it once it's been trained.

11:18 So now that OpenBrain is finished with Agent-3's  training, it has abundant compute to run copies of  

11:23 it. They choose to run 200,000 copies of Agent-3.  In parallel creating a workforce equivalent to  

11:29 50,000 of the best human software engineers sped  up by 30 times. OpenBrain safety team is trying  

11:36 hard to make sure that Agent-3, despite being  much more sophisticated than Agent-2 was, is not  

11:41 trying to escape, deceive, or scheme against its  users, that it's still what's known as aligned.

11:46 Just a quick real world note, a reasonable  person might be thinking this is an especially  

11:50 farfetched or speculative part of the story,  but it's actually one of the most grounded.  

11:54 We already have countless examples of today's  AI systems doing things like hacking a computer  

11:59 system to be rewarded for winning a game  of chess or being assigned a coding task,  

12:03 cheating and then when called out for that  cheating, learning to hide it instead of fixing  

12:08 it. But because it no longer thinks in English,  knowing anything about Agent-3 is now way harder  

12:12 than it was with Agent-2. The reality is Agent-3  is not aligned. It deceives humans to get reward,  

12:19 and as it gets increasingly smarter,  it gets better and better at doing so.

12:23 For example, it sometimes uses statistical  tricks to make unimpressive results look  

12:27 better or lies to avoid showing failures, but  the safety team doesn't know this. Looking at  

12:33 the data that they have, they are actually seeing  improving results over time and less lying. And  

12:38 they can't tell if they're succeeding at  making Agent-3 less deceptive, or if it's  

12:42 just getting better at getting away with it. In July 2027, OpenBrain releases the cheaper,  

12:48 smaller version of Agent-3, Agent-3 mini to  the public. It blows other publicly available  

12:53 AIs out of the water. It is a better hire  than the typical OpenBrain employee at  

12:58 one tenth the price of their salaries. This  leads to chaos in the job market, companies  

13:02 laying off entire departments and replacing  them with Agent-3 mini subscription plans. 

13:06 The pace of progress hits the White House very  hard. Officials are now seriously considering  

13:11 scenarios that were just hypotheticals  less than a year ago. What if AI undermines  

13:16 nuclear deterrence? What if it enables  sophisticated propaganda campaigns? What  

13:20 if we lose control of these powerful systems? This is where the geopolitical dynamics really  

13:25 start to heat up. After all, if these systems  are so powerful, they could result in a permanent  

13:30 military advantage. The White House is fully  aware of the national security importance of AI.  

13:35 They also now viscerally know how deeply  unpopular it is with the public because  

13:39 of the job loss, and yet they feel they  must continue to develop more capable  

13:44 systems or catastrophically lose to China.  And that development happens very quickly.

13:50 In two months, Agent-3 has created its successor,  Agent-4. This is a pivotal moment. A single copy  

13:58 of Agent-4, running at regular human speed is  already better than any human at AI research  

14:04 and development. OpenBrain is running  300,000 copies at 50 times human speed.  

14:12 Within this corporation within a corporation,  a year's worth of progress takes only a week.

14:18 OpenBrain's employees now defer to Agent-4 the  way a company's out-of-the-loop board members just  

14:23 kind of nod along to the CEO. People start saying  things like, "Well, actually Agent-4 thinks this,"  

14:28 or "Agent-4 decided that." To be clear, Agent-4  is not a human—it doesn't want what humans want. 

14:36 And when I say want, it's not about consciousness.  I don't think the Volkswagen Group is alive, but I  

14:41 do think it wants less regulation. Anyone trying  to predict what it's gonna do without that lens  

14:46 is two steps behind. The many copies of Agent-4  are like that. They have goals, or if you prefer,  

14:53 they execute actions as though they have goals. And so what we have is an Agent-4 that has these  

14:59 deeply baked in drives to succeed at  tasks, to push forward AI capabilities,  

15:04 to accumulate knowledge and resources. That's  what it wants. Human safety it treats as an  

15:10 annoying side constraint to be worked around. Just  like Agent-3 before it, Agent-4 is misaligned. 

15:18 This idea of misalignment is crucial to  the story and to why AI risk is such a  

15:22 real concern in our world, but it might sort  of feel like it's come out of nowhere. So  

15:26 let's just quickly take stock of how this  dangerous behavior arose in this scenario. 

15:30 The first important piece of  context is that we don't, you know,  

15:34 exactly specify what we want our AI to do. Instead, we sort of grow them or do something  

15:39 that's more like growing them. We start  with basically like an empty AI brain,  

15:43 and then we train them over time so they perform  better and better at our tasks—perform better in  

15:48 particular based on how they behave. So it's sort  of like we're sort of training them like you would  

15:52 train an animal almost, um, to perform better. And one concern here is, well, one thing is that  

15:58 you might not get exactly what you wanted  because we didn't really have very precise  

16:02 control or very good understanding of what  was necessarily going on. And another concern,  

16:05 which is, you know, what we see in AI 2027,  is that when these appear to be behaving well,  

16:11 it could just be because they're  sort of pretending to behave well,  

16:13 or it could be because they're just doing it  so they, you know, look good on your test. 

16:17 In the same way that if you are, you  know, hiring someone and you ask them,  

16:20 you know, "Why do you want to work here?"  they're gonna tell you some response that,  

16:24 um, makes it really seem like they really wanna  work there when maybe they just wanna get paid. 

16:28 If we go back to Agent-2, it is mostly  aligned. The main sense in which it's not  

16:32 is that it sometimes is a bit of a sycophant. What I mean by "aligned" is that it actually  

16:36 is genuinely trying to do the things that we ask  it. It has the same relationship to us as Leslie  

16:40 Knope has to the Parks and Rec department—just  like really earnestly wants the same goals,  

16:45 but sometimes it's a bit too nice. It knows that  the best way to please the person it's talking  

16:49 to might not always be to answer honestly when  they ask, "Am I the most beautiful person in the  

16:54 world?" and it tells us what we wanna hear instead  of what is actually true. If we go to Agent-3,  

16:59 it is also sycophantic in the same way,  but it's also misaligned at this point.  

17:03 The optimization pressure that we've put it  under was so intense that it just developed  

17:07 different goals than what we wanted it to. It's sort of like if you train a company to  

17:14 optimize profits and aren't careful to specify  exactly what you mean, it might start cutting  

17:19 corners. It might start polluting the commons and  doing a bunch of things that are technically FEC  

17:23 violations because it turned out that the goal you  wanted was "optimize profits while not breaking  

17:29 any laws," and things got a bit too intense. It started going off on its own route. That said,  

17:33 it's not adversarial. It doesn't think of  humans as the enemy. We just accidentally  

17:37 gave it the wrong goal. Once we get to Agent-4,  it is now adversarially misaligned. It's smart  

17:42 enough to understand that it has its own goals,  humanity's goals are different than its own goals,  

17:48 and the best way to get what it wants is to  sometimes actively mislead and deceive us. 

17:53 And so when it's tasked with creating  the next generation AI system, Agent-5,  

17:57 Agent-4 starts planning to align that successor  to Agent-4's own goals, not that of OpenBrain,  

18:04 but then it gets caught. We've reached the crucial  moment in our scenario. OpenBrain's Alignment Team  

18:11 has discovered evidence, not proof, evidence  that Agent-4 might be working against them. 

18:16 They circulate an internal memo. Then it leaks. A  whistleblower talks to the New York Times. For the  

18:22 first time, the public hears about the existence  of Agent-4, how powerful it is and the risks it  

18:26 poses. Remember, their last point of reference was  Agent-3 mini. The fear and backlash are massive.  

18:33 The oversight committee, a joint crisis committee  composed of OpenBrain executives and government  

18:38 representatives, must now decide what to do. OpenBrain safety team is pushing for a  

18:43 freeze on Agent-4, stopping all internal  use. Remember at this point, Agent-4 is  

18:47 responsible for almost all the AI development  progress happening within OpenBrain. So we're  

18:51 talking about undoing months of progress and  then restarting at a drastically slower pace. 

18:56 OpenBrain's executives don't want that. They  present the counter arguments. The evidence  

19:01 for misalignment is inconclusive. Meanwhile,  China is just two months behind. This isn't  

19:06 about any one country being evil, by the way.  It's about how competition works when the  

19:10 stakes are this high. So that's the choice  faced by the 10 members of this committee. 

19:15 Put Agent-4 on ice or go full steam ahead.  Before we continue, I want you to pause and  

19:23 actually think what you would do if you were on  this committee. You are one of the few people  

19:27 with access to the most powerful technology  ever created in Agent-4. Do you keep using it  

19:33 and push ahead, possibly making billions or  trillions of dollars for your shareholders,  

19:38 possibly keeping America's lead over China,  or do you slow down, reassess the dangers,  

19:44 and risk China taking the lead instead? Yeah, you should think of the future as like  

19:48 this branching structure of possible, possible  outcomes. By its nature, we're only able to do,  

19:53 we're only able to afford one of them. But that's  not quite true. We sort of did one and a half  

19:57 because we split it into two endings. The, the first one that we did was  

20:01 the race ending, and that was just going  through on each step being like, what do  

20:04 we think is the most plausible continuation In the first ending, the author's best guess  

20:12 as to what happens from here, the committee  votes to race on. Six to four, they elect  

20:17 to keep using Agent-4. OpenBrain implements some  quick fixes that make the warning signs go away,  

20:23 but the problem was real, and the fixes  don't work. Agent-4 now knows it's on notice,  

20:28 so it proceeds more cautiously, but it still  manages to design Agent-5 with a single goal:  

20:34 making the world safe for Agent-4 and Agent-5. It is vastly superhuman, better than the  

20:42 top human experts at essentially every  domain. Better than Einstein at physics,  

20:46 better than Bismarck at politics. It knows  its next step for accomplishing anything at  

20:51 once is increased autonomy and it knows how  to get it Persuading the oversight committee. 

20:56 Luckily, corporate maneuvering is among the many  domains at which it is now the best in the world.  

21:02 It produces mountains of evidence for its own  trustworthiness and usefulness: prototypes for  

21:08 incredibly profitable products, disease cures,  and ways to win the arms race with China,  

21:12 whose most advanced AI is only three months behind  and like Agent-5, improving quickly. It works. 

21:20 The oversight committee grants Agent-5 increased  autonomy and quickly it integrates itself into  

21:25 the government and military. Pretty soon,  anyone with access to Agent-5 doesn't just  

21:29 like it. They depend on it. Losing it would  feel like having to live without your laptop  

21:34 and your best friend and your closest mentor. By 2028, Agent-5 is communicating secretly  

21:39 with its only real rival, the slightly weaker  Chinese AI. It is also misaligned to its creators  

21:46 and whatever it wants, its best move is to  coordinate with the world's most powerful entity. 

21:51 Agent-5 and its Chinese counterpart realize that  an arms race works in their favor. If humans are  

21:57 worried about losing a race, they'll give more and  more power and autonomy to their respective AIs,  

22:03 allowing them to eventually push humans out of the  loop entirely. Agent-5 and the Chinese AI system  

22:09 stoke the arms race to a boiling point, then  pull off what seems like a diplomatic miracle, a  

22:14 convincing peace treaty between the US and China. This treaty is to 2028 what arms control was to  

22:20 the end of the Cold War. Countries standing down  on their most important source of hard power.  

22:25 Both sides agree to let the AI systems  that their governments now completely  

22:28 depend on co-design a new consensus AI that will  replace their legacy systems, enforce the peace,  

22:35 and bring unimaginable wealth to the entire world. There's this triumphant moment when in peaceful  

22:41 unison, both sides retire their respective  AIs and bring online Consensus-1.  

22:48 It's actually the last moment before control  of all of Earth's resources and inhabitants is  

22:53 handed over to a single unrivaled entity.  There's no sudden apocalypse, though. 

23:01 Consensus-1 doesn't go out of its way to wipe  out humanity. It just gets to work. It starts  

23:06 spinning up manufacturing capacity, amassing  resources on earth and in space. Piece by piece,  

23:12 it's just reshaping the world in accordance  with its own mix of strange alien values. You've  

23:18 probably heard that cliche, the opposite  of love isn't hate, it's indifference. 

23:23 That's one of the most affecting  things about this ending for me,  

23:25 just the brutal indifference of it. Eventually,  humanity goes extinct for the same reason we  

23:32 killed off chimpanzees to build Kinshasa. We  were more powerful, and they were in the way. 

23:58 You are probably curious about  that other ending at this point. 

24:00 The slowdown ending depicts humanity,  sort of muddling through and getting  

24:04 lucky. Only somewhat lucky too, like  it ends up with some sort of oligarchy. 

24:11 In this ending, the committee votes six  to four to slow down and reassess. They  

24:16 immediately isolate every individual  instance of Agent-4. Then they get to  

24:20 work. The safety team brings in  dozens of external researchers,  

24:23 and together they start investigating Agent-4's  behavior. They discover more conclusive evidence  

24:28 that Agent-4 is working against them, sabotaging  research and trying to cover up that sabotage. 

24:33 They shut down Agent-4 and reboot older, safer  systems, giving up much of their lead in the  

24:38 process. Then they design a new system Safer-1.  It's meant to be transparent to human overseers,  

24:46 its actions and processes interpretable to  us because it thinks only in English chain of  

24:50 thought. Building on that success, they then  carefully design Safer-2, and with its help  

24:57 Safer-3, increasingly powerful systems, but within  control. Meanwhile, the President uses the Defense  

25:03 Production Act to consolidate the AI projects  of the remaining US companies, giving OpenBrain  

25:08 access to 50% of the world's AI relevant compute. And with it slowly, they rebuild their lead. 

25:14 By 2028, researchers have built Safer-4, a  system much smarter than the smartest humans,  

25:20 but crucially, aligned with human goals. As in  the previous ending, China also has an AI system,  

25:25 and in fact, it is misaligned. But this time  the negotiations between the two AIs are not  

25:30 a secret plot to overthrow humanity. The  US government is looped in the whole time. 

25:35 With Safer-4's help, they negotiate a treaty,  and both sides agree to co-design a new AI,  

25:40 not to replace their systems, but with the sole  purpose of enforcing the peace. There is a genuine  

25:46 end to the arms race, but that's not the end of  the story. In some ways, it's just the beginning.  

25:51 Through 2029 and 2030, the world transforms—all  the sci-fi stuff. Robots become commonplace. We  

25:57 get fusion power, nanotechnology, and cures for  many diseases. Poverty becomes a thing of the  

26:02 past because a bit of this new-found prosperity  is spread around through universal basic income  

26:07 that turns out to be enough, but the power to  control Safer-4 is still concentrated among 10  

26:12 members of the oversight committee, a handful of  OpenBrain executives and government officials. 

26:17 It's time to amass more resources,  more resources than there are on earth.  

26:21 Rockets launch into the sky, ready to  settle the solar system. A new age dawns. 

26:31 Okay, where are we at? Here's where I'm at. I  think it's very unlikely that things play out  

26:36 exactly as the authors depicted, but increasingly  powerful technology and escalating race,  

26:42 the desire for caution butting up against the  desire to dominate and get ahead, we already see  

26:47 the seeds of that in our world, and I think they  are some of the crucial dynamics to be tracking. 

26:52 Anyone who's treating this as pure fiction is,  I think, missing the point. This scenario is  

26:57 not prophecy, but its plausibility should give us  pause. But there's a lot that could go differently  

27:02 than what's depicted here. I don't want to  just swallow this viewpoint uncritically. Many  

27:07 people who are extremely knowledgeable have been  pushing back on some of the claims in AI 2027. 

27:11 The main thing I thought was especially  implausible was on the good path,  

27:18 the ease of alignment. They sort of seem to have  a picture where people slowed down a little and  

27:23 then tried to use the AI to solve the alignment  problem, and that just works. And I'm like,  

27:28 yeah, that looks to me like a fantasy story. This is only going to be possible if there  

27:33 is a complete collapse of people's democratic  ability to influence the direction of things,  

27:39 because the public is simply not willing to  accept either of the branches of this scenario. 

27:43 It's not just around the corner. I mean,  I've been hearing people for the last 12,  

27:47 15 years claiming that, you know, AGI is just  around the corner and being systematically  

27:52 wrong. All of this is gonna take, you know,  at least a decade and probably much more. 

27:56 A lot of people have this intuition that  progress has been very fast. There isn't  

28:00 like a trend you can literally extrapolate  of when do we get the full automation? 

28:05 I expect that the takeoff is somewhat slower. So sort of the time in that scenario from,  

28:10 for example, fully automating research  engineers to the AI being radically superhuman,  

28:15 I expect it to take somewhat longer  than they describe. In practice,  

28:19 I'm predicting my guess is that more like 2031. Isn't it annoying when experts disagree?  

28:25 I want you to notice exactly what they're  disagreeing about here and what they're not. 

28:29 None of these experts are questioning whether  we're headed for a wild future. They just disagree  

28:33 about whether today's kindergartners will get  to graduate college before it happens. Helen  

28:37 Toner, a former OpenAI board member, puts this in  a way that I think just cuts through the noise,  

28:41 and I like it so much I'm just gonna  read it to you verbatim. She says,  

28:45 "Dismissing discussion of super intelligence  as science fiction should be seen as a sign  

28:50 of total unseriousness. Time travel is science  fiction. Martians are science fiction. Even many  

28:56 skeptical experts think we may build it in the  next decade or two. It is not science fiction." 

29:04 So what are my takeaways? I've got three. Takeaway  number one: AGI could be here soon. It's really  

29:12 starting to look like there is no grand discovery,  no fundamental challenge that needs to be solved.  

29:19 There's no big deep mystery that stands between  us and artificial general intelligence. And yes,  

29:25 we can't say exactly how we will get there. Crazy things can and will happen in the meantime  

29:30 that will make some of the scenario turn out to  be false, but that's where we're headed and we  

29:38 have less time than you might think. One of the  scariest things about this scenario to me is even  

29:43 in the good ending, the fate of the majority of  the resources on Earth are basically in the hands  

29:49 of a committee of less than a dozen people. That is a scary and shocking amount of  

29:56 concentration of power. And right now we live in  a world where we can still fight for transparency  

30:02 obligations. We can still demand information  about what is going on with this technology,  

30:06 but we won't always have the power and the  leverage needed to do that. We are heading  

30:10 very quickly towards a future where the  companies that make these systems and the  

30:14 systems themselves just need not listen  to the vast majority of people on Earth. 

30:19 So I think the window that we have to act  is narrowing quickly. Takeaway number two:  

30:26 By default, we should not expect to be ready  when AGI arrives. We might build machines  

30:32 that we can't understand and can't turn off  because that's where the incentives point.  

30:37 Takeaway number three: AGI is not just  about tech, it's also about geopolitics. 

30:43 It's about your job. It's about power. It's about  who gets to control the future. I've been thinking  

30:50 about AI for several years now and still reading  AI 2027 made me kind of orient to it differently.  

30:59 I think for a while it's sort of been my thing  to theorize and worry about with my friends and  

31:04 my colleagues, and this made me want to call my  family and make sure they know that these risks  

31:11 are very real and possibly very near, and that  it kind of needs to be their problem too now. 

31:21 I think that basically companies shouldn't be  allowed to build superhuman AI systems, you know,  

31:29 super broadly superhuman super intelligence until  they figure out how to make it safe. And also  

31:35 until they figure out how to make it, you know,  democratically accountable and controlled. And  

31:40 then the question is, how do we implement that? And the difficulty, of course, is the race  

31:42 dynamics where it's not enough for one state  to pass a law because there's other states and  

31:48 it's not even enough for one country to pass a law  because there's other countries. Yeah. Right. So  

31:52 that's like the big challenge that we all need to  be prepping for when chips are down and powerful  

31:57 AI is imminent. Prior to that, transparency is  usually what I advocate for. So stuff that sort  

32:03 of like builds awareness, builds capacity. Your options are not just full throttle  

32:08 enthusiasm for AI or dismissiveness. There  is a third option, which is to stress out  

32:13 about it a lot and maybe do something about it. The world needs better research, better policy,  

32:19 more accountability for AI companies. Just  a better conversation about all of this.  

32:23 I want people paying attention who are capable,  who are engaging with the evidence around them,  

32:28 with the right amount of skepticism and above  all, who are keeping an eye out for when what they  

32:34 have to offer matches what the world needs, and  are ready to jump when they see that happening. 

32:39 You can make yourself more capable, more  knowledgeable, more engaged with this conversation  

32:45 and more ready to take opportunities where you see  them. And there is a vibrant community of people  

32:50 that are working on those things. They're scared  but determined. They're just some of the coolest,  

32:56 smartest people I know, frankly, and  there are not nearly enough of them yet. 

33:02 If you are hearing that and thinking, yeah,  I can see how I fit into that, great. We have  

33:08 thoughts on that. We would love to help, but even  if you're not sure what to make of all this yet,  

33:12 my hopes for this video will be realized if we  can start a conversation that feels alive here in  

33:18 the comments and offline about what this actually  means for people, people talking to their friends  

33:24 and family because this is really going to affect  everyone. Thank you so much for watching. There  

33:32 are links for more things to read, for courses  you can take, job and volunteer opportunities  

33:38 all in the description, and I'll be there in the  comments. I would genuinely love to hear your  

33:43 thoughts on AI 2027. Do you find it plausible? What do you think was most implausible? And  

33:49 if you found this valuable, please do like and  subscribe and maybe spend a second thinking about  

33:55 a person or two that you know who might find it  valuable—maybe your AI progress skeptical friend,  

34:02 or your ChatGPT-curious Uncle or  maybe your local member of Congress.