AI Safety Expert: Ban Superintelligence!

0:01 Connor, I remember reading that your middle name is actually John.

0:05 Are you John Connor?

0:07 The promised leader of the resistance, protector of humanity,

0:12 fighter of the machines, and how is this not your brand, your username?

0:17 Did you fire your marketing guy?

0:19 So, it's actually funnier than this.

0:20 So, my mother is German.

0:22 So, my middle name is actually not John,

0:23 it's Johannes, which is the German equivalent of John.

0:27 So, it's kind of like I'm like

0:29 one step like cobalistically removed, so to speak.

0:33 So, uh it's a it's a it's a funny brand.

0:36 When people find that out, it's always it's always a good laugh.

0:40 It is definitely meaningful in many ways.

0:42 Uh anyone who saw the Terminator series expects a lot from you.

0:47 But you started like many of us working on capabilities research.

0:51 I think you were trying to open source GPT2, maybe GPT3.

0:55 How did you flip from the dark side to safety team?

1:00 So I actually flipped to the well, you know, light side,

1:02 dark side depending on how you want to see it even before then.

1:05 Um, so it's a bit of it's a bit of a more, you know, nuance story than that.

1:08 So really when I first got involved with AI, I was like 16 or something.

1:13 So this is like a long time ago.

1:14 And you know, I was a teenager.

1:16 I was pretty stupid.

1:17 So um, my capabilities work was truly nothing of a threat to the world.

1:21 Let's say this is before even like this is before transformers.

1:25 This is kind of in like 2011 maybe.

1:29 So like really early like even neural networks

1:31 like not really that big of a thing.

1:33 Um and then I did get uh you know

1:36 it was pretty obvious to me pretty quickly that you

1:38 know if you could build something that's this powerful

1:42 you know which I was hoping for right like

1:44 something that can cure cancer and you know

1:45 that can solve all the problems in the world make

1:46 the world a better place that's a very dangerous thing

1:49 and like how do you even control something like that?

1:51 We don't even know.

1:52 So I was already interested in the problem of control

1:55 at the time when I started working on open source.

1:57 The the reason I worked on open source basically was that I grew up a hacker,

2:02 you know, self-hacker type of guy, you know,

2:05 open- source world, IRC world, you know,

2:07 just really kind of that kind of I was I was really steeped in this culture.

2:10 And one of the things of this culture is this kind

2:13 of belief that in the open source community, the hive mind,

2:17 you know, the larger world of hackers and safety enthusiasts,

2:21 as the best way to make software safe is to understand it.

2:25 It's not to hide it.

2:26 It's to make it open.

2:27 It's to make it accessible so you know people across the world can pull all

2:31 of their great talents and interests and so

2:33 on in order to make software better and safer.

2:35 This is kind of the thing that like I

2:37 was like very steeped into this culture at the time.

2:39 So at the time really what I believed was that you

2:44 know AGI was definitely going to come but it's still decades away.

2:47 You know maybe 2040 2050 this might start becoming a problem.

2:51 Um, but like I tell them I have a ton of time, you know,

2:54 I have a ton of time to, you know,

2:56 do a lot of research, you know, build a lot of career capital,

2:59 have a family, you know,

3:00 do all do all that kind of stuff and then so we'll be like,

3:02 you know, I'll be ready in 2040 when it becomes an issue.

3:06 And that all flipped in 2019, uh, when GPT2 first came out,

3:11 uh, when I saw GPT2, for me it was this, oh[ __] moment.

3:15 It's happening now.

3:18 It's happening much faster than I thought.

3:19 I didn't think we'd see some of GPT until two until like you know 2030s 2040s.

3:24 And then when I saw that I basically had to like just throw over all of my plans

3:28 and be like all right[ __] like I just need to like understand this right now.

3:31 And like the best way I knew to do

3:32 that at the time is to build it to build my own

3:35 to understand it and to you know hopefully tap into you

3:38 know the wider community of you know hackers security researchers etc.

3:43 And uh well that's not what happened.

3:46 um I became very burnt on the open source movement where for me

3:50 it was obvious is that we need to build these things you

3:52 know before they're dangerous like GBT2 is not going to end the world

3:56 you know and then we need to understand them very very deeply

3:59 we need to like take them apart we need to you know actually

4:02 make them safe before we can you know even consider working on something

4:06 more dangerous and that's truly just not what happened like no one

4:10 gave a[ __] everyone just wanted

4:14 things faster bigger your open source uncensored,

4:16 you know, like who cares what the consequences or who cares who gets hurt?

4:19 Like that's not my problem.

4:21 And so I got became very burnt on this like

4:24 ideology um that I grew up in as a kid.

4:26 Like you as a teenager, it's like libertarianism is like very appealing.

4:29 It's kind of like baby first politics.

4:32 Um but as you grow up, you just realize that this is like a very

4:35 just not a realistic way to run the society.

4:38 it's just like very childish and not commensurate

4:42 to the kinds of issues that we have to face here.

4:45 And so I moved away from the open source

4:48 movement as a result of this and you know continue

4:51 to do a lot of work on the technical side

4:52 of things then in you know more of a private

4:54 setting you know my own company and then eventually

4:57 now moved away from that as well towards more political

5:01 um work because I think that's the most pressing

5:04 and the most important thing to be working on right now.

5:08 So a few questions to follow up.

5:09 Do you think there is any specific reason argument

5:13 you see for why people like you maybe I discover

5:16 this kind of internally and switch and others even after

5:21 being told and debated still resist this pretty obvious conclusion?

5:28 I think there's no answer to this question

5:29 that doesn't you know shade into psychoanalysis basically.

5:33 I don't think it's a technical thing.

5:35 I think it's purely an emotional thing.

5:37 There's a thing whereas most nerds,

5:41 most tech people get into tech because they like tech,

5:44 because they like computers, because they like math, because whatever.

5:47 This is not really why I got into it.

5:49 I got into it because I like people

5:51 and they wanted the world to be a better place.

5:53 And this seems like the best way I could

5:55 make the world better for people and I could,

5:57 you know, help people live better lives

5:58 and solve problems that the world is facing.

6:01 I also like technology, sure, but it never was my core motivation.

6:04 like if in a sense for me it was just

6:07 like if this is not the thing that will help

6:09 the most people then okay I'll just go do something

6:11 else um that's all right um and I think this is

6:14 a very hard pill to swallow for many people because

6:16 I think many people don't really care that much like

6:19 that not their main motivation like their spiritual motivation um

6:23 you know I'm not saying you know not some motivation

6:26 there but like fundamentally I want people to be okay

6:29 I want people I want the world to be better

6:31 and I take this very very seriously And this often

6:35 means reckoning or like wrangling with you know uh problems

6:40 that aren't technical in nature but are like emotional or you

6:43 know even spiritual in nature in a sense where like

6:46 you have to just like for example confront that maybe

6:48 you were wrong and maybe you've done things that were bad.

6:50 I think I've dinged things that are morally bad like I

6:52 think I probably shouldn't have done uh as much open source

6:55 work as I think it was very reasonable at the time

6:58 given the information I had and the context I was in.

7:01 I think it was a very reasonable thing to do, but that doesn't mean it was good.

7:04 You know what I mean?

7:04 And so sometimes just you have to be able to say, "Oh yeah,

7:07 I was wrong or you know, I maybe shouldn't have done that or like, you know,

7:11 I um and other people shouldn't do it either." And that's just a hard it's

7:14 more of an emotional bottleneck to admit

7:16 that like maybe technology isn't a good thing.

7:18 You know, maybe it actually is a double-edged sword.

7:22 And um maybe you know maybe those extremely

7:25 annoying liberal arts people had a point about some

7:28 of the things they say you know um

7:30 and sometime and yeah there's just there's nuance here.

7:33 There's tribalism.

7:34 There's a lot of tribalism among nerds and tech people.

7:38 And of course there's the obvious, you know,

7:39 Upton Sinclair thing of just like it's very hard to make

7:42 a man understand something when his salary depends on him not understanding it.

7:45 Like I think people just truly do not

7:48 appreciate how true this is even of themselves.

7:50 like how much you can trick yourself into just like ah you know

7:54 it's you know it's kind of reasonable it's not so bad is it

8:00 um so yeah I think all these factors matter a lot um I

8:03 think culture matters a lot if all your friends are tech people if

8:06 your employer is tech people if your you know spouse works in tech

8:11 and so on just kind of like reckoning with this idea that like

8:14 maybe we're the bad guys it's just like actually really hard to do

8:18 and if you're very smart you're very good at coming up with excuses.

8:21 So you can always come up with some excuse about why actually it's okay.

8:26 Actually it's fine.

8:26 Don't worry about it.

8:28 Um so you know as I say it it it shades a lot of the psychoanalysis here.

8:32 I think the cynical take is just like it's where their check and their you

8:35 know ego comes from and getting over people's

8:37 money and ego is just like pretty hard.

8:40 Stock options not even salary stock options.

8:44 You switched to safety.

8:46 Do you feel you made important contributions in technical

8:50 safety or is you leaving safety now for governance

8:54 for lobbying an indicator that it is kind

8:57 of dead end when it comes to advanced AI?

9:01 I I did a lot of work that I'm very proud of that helped me

9:04 a lot in my thinking of how these things work and like how hard this problem is.

9:09 But fundamentally, I don't think I advanced the field by any significant amount.

9:13 for the matter of fact I don't think basically almost anyone has in the last

9:17 years um you know sorry to all the people working on this field

9:22 but we have made almost zero progress on the idea on the actual hard problems

9:25 sure we have a bunch of papers

9:27 about like some statistical distributions of sparse autoenccoders

9:31 or some[ __] but like this is just completely irrelevant to the actual kind

9:34 of problems we need to solve um I think I have deconused myself a lot

9:38 in terms of like why this problem is hard and like why the things

9:41 that are we're currently mine don't even

9:43 have a chance of working even in theory.

9:45 Um I think this is very valuable.

9:48 Um at least to me, but fundamentally I think the idea

9:50 that we could solve safety of super intelligence is just ridiculous.

9:55 Like it's like it's so like to me like it's like such

9:57 a it's like a it's like a Marvel comics kind of thing.

10:00 It's like something you'd see in a in like a in like a silly movie, right?

10:03 Like this is not how real world things work at all.

10:06 It's kind of like saying we're going to build

10:09 a world government that can solve all problems of industry,

10:13 of morality, of psychology, of politics, of game theory, of physics, of biology.

10:20 So all of these problems in one go using software and there will be no bugs.

10:28 And like I'm sorry if this is the spec of your project.

10:30 This isn't a serious prop.

10:31 This is not a serious project.

10:33 This is not how you actually like you cannot this can't be done.

10:36 You cannot build a system like this.

10:37 It cannot be done in any sensible way.

10:40 Um and if you believe that you can execute

10:41 on a spec like this like I'm sorry you're delusional.

10:44 Like it's just not doable.

10:45 And this is what it would mean to build an aligned super intelligence.

10:48 Building a super intelligence that like does what

10:50 you want would solve all of these problems.

10:52 And if you think you have a spec that can do this like I'm sorry you don't.

10:56 Like this is a problem that you know I'm open to the idea

10:59 of if all of our greatest scientists with trillions of dollars of budgets worked

11:04 on this problems for three generations I'm open to it like maybe right

11:09 but like anything less than that it's I'm sorry it's just not it's not possible.

11:13 So you're saying theoretically the problem is

11:16 solvable just very hard in practice we don't

11:18 have time brains resources to do it in a time frame I if you think even

11:23 theoretically it is possible can you give me an idea kind of a pseudo code level

11:27 description of what that means to align

11:30 or control super intelligence what that world looks like

11:34 it's very hard to imagine and I think a lot of good epistemology

11:38 is not thinking about things that break your epistemology um Like a lot

11:43 of a lot of the reason people are bad at this is because

11:47 it involves thinking about things that we're

11:48 very confused about such as morality.

11:50 You cannot address a question of like governance or morality or like

11:56 so on or a triage without thinking about moral philosophy to some degree.

12:01 And humanity as a whole is extremely confused about morality.

12:04 It's not just we haven't solved it.

12:06 Like we don't even know what it means to make progress on this.

12:08 Like we can't even agree on like the basic epistemological

12:11 foundations of how do we make progress on this problem.

12:14 And um questions of like alignment and super

12:19 intelligence are not questions about computer science.

12:22 There are questions involved about computer science but many

12:25 of the questions are much closer to governance.

12:27 They're much closer to how do you design a state?

12:29 How do you design a large entity?

12:31 you know, under game theoretic considerations that gives,

12:35 you know, that can, you know, function under, you know,

12:38 complex moral dilemmas where often there is no solution.

12:42 Like often if you're a government, you're doing triage, right?

12:45 There's only so much money to go around.

12:47 There are infinite good projects you could work on.

12:49 How do you triage?

12:50 How do you triage?

12:51 How do you handle, you know, war?

12:53 How do you handle law enforcement?

12:55 How do you handle like these are like many, many things.

12:57 The world is actually very complex.

12:59 Like this is a thing that you know um as a kid you know

13:03 you'd always try to be resistant to is like ah yeah the old people say

13:06 everything is complicated but I got to figure out but no the world in fact

13:09 has many parts not infinitely like there are you know but like it is

13:14 actually a lot if you build a super intelligence that is aligned this is

13:18 a system that is so powerful

13:19 that it would fix everything you know hypothetically

13:22 that it could and it would do so in a way that we would endorse

13:25 and thing is we're currently too confused

13:27 to even consent to Like there's a thing

13:30 where like the reason consent varies by like

13:32 you know people's mental ability and like

13:35 maturity and so on is because just like there are things that are too

13:38 confusing for example for a child to consent to you know whether that's you

13:41 know uh you know drugs or surgeries or sex or whatever like there just like

13:46 things that are dangerous and like

13:48 a child can't understand this and so obviously

13:51 we have to shield them from engaging

13:53 in such activities and there's a similar thing

13:56 here that also applies to adult humans where Like the idea that we could

13:59 like consent to something reshaping all

14:02 of the world and solving all of moral philosophy, like we can't consent to this.

14:05 Like we're too immature.

14:07 Our our understanding as a society of what this would look like

14:10 and like what this would mean is so immature that like we couldn't know,

14:16 we couldn't understand it.

14:18 So the way it would really look in practice, right,

14:21 is if you actually made progress on the actual alignment problem,

14:25 you would on the way there figure out how to build much more just institutions.

14:31 You would figure out how to bake much more capable

14:35 governments that are much more fair and much less corrupt.

14:38 You would figure out how to solve largecale disagreements among humans.

14:43 You would like better debate platforms or something like this.

14:46 You would solve uh political polarization.

14:50 You would get really good at handling mimedic outbreaks for example.

14:54 Like you would be really good at knowing how do you prevent

14:57 like dangerous or like you know um false you know parasitic ideas

15:01 from like affecting people at scale because you need to solve all

15:04 these problems in order to build you know a a system this powerful.

15:09 You would also solve cyber security.

15:11 You know, you would learn a lot about how to build

15:14 complex systems operating in adversarial environments

15:17 in ways that they don't break.

15:19 You know, you would have to make

15:20 massive progress unlike formally verified programming.

15:23 You would have to make massive progress on modeling social systems,

15:27 psychological systems, economic systems, etc.

15:29 You would have to make massive progress on, you know,

15:33 unsolved theoretical problems of like how embedded agents

15:36 in the environment think about their own mind.

15:39 like you would have to make progress on so many different problems.

15:42 Like there's just like you would have to solve like you know most of science

15:47 kind of to get to something that we

15:50 would actually have that justified belief in trusting.

15:53 I think you can definitely grow something

15:55 in a digital petri dish that is this powerful.

15:57 But if you want to have confidence that you know what you're doing and that is

16:01 actually as possible like this is

16:04 the hardest scientific problem humanity has ever faced.

16:06 I don't think it's necessarily impossible, at least, you know, up to some bound.

16:09 You know, I'm not saying 100% is definitely possible because,

16:12 you know, the world is chaotic, but like getting it to a bound that we

16:16 might be able to consent to, I could imagine it,

16:19 but like it would take generations.

16:21 Can you translate your beliefs about difficulty or impossibility

16:25 of that problem and poom estimates 5 years, 10 years, 20 years out?

16:32 Um the way I generally think of this is that um

16:35 there's like you know pdoom if we don't do anything

16:37 and this pdoom if we do do something I think if

16:40 we don't do something it's you know very close to 100%.

16:44 You know maybe 90% or 95 or something just to account for uncertainty.

16:48 Um, but like the way things are current

16:50 stand there's like there's no way this goes well.

16:52 Like there's just no realistic world where just like we build

16:55 the first things we can as quickly as possible with zero oversize,

16:59 zero thinking, zero planning,

17:01 zero risk mismanagement and just pump it all out as fast as possible.

17:05 This cannot go well.

17:06 Like unless there's like some crazy facts about reality, you know,

17:12 maybe aliens come from outer space and fix it for us.

17:15 But like like I'm sorry like this is just

17:18 not again this for me this is like Marvel you

17:20 know like comic book nonsense like the idea that this could

17:24 go well for me is just like very childish.

17:27 Um if we do do something I think it's very different.

17:30 I think it's absolutely possible um for humans

17:32 to solve like public goods problems like these.

17:35 I think it is possible um for say you

17:38 know governments to say the current level of risk

17:40 that is being externalized onto you know the general

17:43 public is unacceptable and we need to step in.

17:46 We need to uh like not build things that we think are

17:50 this dangerous until we're ready for it and I'm not saying it's

17:54 easy by any means but this is a thing that can be

17:56 done and we do have institutions that have the power to do it.

18:00 This is not this is not always the case, right?

18:02 Like many times in history,

18:04 there were no entities like governments that were, you know,

18:07 powerful enough to be able to regulate things like this, but they do exist.

18:11 They're dysfunctional in various ways, but they do exist and it can work.

18:15 And I think if we take it seriously

18:17 um and you know we slow down the development,

18:20 we don't develop these super dangerous things or we only do it

18:23 like in a very very high security context and we do this internationally.

18:27 I think I think we could do it like I think I think it's possible.

18:31 I think it'll be very messy.

18:32 There's no way the 21st century isn't messy.

18:34 But like that that you know if we get

18:36 a if we get a global ban on super intelligence,

18:39 you know that puts my P doom down like a lot.

18:42 I don't even know how much but like a lot.

18:45 Why do you think AI safety community as a whole is a lot more optimistic?

18:49 They really feel they can actually find a technical solution in many cases

18:53 because their salary depends on them believing that that is the only reason.

18:57 There is no scientific technical reason to be optimistic from what you've seen.

19:03 Like there's this thing where like um scientific sounds

19:06 objective but it's only objective in a paradigmatic field.

19:10 uh if you're in the pre-paradic field like obviously

19:14 you can generate a scientific looking paper that confirms

19:18 whatever they believe that's the problem is that like

19:20 it's the same problem we have stand psychology like

19:23 I can come up with you know basically any

19:26 theory of psychology and I can I can generate

19:29 a scientifically you know p equals 0.05 05 whatever

19:32 you know paper that justifies whatever I was saying.

19:35 Um the epistemology of science is like is

19:37 actually harder than this and because we have such

19:40 horrible understanding of what's actually going on here

19:42 and it is actually very subtle anyone can basically

19:45 justify anything they want with a sufficiently complicated

19:48 argument and the the the correct step to do

19:52 here is is not to be like well I have more papers than your paper or something.

19:56 It's just to be like take a step back.

19:58 Obviously we can't agree here so we should be careful.

20:01 The only thing all experts can agree on is

20:03 that there's a chance that this goes terribly terribly catastrophically wrong.

20:06 And the only thing we can all agree upon

20:08 is that not building super intelligence would prevent the risk.

20:11 That's the only thing we can agree upon, you know,

20:14 and then we the correct decision theory here is to stay

20:17 step back and be like whatever the truth of the matter is,

20:20 we don't know what the truth of the matter is.

20:22 We're not close to having human resolve these conflicts and therefore

20:25 we should take the cautionary approach because the risks are so high.

20:28 So, it's really more of a strategic thing here

20:31 where it's like it doesn't matter what they publish.

20:33 Like it their technical arguments like people will tell me, "Okay, Connor,

20:36 but have you seen the latest paper that proves?" I'm like, "It doesn't matter.

20:39 I'm not going to read it.

20:40 It's irrelevant.

20:41 It doesn't matter." So, I can completely agree with you.

20:45 I've been kind of preaching the same story for a while.

20:48 But the next logical attempt is to do something with governance, with lobbying.

20:53 I think that's what you are doing with your organization.

20:56 Can you tell me about the organization and what you're doing in DC with it?

21:00 Yeah, so I am now the US executive director of Control AI.

21:04 Um, so it's a nonprofit organization with presence here in DC,

21:08 the UK, Canada, and Germany.

21:10 Um, it's been around for a couple years.

21:12 I've been on and off involved as an adviser and now I'm here full-time.

21:17 Really what has pushed me to uh do this now

21:20 full-time here and have now moved to Washington DC is

21:24 this you know realization is understanding that there is no technical

21:27 solution here and that the this is fundamentally a political problem.

21:33 The problem we're solving is that we need to solve is a political problem.

21:36 The question isn't you know as we've been talking about like oh you know do we

21:39 use this technique or that technique to make

21:41 our AI safe whatever the hell that means.

21:43 It's much more who gets to make these choices.

21:45 Who gets to decide what level of risk

21:47 the public is exposed to if we aren't unsure?

21:50 If we don't know whether a nuclear reactor will or will not melt down,

21:54 who decides whether we build the nuclear reactor?

21:57 And in every other industry, and every other risky industry, whether that's,

22:01 you know, uh, you know, the weapons industry or nuclear or whatever,

22:06 you know, even just like normal industrial processes or even restaurants,

22:09 you know, this is decided by the government.

22:12 This is we that you the government has the mandate

22:15 uh you know the mandate to protect and set

22:19 the limits of what level of risk the public

22:22 is exposed to from various types of industries and activities.

22:25 It is illegal to attempt to build a bomb like in your garage.

22:29 Uh even if you fail, even if the bomb doesn't go off, it's still illegal to try.

22:33 It's still illegal to do that because you are

22:35 putting other people at risk by doing this activity.

22:38 Now we do have factories that make bombs.

22:41 That is definitely a thing and the government

22:43 decides how safe those factories have to be,

22:46 who gets to build them, how it works, etc.

22:49 And there's and for some reason, well, we can talk about the reasons,

22:53 this is just not the case in AI.

22:55 Uh, currently there is more regulation

22:57 on selling a sandwich to the general public

23:00 than there is on building super intelligent

23:02 AI that could kill literally everyone on Earth.

23:05 And these companies are lobbying extremely hard to keep it that way.

23:09 this they have assembled the largest lobbying force in history like

23:12 the some of the largest political donations you know super PACs

23:15 etc ever in order to keep it this way so

23:19 that they can race for super intelligence um unhindered and this is

23:25 not the will of the people this is not what I

23:28 or you know the general public you know here in America

23:30 or abroad want and so I'm here together with control

23:35 and others to help the government you know understand these problems

23:38 realize what's happening and what can be done about it

23:41 and to catalyze the action necessary to get us to a true

23:45 international ban on the development of super intelligence anywhere because

23:49 fundamentally this is a grave national security risk whether it's

23:53 a domestic US company building a recklessly dangerous system that harms

23:58 you know you know citizens or it's a geopolitical rival such

24:02 as China building such systems it doesn't really matter if

24:05 a super intelligence uncontrolled is built by anyone on anywhere on earth.

24:09 This is an unacceptably high level of risk to you know

24:13 the American people and everyone on earth and so the government must act.

24:17 I want to set up a few thought experiments.

24:20 So the first question would be are you a US citizen?

24:24 Yes.

24:23 So that that helps a lot.

24:25 So let's say tomorrow you wake up in a body of the president of United States.

24:29 You have complete executive power.

24:32 Any executive orders you want.

24:33 What is it you actually writing?

24:36 What are you signing?

24:37 So there's a couple things.

24:39 Um, one of the first things is that I would

24:42 get on national TV immediately and say it is now

24:45 the the priority of the of the American state

24:48 to make sure that super intelligence is not built by anyone.

24:50 Go live on TV and then I'd set a memo

24:53 to all the heads of all the different departments,

24:54 you know, Department of War, Department of Energy,

24:57 you know, uh, Department of Homeland Security.

25:00 And I tell them all it make me a plan right now how we prevent anyone

25:05 on Earth from building um super intelligence and like get it to me by next week.

25:09 I want it on my desk.

25:10 That would be what I would do.

25:11 So I I get the CIA, I get the NSA,

25:13 I get the you know um the military, I get everyone.

25:17 You're making doctrines now of how do we prevent both domestic actors,

25:21 you know, international rivals and rogue

25:23 actors from building super intelligence.

25:25 You figure that out where the hell now and get it on my desk by next week.

25:28 That would be the first thing I would do.

25:30 Um, do you think they would have to nationalize our labs at least?

25:35 I think there are many ways to skin a cat.

25:37 I think there are many ways to approach problems like this.

25:39 I think nationalization is one instrument, but there are others as well.

25:43 You know, there are a lot of things of how we,

25:46 for example, regulate uh weapons manufacturers.

25:49 So, a lot like Loheed Martin for example, you know, like you know,

25:53 the classic Area 51, you know, is a highly secure facility.

25:57 um that you know does not leak information you know except the occasional

26:00 UFO um and is you know the kind of level of security

26:05 that you might want when building systems like this and it's mostly

26:08 run by Loheed Martin which is a private company um so we do

26:12 actually have a lot of experience with regulating companies um to the degree

26:17 that we can trust them to have like very high info security

26:20 that they are building what the United States actually needs and not some

26:23 other thing that they're not leaking these capabilities to our advis etc etc.

26:28 So there's a lot of a lot of framework we can already build on.

26:31 Um nationalization might be necessary but I think this is a very much

26:36 like a technical detail that um doesn't really matter for the overall thing.

26:40 Fundamentally what we need is not a specific piece of legislation.

26:43 What we need is a sufficiently large coalition of powerful people, you know,

26:47 such as the president, um, but also the general public, you know,

26:51 the heads of various agencies and so on to want ASI to not exist and then

26:58 use their capabilities to figure out the best

27:01 tools at their disposal to achieve this objective.

27:04 Fundamentally, the policy objective must be that super intelligence

27:08 never exists or at least not until we're ready.

27:10 And then we have to consider that we

27:12 will need a manypronged approach because of course

27:15 these companies rogue actors you know criminal

27:17 elements you know you know rogue states such

27:19 as Iran and North Korea will attempt

27:21 to acquire these capabilities and so we will need

27:24 a many many prong approach to be able

27:27 to how can we prevent these kind of things?

27:29 How can we create a trust verify regime with countries such as China?

27:33 Uh where we ultimately need an international agreement of some

27:37 sort that all of us do not pursue such capabilities,

27:40 do not build super intelligence.

27:42 But you know we should and we should trust but we should also verify.

27:46 It's very important that we also have the technical verification

27:49 methods so that you know say the Chinese state can prove

27:52 to us that they are not building super intelligence and that we

27:55 can prove to them that we are not building super intelligence.

27:57 And all of this just requires truly many moving parts.

28:01 I think this is a thing that is doable.

28:03 And that's why I say like if I was the president,

28:05 the first thing is I would take all of my you know ministers and make clear

28:10 that this is our priority and that they

28:13 need to figure out how to make this happen.

28:16 If uh instead of being a government leader,

28:18 you were in a position of Sam Alman or Elon Musk,

28:22 how would your actions be different?

28:25 Uh very different.

28:26 Um, fundamentally I would immediately resign.

28:28 Uh, that would be the first thing I

28:30 would do because you can't function in this role.

28:33 Like if your job is make super intelligence,

28:36 you can't become the anti-super intelligence guy because you lose your job.

28:39 Like even Sam Ultimate or Elon can't

28:41 really do that without crippling their own company.

28:44 Um, and so the first thing you would need to do is, you know, quit your job.

28:48 Then what I would do is if I was

28:50 say Sam ultimate is I would take my prodigious wealth,

28:53 connections, charisma, intelligence, you know,

28:57 and make it my priority to make sure this happens.

29:00 I call in all the favors I can with all government officials I know.

29:03 I would start a media campaign.

29:05 I would, you know, maybe build an organization or donate a lot

29:08 of money to organizations to make sure these kind of things happen.

29:10 I would go on national TV and say I was the CEO of these companies and this is

29:15 what we need to do and I call you

29:17 know upon the government to make these happen etc.

29:19 Um yeah, those are the things that I would do.

29:22 It's because again only the government can has the the dour

29:28 kind of ability to even do what needs to be done here.

29:30 Even if say Sam Alman wanted to pause or something he can't he doesn't have

29:36 the capability to do this and especially

29:38 he doesn't have the capability to enforce it.

29:40 So even if Sam, Elon, Demis,

29:42 all these people come together and they all make a pack,

29:43 they all shake hands and like yes,

29:45 we will not build super intelligence and we're going to pause right now.

29:48 Doesn't matter because a they can't force enforce it amongst each other.

29:52 They don't have law enforcement capabilities.

29:54 So if one of them defects,

29:56 how will the others know A and B, how will they stop them?

29:59 Like it's not illegal, so they can't send the police.

30:02 What are they going to do?

30:04 Nothing they can do.

30:05 Um that's why we have law enforcement is

30:07 that law enforcement can make you stop because it's illegal.

30:10 And even if these companies stopped, you know,

30:13 will that make companies in China stop?

30:15 Will that make other American companies stop?

30:17 Will that make, you know, Yan Lakun's new company in France stop or something?

30:22 No, obviously not.

30:23 And even if all of these companies stop,

30:25 the next batch of startups is already going and like they will

30:28 come into being and they will do it as long as it isn't,

30:31 you know, we don't have a both domestically and internationally

30:33 forced agreement that stops super intelligence and makes it illegal.

30:38 We've seen many employees of those companies,

30:40 especially in the safety departments, quit,

30:43 resign in protest, resign because they think

30:47 they'd rather go write poetry for a year.

30:49 Do you think it's the right decision or is it

30:51 better to stay and try to slow it down from inside

30:56 for all these people?

30:57 Obviously, you should quit.

30:58 Let me be I'm going be unambiguous here.

31:00 If you work an AI company, um quit your job.

31:02 You should quit it right now.

31:04 Um the there's a couple reasons for this.

31:06 One is fundamentally if you're doing a bad thing, stop.

31:10 You know, just like deontology, standard morality.

31:14 If you find yourself working, you know, at a death camp, stop.

31:19 Like, first thing, stop, drop, and roll.

31:21 You know, just like stop making things worse.

31:24 That's the very first thing you should do.

31:25 Stop making things worse.

31:27 The second thing is is that this idea

31:30 of like doing things from the inside has never worked.

31:33 Like people absolutely suck at this.

31:35 Look, if you are a brilliant CIA sociopath lizard man, you know,

31:40 and you want to like break the law by sabotaging internally,

31:43 like maybe, but like hell no.

31:47 Like this is not how we do things in this world.

31:49 Like this is not how we do things in the west.

31:52 We don't this is vigilante justice is what this is.

31:56 And we don't do vigilantism.

31:57 This is not how an orderly society works.

32:00 So if your goal is to do vigilantism, I say no.

32:04 this is not how we do things.

32:06 It's just going to make things worse and it will not help.

32:09 But really, most people when they take jobs like this, it's just it's just

32:14 their conscience and that like they want to make a big amount of money

32:17 and they want to hang out with cool people and talk about tech all

32:19 day and they want to be in the room because it makes them feel good.

32:22 They're high status, you know, they're it makes them feel famous.

32:25 But this is a thing that these companies exploit very heavily.

32:28 They're not stupid.

32:29 They make you feel like, "Oh yeah, we're definitely listening to you.

32:33 Wow, you're making such a difference.

32:34 Well, they put you in a little terrarium

32:36 where you're completely insulated from anything actually ever happening.

32:40 And then if you ever actually do something,

32:42 as many of these people have found out, suddenly it stops working.

32:46 Like suddenly they hit all these barriers and then they quit.

32:50 So basically, I think these people are,

32:52 you know, either willfully or non-willfully just being diluted.

32:56 Um these companies are extremely optimized

32:58 as an entity to prevent exactly this thing.

33:01 Like why would you think that going into the territory

33:04 controlled by the thing you're trying to, you know,

33:08 prevent would somehow be the easiest way to act.

33:12 And especially if these companies are like advertising to you,

33:15 that obviously means that this is a trap.

33:18 Um, if you're playing chess and your opponent says, "Wow,

33:21 that's such a great move." That's not good for you.

33:27 Governance should work really well for large projects.

33:30 something you can monitor easily.

33:31 Manhattan style huge compute.

33:35 But every year compute becomes more efficient, algorithms become more efficient.

33:40 How long before none of that matters?

33:42 Anyone with a laptop can create super intelligence?

33:45 I don't know.

33:46 I think if it's possible to build super

33:49 intelligence on a laptop and next couple years, it's probably just game over.

33:53 Like I think it's just nothing we can do basically.

33:56 I think yeah it's this is a fact about computer science more so than anything.

34:01 If it turns out that just like there is an algorithm just like

34:04 runs on a MacBook and it just builds super intelligence in one go

34:08 and we discover it in the next say you know 10 20 30

34:11 years then I think we're just screwed and there's basically nothing we can do.

34:14 It's just too like that is just too crazy.

34:16 You would need to burn every single CPU on the planet

34:18 and this is just like not a thing that can be done.

34:20 Um, in that case, I would recommend retiring

34:22 to a tropical island with your family and, you know,

34:25 enjoying some good time off.

34:27 I would be surprised if this algorithm existed

34:30 or if we would discover it in the next, you know, 20, 30 years.

34:34 Um, it's not my main expectation, but I don't know.

34:37 It's definitely possible.

34:38 Um, I do expect it will remain a significant

34:42 engineering problem at least for the near future, but not for much longer.

34:46 We are very close to AGI.

34:48 We are very close to ASI.

34:50 we are and once we cross the event horizon of you know even a single ASI

34:55 existing it's too late you know it can be copied you know the technique can be

35:00 improved etc so I think if we were as a civilization three steps wiser than you

35:08 know we were we would have never done chat GBT in the first place if we were

35:12 two steps wiser than we currently are

35:14 we would have done chat GBT immediately seen

35:16 that as like this crazy thing they can

35:18 do all this crazy stuff got a million user

35:20 overnight and we would have been we would have been like oh no no no no

35:23 shut it all down and studied it very

35:25 very carefully for years before slowly rolling it out.

35:28 If we were one step wiser than we currently are we stop right now because

35:32 every step forward we take it also

35:33 means makes turning back harder exponentially more expensive.

35:37 Um if we get to the point where like I I think there will be

35:41 a point that even if we stop now

35:42 algorithms continue to improve where like very dangerous

35:46 AI systems can be built on very small

35:48 amounts of compute and the kinds of mechanisms

35:52 needed to deal with this kind of stuff

35:53 are the things that take decades to build.

35:55 Um and we don't even know what those would look like yet

35:59 and so we need the time to even be able to figure that out.

36:03 A lot of red lines proposed by I safety

36:06 community as long as decade ago have been crossed completely.

36:09 The systems are lying, cheating, escaping, blackmailing.

36:13 Are there any remaining red lines which you are

36:15 particularly concerned about which have not been crossed yet?

36:19 I mean you can always come up with something but I basically agree with you.

36:22 It's like we have crossed so many red lines and we constantly do it.

36:26 We now have systems such as Mythos which are you know

36:30 more capable at hacking than like you an NSA strike team.

36:34 This is like it's hard to overstate

36:37 how many red lines have already been crossed.

36:40 And this is why I think a lot of this you

36:43 know thinking that like oh there'll be a warning shot.

36:46 Oh once a big thing happens everyone will act.

36:48 And I think this is just completely false.

36:50 Empirically we've crossed massive red lines like you know the touring test.

36:53 If you remember when everyone said once a touring guest gets passed

36:56 then we know it's serious no one even cared when it happened.

37:00 Now we have systems, you know, that are like on par with like the NSA

37:03 when it comes to hacking and stuff like that.

37:05 Like actual nation state level cyber weapons, you know,

37:07 systems that are like as dangerous as, you know,

37:10 more dangerous some sense than missiles, you know,

37:13 and, you know, we just keep going.

37:15 So I think there are, you know, some levels like yes,

37:19 AI is it's not yet AGI, you know,

37:21 it can't yet self recursively self-improve itself.

37:24 It can't yet run fully autonomously 24/7 for years at a time.

37:30 Yeah, but I think not many of them left.

37:34 If you look at kind of OGs of AI safety,

37:37 Yutokavski, Hugo Dearis, what mistakes do you think they made?

37:41 They had a lot of early advantage.

37:43 They started, you know, 20 years ago.

37:47 Is there some missed opportunities we can learn from?

37:50 Likewise, in your career,

37:52 do you feel you made a big mistake which could have made a difference?

37:56 I think the biggest mistake fundamentally is that the entire AI safety

38:00 movement was built on a corrupt

38:03 ideology of like libertarianism and utilitarianism.

38:08 And these are just the wrong morals.

38:11 They're the wrong ideology.

38:13 If you build a system on these, they cannot handle these types of problems.

38:16 There's a reason that our governments

38:18 and our civilizations are not built on these norms

38:21 because they don't work and they can't

38:23 handle these types of large scale coordination problems.

38:28 There's a really fascinating essay from Eleazar, I think from like 99 or 98.

38:32 He was very young when he wrote it, so he I'm sure he disavows it.

38:35 Let's not, you know, harp on him too much.

38:37 Like Alzar is truly someone who has changed his mind and has updated

38:40 a lot and I have a lot of respect for him um for this reason.

38:44 Um but it's a really telling piece where

38:47 he talks about like super intelligence and like

38:49 you know I will change everything blah blah

38:51 blah and towards the end of the essay

38:53 he says like okay so like what should we do one of the things he

38:57 says like in big bold letters is importantly

38:59 so we don't tell the government and we

39:02 don't tell the general public because they're

39:04 stupid and they might panic and you do

39:06 something else we the smart people have to do it in secret we have to do

39:11 it right and this is fundamentally built built

39:14 on this libertarian seed of ide ideology where

39:18 like this like it's very funny like I I I was talking to some Japanese people.

39:23 We had a little AI safety conference in Japan a couple years ago.

39:26 I was talking to some Japanese people

39:27 and we're explaining the concept of like effective altruism,

39:30 AI safety, you know, MIRI and this kind of stuff to them

39:33 and the idea of catastrophic risk made perfect sense to them.

39:36 Yep, that seems makes sense.

39:38 That seems really bad.

39:39 We definitely should do something about that.

39:41 But this but there's one thing that was really confusing

39:44 to them and they're like but why are there charities involved?

39:49 Why are charities or private companies involve?

39:52 This is something the government the military should be handling.

39:55 Why are there private charities funded by like some

39:58 philanthropist handling threats of this level to entire nation states?

40:02 That doesn't make any sense.

40:03 Why aren't you going to the military?

40:06 I think this is a really good question and I because this these are

40:10 the bodies that we have built to solve these types of problems.

40:13 There's this thing also this mistake I think that was made very early

40:16 on where people phrase this problem of super intelligence like a new type

40:19 of problem like nothing in history is even close like this and you

40:23 know on some axis this is true but I think it's vastly overplayed.

40:27 This is a classic economics 101 public goods problem.

40:33 one of it's very clear that markets are great.

40:36 I like markets.

40:37 They provide my food.

40:38 You know, they're good for many things.

40:40 I I think you know, free market has done a lot of good in the world.

40:44 Um but it's one tool in our tool belt.

40:48 One of the big mistakes that a lot of people make,

40:49 especially America and especially the libertarian side of things,

40:53 is that they think of markets as a solution to everything.

40:55 But markets are one specific tool in our tool

40:57 belt to solve a certain class of problems.

40:59 And we know you know since like the you know 60s7s or even earlier

41:03 than that we know from standard economic

41:05 theory which types of problems markets cannot solve.

41:09 And the larger class of problems that markets

41:11 we know cannot solve is public goods problems.

41:14 Um the classic example is you have a river

41:16 you know it's unpolluted and you have a chemical

41:19 company and you could put all your toxic sludge

41:21 into the river and that would make your product cheaper.

41:24 Uh but it would poison the river.

41:26 Now maybe you're a nice guy and you don't put your sludge into the river.

41:30 But the guy next door, maybe he's less nice of a guy.

41:33 He puts the sludge into the river.

41:34 Now his product is cheaper.

41:36 He outperforms you.

41:37 You go out of business and now the river is polluted.

41:40 And now sometimes people get a bit confused here and they think like,

41:43 okay, but what if everyone is drinking from the water?

41:45 Well, then surely they wouldn't do that.

41:47 Okay, let's say the[ __] you know,

41:50 he finds out, oh, he's drinking from the water, too.

41:52 His kids are drinking from the water.

41:53 So he stops doing it.

41:55 But what happens?

41:55 Well, then some guy comes around who's actually crazy and thinks,

41:58 "Well, actually, drinking poison is good for you.

42:00 Actually, it's a liberal conspiracy that, you know,

42:02 drinking poison is bad for you.

42:04 Actually, he starts a chemical company and now the[ __] goes out

42:07 of business and now the crazy person is the one running the company." So,

42:11 the mech the problem here is the market.

42:13 It selects for price.

42:14 If you select for price,

42:16 you will select for someone crazy enough to pollute the river.

42:19 The same way is that the current market is selecting

42:21 for people crazy enough to think they can solve super intelligent alignment.

42:25 It's the same mechanism.

42:27 And so the sol and we know the solution to this problem.

42:31 The solution to this problem is regulation.

42:33 This is exactly what governments are for.

42:35 The only way to stop the chemical companies

42:37 to have from having a race to the bottom

42:39 is to have regulation that everyone cannot pollute

42:42 the river or we will throw them in jail.

42:44 This solves the problem.

42:46 Once everyone has to compete on a same level playing field, fine,

42:52 you know, have them compete on making the, you know, nicest plastic or whatever.

42:55 But we don't, we're not letting them put the sludge in the river, right?

42:58 That's fine.

42:59 And this is how we handle every other product,

43:04 every other form like there is no such

43:06 thing as an unregul unregulated market on anything.

43:09 And this is the reason this is a very very standard textbook,

43:12 you know, undergrad economics, very standard thing.

43:15 But for some reason early on the AI safety move,

43:18 these people got into their head that this is not

43:22 that this is like stupid or not needed or whatever.

43:24 You know, we could talk about why that is.

43:26 I think it shades again into psychoanalysis,

43:28 but fundamentally they were trying to solve

43:30 a a coordination problem with the wrong tool.

43:34 Private companies, charities and so on are not the structure that can

43:38 solve a problem like this, especially an international problem of this kind.

43:41 This is what we have governments, militaries, regulations, etc.

43:45 for.

43:46 I'm not saying these systems are perfect by any sense.

43:49 They're extremely flawed.

43:51 But they are the right tool for the job

43:53 unlike these other systems which a priority cannot work.

43:56 Like they're just the wrong tool.

43:59 So we mostly talk about and are concerned about what existential risks.

44:03 Should we spend some time on suffering risks?

44:06 Are they neglected?

44:07 Are they important to address?

44:08 Can they be used as a tool for personal

44:12 self-interest to motivate people not to build super intelligence?

44:16 I'm very I'm very worry about stuff like this.

44:20 So, a lot of again as I said earlier,

44:22 a lot of the things we're dealing with are very confusing.

44:24 Morality is very confusing.

44:25 Thinking about the far future is very confusing.

44:28 Thinking about utilitarianism, suffering, simulations,

44:31 consciousness, it's all very, very confusing.

44:34 And so there's the way one of the important things is when you're in a very

44:39 confusing environment is to often not think

44:43 about complex things because you will troll yourself.

44:48 You will if you're not grounded enough,

44:51 you will just kind of make up whatever and just kind of drift off.

44:55 So I think suffering risks are a very complex, very confusing thing.

45:02 they're very far off and they're basically universally post singularity.

45:06 They're post super intelligence in almost all cases.

45:10 Um, and so I generally don't think about them much

45:13 and not in the sense that they like are impossible.

45:16 Unfortunately, I do think they are possible.

45:18 I think they're unlikely, but like I do think they are possible,

45:21 but in the sense that they're not actionable.

45:24 They're not actionable.

45:25 They're very abstract.

45:26 They're very hypothetical.

45:27 They're very confusing to think.

45:28 I'm too confused to be able to solve this.

45:30 But the only thing again we can all agree upon is if

45:33 we don't build super intelligence we don't have to worry about it.

45:36 This is the one thing the very very simple thing

45:38 that we can all agree on and that's good enough.

45:41 So my my answer to that is just like we need

45:44 to not build super intelligence and if we build super intelligence

45:47 we're in such a different regime where we're just so screwed

45:51 that you know it's in it's in God's hand at that point.

45:55 But I guess there could be degrees of screwed.

45:58 Does it matter which company builds super

46:00 intelligence first or is it mutually assured

46:02 destruction regardless it doesn't matter at all

46:05 China US topic open AI same animal doesn't

46:09 yeah I don't think I don't think it matters mostly

46:11 because we have to think about this this is not

46:13 going to be like one AI that like one guy

46:15 built what's going to be is an ecosystem of billions trillions

46:19 of AIs you know copied mutated modified self-improving you know

46:25 all competing with each other like thinking super intelligence like

46:28 a monolithic entity is like you know we don't even

46:32 know how to do that like we don't even have designs

46:35 that could like do that if you have systems you

46:38 know grown as the current systems are they're not written

46:41 they're grown they're like are you know they're we don't

46:44 understand how they work internally and they have you know billions

46:46 of copies of them running and potentially you know self-improving

46:49 so writing their own code changing their own code this is

46:52 you know much more like runaway evolution than it is

46:55 like you know a specific being built by a specific person.

46:59 It's more like once this loop is truly

47:01 kicked off and humans are out of the loop,

47:04 who knows what comes out of that like it's just unknowable.

47:07 We we and like it won't be good.

47:09 That's the only thing we can know.

47:10 And you know whether the Chinese or Microsoft or OpenAI kick off

47:15 the loop currently there's no re they're all using the same techniques.

47:19 all their models have the same level of you

47:21 know stupidity basically or like insanity or evilness

47:25 like you know within error margins and so

47:29 I don't see why it would make a difference

47:31 Nick Bostonramm talked about singleton the first super intelligence

47:34 would prevent others from coming into existence that kind

47:37 of goes against your idea of billion of computing

47:41 competing super intelligences uh do you think he's wrong

47:45 I think the singleton will be made of super

47:47 intelligences but it will be much weirder than that.

47:49 I think it will in some sense, you know, likely converge to some kind of system

47:54 the same way that cells converge to a body.

47:56 Um, but first of all,

47:59 the transition there can be arbitrarily long and arbitrarily messy.

48:03 Um, like it's just impossible to predict.

48:07 Uh, we just don't have theory that can predict these things.

48:10 Like we just don't know, you know, like evolution still like even just like

48:14 normal biological evolution animals still constantly confuses us.

48:17 like we're still constantly surprised.

48:19 There's constantly new papers about like you

48:21 know our theory of biology was wrong and like it turns out evolution did

48:25 something crazy that we would have never predicted.

48:27 Um like we just don't actually have complete theories

48:30 or understanding here and even then I expect if

48:33 there was a singleton it wouldn't be like you

48:36 know a nice binary file on a single computer.

48:38 It would be a massive distributed network of like extremely

48:41 complex blobs of like you know unknowable things doing unknowable things.

48:47 Like even if we if we had an you know an ASI you turned off on a hard

48:53 drive and we could look at it we would have no idea what we're looking at.

48:59 So what is the main constraint right now on success?

49:01 Is it talent?

49:02 Is it funding?

49:03 If I gave you a trillion, not I,

49:05 but someone else gave you a trillion dollars uh towards your goals,

49:10 how would you spend it?

49:12 Uh funding is truly one of the biggest constraints right now.

49:15 Politics is expensive.

49:17 Um there's a lot of to give you a bit of a feeling.

49:20 Um a a medium to large scale political campaign in the United

49:25 States um can easily run you at least $500 million.

49:29 So, uh, for example, the pro Marana legalization campaign had a budget

49:34 of about $650 million and a typical,

49:38 uh, presidential campaign is about 1.1 to$1.2 billion.

49:43 So, this is what it would mean to have like one shot,

49:46 one serious iteration of a political campaign in the United States.

49:51 Uh, this is the average budget.

49:53 I think it can be done with less,

49:54 but like it's actually really uphill battle in terms of like,

49:59 you know, staff, advertising, you know,

50:02 there's just like a lot of things you need to do.

50:05 Um, there are, you know,

50:07 500 something uh members of Congress here in the United States.

50:12 Um, I've talked to maybe 10 of them personally and almost

50:16 every single one had not heard about this issue even once.

50:21 just, you know, maybe they've heard the word once somewhere,

50:24 but no one has sat them down and explained it to them for 30 minutes.

50:28 And often when I sit them down, like 80 plus% of the time,

50:31 I sit them down for 30 minutes and just explain the issue,

50:34 they get it immediately, they're just like, "Oh, wow.

50:36 Yeah, that's really bad.

50:37 What do we do?" Um, it's not that hard to explain.

50:41 It's not that hard to persuade people.

50:42 Even if you just tell people like, "Oh,

50:44 they're building things that are might be smarter than humans and they don't

50:46 know how to control them." And then how do you feel about that?

50:49 And the answer is universally bad.

50:51 People feel really bad about that.

50:53 That seems, you know, really dangerous and really bad.

50:56 And so, um, but if you want to brief all 500 of those members,

51:01 you know, say within a year or something, that's a lot of work.

51:05 You know, getting those things scheduled, getting facetime with them,

51:08 and having people to deliver the meetings,

51:10 that alone is like takes a lot of people, you know, it takes a lot of effort.

51:14 It's not glamorous in the advertising industry.

51:18 Um, generally the way one of the rules of thumb is if

51:21 you want someone to internalize a message or to remember a message,

51:25 they have to hear the message seven to 10

51:27 times over the course of like 6 to 12 months.

51:30 I think this is a pretty good goal.

51:32 How expensive or how hard would it be to get

51:34 say every citizen of the United States or, you know,

51:37 top 50% or top 10% whatever, right,

51:40 to hear this message seven to 10 times over the next 12 months?

51:45 And the answer is, you know, we've done we've, you know,

51:48 done some of the math and it's like, yeah,

51:50 probably around 500 million is probably what this costs.

51:53 Um um and we have some change to spare to do it like

51:58 a a smaller version probably LG20 countries um with that amount of money.

52:03 I think currently um you know, we wrote a post recently.

52:07 I think if we had $50 million,

52:09 that's the smallest possible amount of money where I think we have

52:11 a chance where I think we could run a large enough media campaign,

52:14 we could, you know, get enough stuff done that we

52:17 have maybe a 10% chance of getting this work to work.

52:21 I think if we have 500 million, we'd have a 40% chance of pulling it off.

52:26 Um, both of these are, you know, obviously just my feelings.

52:30 You know, there's not like you can objectively measure anything like this.

52:32 This is just you know our back of the envelope

52:35 calculations of like you know how many times could you

52:37 get various people to see things for a given amount

52:40 of ad budget and a given amount of you know management.

52:42 How many people would you need to perform various tasks like these?

52:46 Um yeah we at control AI have spent years now building a playbook that scales.

52:52 This is something that the community and like

52:54 the general project have been lacking for a long

52:56 time is that a lot of the things

52:58 that have been done historically are very bespoke.

53:00 You know, like Ellie Ezer wrote a book and it was a great book, right?

53:02 And I'm, you know, glad he did it,

53:04 but you know, only Elezer could have done that.

53:07 And we can't give him 10 times more money to write 10 times more books.

53:10 That's not kind of how these things work.

53:11 It's not scalable.

53:13 Um, but what we have done is we've built something that's very scalable.

53:16 You can give us 10 times the amount of money

53:17 and we will do 10 times as many things.

53:19 It's very easy for us to teach new employees.

53:21 It's very easy to find new employees that are good enough to do this.

53:24 We don't need super geniuses, you know, we just need like good,

53:26 you know, conscientious people who want to do good and, you know,

53:29 h have the courage to actually, you know,

53:32 do the playbook and then I think we can get there.

53:36 Um, it's very hard and a lot of it depends on times time frames.

53:40 If we had like 10 years, I'm really quite confident we can do this.

53:46 Like I'm really quite confident.

53:48 If we have two years, yeah,

53:51 it's going to be real tricky and we're gonna need the 500 million for that.

53:54 How much do you have right now who's funding Control AI?

53:57 Uh we're funded by, you know, Phil uh philanthropic donors,

54:00 you know, Yan Talin and, you know,

54:02 that type of person and, you know, we don't disclose the exact numbers,

54:05 but it's like a lot less than 50 million.

54:08 Have you considered running for office instead of persuading others?

54:12 Um I have definitely briefly thought about it and I think

54:15 it's not very uh feasible in the sense that it's very slow, it's very expensive.

54:20 Um it requires a bunch of the similar work we

54:22 do now and it's like lower leverage in many ways.

54:25 Um I think if I had 10 or 20 years maybe um but there's a big premium

54:30 on speed right now and so talking directly

54:33 to people at scale and helping politicians that currently exist.

54:36 The other thing politicians love if is if you solve a problem

54:40 and they can take the credit and we are happy to do that.

54:43 We're very very happy for us to solve all

54:46 the problems and they can take all the credit for it.

54:48 That's no problem for us whatsoever.

54:51 So there's a lot of leverage in this uh where we don't have to polarize a thing.

54:54 We can just like we can provide solutions bipartisan

54:57 Democrat Republican you know we can solve the problem.

55:00 We can tell you exactly what to do.

55:01 We can help you do it.

55:02 We can run polling.

55:03 can do.

55:04 We can do all these kinds of stuff and we're

55:05 happy if the politicians take the the credit for it.

55:08 That's not a problem for us at all.

55:11 On the kind of opposition side,

55:12 acceler accelerationist side, who is intellectually worth debating?

55:17 Who is making actually good arguments, not just propaganda, anyone?

55:25 Uh, who is the best pro- tobacco lobbyist?

55:29 Um, I was actually told by someone who that is

55:32 and they hired them to do promotions for AI safety.

55:35 So, there isn't to that question.

55:36 I don't know what the name is.

55:38 Well, there you go.

55:39 Maybe you should go talk to him because I

55:40 guess that's as good as it's going to get.

55:42 I mean, fundamentally, like it is just a fight.

55:45 like it is just actually politics at this point where like um

55:49 you know I'm sure there's some academic philosophers that are saying some[ __]

55:53 about like well actually it's morally required to for us to do

55:59 it because it's okay to take a 90% risk of everyone dying because

56:02 there's a 10% chance of simulated souls or some[ __] like I've

56:06 heard this stuff from you know people before especially like the you know

56:09 extended bowrm cluster but like to me this is not a it's

56:12 like a Mickey Mouse argument you know it's like a Marvel comics argument,

56:15 something that Thanos would say, you know, some kind of like,

56:17 you know, evil alien would say in a sci-fi movie,

56:19 like this is not a real argument, um,

56:22 that like normal humans like would take seriously.

56:25 So, I expect if you want like, you know,

56:28 like most of this is just being pushed cynically,

56:30 like we just got to be real here.

56:31 Like there's just a thing where just like

56:34 there are just cynical forces in the world,

56:37 you know, whether they're evil or not, we can argue, but they are cynical.

56:40 Like fundamentally, they just want power.

56:42 or they're just, you know,

56:43 they're the guy with the chemical factory and they want to pollute

56:45 their waters and they'll say whatever to get you to keep doing that.

56:52 I think it's um I think it's much more valuable would be to have

56:55 debates and conversations with people outside of tech

56:58 like people who are not financially incentivized.

57:01 I think you know academics from other fields

57:04 um politicians you know union leaders civil rights

57:07 groups stuff like this I think and that are

57:09 like potentially skeptical about AI for various reasons.

57:12 I think this is much more productive.

57:14 I think there's a lot of very productive conversations

57:16 and dialogue to be had there in groups outside of this.

57:18 But like I think just like the core tech cluster, the core AI cluster is just so

57:23 extremely captured by financial interests that it's

57:26 like very very hard to have any

57:27 kind of like serious intellectual discussion here.

57:31 Um, you know, I'm sure there are some nice people in this field, right?

57:35 I'm sure they exist and that are more unbiased.

57:38 I don't know them, but like I'm sure they exist.

57:41 Um I don't really know them but I do know that outside of a core

57:46 unreason you know um accelerationist camp outside of that camp there are many

57:50 very reasonable nice people who may have

57:53 some skepticism or uncertainty around you know say expos and so on with whom

57:58 you can definitely have very productive conversations.

58:01 So I want to end in a more lighter set of questions.

58:05 You brought up Nick Boston's kind of intellectual cluster.

58:08 Do you have any feelings about simulation hypothesis?

58:12 It's unfalsifiable.

58:13 It's not really hypothesis.

58:14 Um like it's the kind of thing where like if

58:17 you're in a perfect simulation by definition you couldn't know.

58:20 Therefore it's equivalent.

58:21 Like the two universes are indistinguishable.

58:24 So it's kind of like unscientific to distinguish them.

58:26 Is it 50/50 you in a simulation then?

58:29 It's undefined.

58:30 Like there there are three truth values for every statement.

58:34 True, false or undefined.

58:36 And this is undefined.

58:39 Fair enough.

58:40 Last time we met at MIT, I think uh you had long handsome hair.

58:46 I suggested that maybe you should cut them

58:48 off to be taken more seriously by politicians.

58:51 Is this is uh what I'm looking at?

58:53 Is this what's happening?

58:54 Was that right?

58:56 It's definitely a part of it.

58:57 I But honestly, those hair that hair had to go.

59:00 It was so such a pain in the ass.

59:02 It's a long overdue.

59:04 You know, it was cool.

59:05 You know, when you're younger and you're like a weird tech guy,

59:07 but like oh man, it was such a pain.

59:08 It took two hours for it to dry every day.

59:12 Just like such a No, no, no.

59:13 I'm so glad that that's gone.

59:15 Um, so yeah, that's, you know,

59:16 partly that, but partly it's also like it's not worth the effort.

59:23 I hear you.

59:24 I got a little bit going there and it's uh definitely overhead.

59:27 Final question.

59:29 You have to come up with a clickbait title for this episode.

59:34 H what would be a what would be a good one?

59:40 This is the only way to stop AI.

59:44 I love that because Google algorithm wants it to be about three

59:47 words long to fit it into the tiny preview thumbnail.

59:52 John Connor speaks.

59:53 So what do we have there?

59:57 Um h three words.

59:58 Let me think.

59:59 Can I do better on three words?

1:00:02 Um, I I don't want to say anything too negative,

1:00:09 but so maybe it's something like it's not over.

1:00:13 I love it.

1:00:14 Thank you so much, Connor.

1:00:16 I wish you tremendous luck.

1:00:17 I hope to visit you in DC and yeah,

1:00:20 see all the accomplishments, hang out with all the senators.

1:00:23 Uh, yeah, sounds great.

1:00:26 Thank you so much for having me on.

1:00:27 Thank you for all the work you've done over the years here as well.

1:00:31 And yeah, good luck to all of us.