The Pentagon’s AI war machine

0:00 He's never suggested he wanted to kill more people.

0:04 He doesn't use rhetoric like that.

0:06 Other people do and other people on the team said to me anonymously,

0:11 "Great, let's get AI.

0:12 Now we can kill people all the time." Uh So,

0:15 there are there are definitely people who considered that AI would help

0:20 speed up and scale killing of the people perceived as America's enemies.

0:44 Katrina Manson, welcome to the show.

0:46 Thanks.

0:47 Thanks for having me.

0:48 Let's just assume that everyone listening

0:51 and watching has never heard of Project Maven, knows nothing about it.

0:56 What is it?

0:57 It was an effort that began in 2017 for the Pentagon to develop an AI tool.

1:03 Uh develop a way of using AI on the battlefield so that AI could

1:09 AI computer vision a specific kind of AI could look at drone feed video footage.

1:16 Um this was the the the feed

1:18 that the US was taking in uh the counter-terrorism wars,

1:22 the GWOT, global war on terror, apply AI to look at it better.

1:27 That was ostensibly what it was,

1:29 but it was also part of a much bigger effort uh to really bring AI

1:34 to the battlefield and speed up the way

1:37 the Pentagon was thinking about the future of war.

1:40 Uh the US had been in uh was still in Afghanistan,

1:45 um Iraq, and uh Somalia, Yemen.

1:50 There was a lot still going on and people

1:53 were beginning to think, "What about China?

1:56 What if the US has to up to China

1:58 and there was a group of people under the first

2:00 Trump administration who said the US was behind despite

2:03 having the biggest defense budget in the world by none.

2:08 Um the US started to feel that it was using too unsophisticated

2:15 tools and needed to catch up with what the commercial sector was doing,

2:19 bringing AI, trying out driverless cars.

2:23 Could the US move to automated warfare in such a way that folded in AI.

2:31 So, Project Maven was was birthed by the Deputy

2:35 Defense Secretary at the time, Bob Work.

2:36 So, it had serious buy-in from the the top leadership.

2:42 But what the people who really tried to forward this always

2:46 felt was that they were going to be up against it.

2:50 Um they felt that they risked putting the intelligence side of the shop

2:56 out of out of a job if they could bring AI into operations.

3:01 Yeah.

3:02 Um that would essentially be cutting out or going around or somehow undermining

3:05 the intelligence folks or they thought they

3:07 would the intelligence folks would feel that way.

3:10 And um the colonel who who was that he wasn't the director,

3:13 the director was a two-star Air Force general,

3:16 three-star Air Force general named Jack Shanahan.

3:18 Um but the colonel who led it as chief, so the day-to-day operator,

3:23 the doer, who also had a lot of the vision was called um Drew Cukor.

3:28 He was a Marine colonel.

3:29 W- What's his deal?

3:31 What's his role in this history?

3:34 He was the chief of Project Maven and really

3:37 got it going for the first five, six years.

3:40 He was also one of the visionaries behind even pursuing it.

3:44 And he worked tirelessly uh to try and bring AI warfare to life.

3:53 He wanted to get AI out into the battlefield in a safer way as possible,

4:00 but to test it in as real-life like scenarios as possible.

4:06 And he came from this background of being

4:10 a marine for years before he started doing Project Maven.

4:13 He'd been sent into Afghanistan in October 2001.

4:19 And had lived through seeing marines not

4:23 have sufficient information to keep them safe.

4:27 And he told me that he had carte blanche to fire anyone who got in his way,

4:33 which immediately got me thinking, why would anyone be in your way?

4:38 Surely you want to do the same thing.

4:39 But the answer was was no, they didn't.

4:42 And he was very clear from the beginning that AI could put people out of work.

4:47 It would test people's resolve, their metal,

4:51 their their way of thinking about how war was done.

4:54 And he came at that from this position of I think it's better

4:58 to say always believing that the the operator

5:00 had been unfairly insufficiently supported.

5:03 So the people who were on the front lines needed to get more information.

5:06 And for him, AI was a way of getting information to those on the front lines.

5:13 In the early deployments of Maven,

5:15 um if you were an analyst or drone pilot or a targeter sitting

5:20 at a screen in the Pentagon or Nevada or some forward operating base,

5:24 how did Maven change your job?

5:28 In the really early days, so if we say before 2020,

5:30 the first two or three years, it was actually just a mess.

5:34 It was such a mess that the people

5:37 involved in Maven would say these algorithms don't work.

5:41 So the first users they worked with were in Somalia

5:45 in 20 at the very end of 2017 into 2018,

5:50 they used it the system was so annoying they stopped using it.

5:53 They then sent someone to try and encourage them

5:55 so so much of what this project Maven team,

5:58 which was quite a small knit team of mostly Marines but not not just Marines,

6:03 was trying to do was just to get someone to even try it.

6:05 So, it wasn't in all those places that you just listed.

6:08 They did really well with special operations command,

6:11 the more forward leaning tech part of the military.

6:14 You can have all personal relationships

6:16 with with the commanders and the operators.

6:19 Some of them already had relationships with them in in former deployments,

6:23 so they drew on those.

6:25 And it was simply to identify something.

6:29 If you think about drone video footage,

6:31 there are multiple frames in a second of footage

6:34 and the algorithm went to work on each frame.

6:37 So, if the algorithm failed to identify

6:39 the same object frame to frame, it would flash.

6:42 So, the operators were having real difficulty even looking at it.

6:45 And if the sensitivity was set very low, everything was being identified.

6:51 So, in some of those early experiments, there could be dozens,

6:54 hundreds of boxes all flashing at the same

6:56 time and people would just turn it off.

6:58 So, they started to improve

6:59 on that with the with the very negative feedback that they got.

7:04 One of the breakthroughs that I was

7:06 told about is probably the 2018-2019 time frame.

7:10 And there were some um Marines had been

7:15 booby-trapped in a raid against a compound in Afghanistan.

7:20 And so, they were getting fired

7:23 at from multiple places and the wall had exploded.

7:27 And the AI helped perceive through the smoke I

7:31 mean the the drone feed was picking it up

7:33 but the AI spotted through the smoke the individual Marines

7:37 much more quickly than a human eye would have done.

7:40 And of course, one of the problems the US has faced is is I mean,

7:44 it's called friendly fire, but uh not identifying their own people.

7:50 And so, being able to count out the Marines with AI very quickly in a specific

7:54 kind of scenario like that made people start to believe in the potential of AI.

8:00 But, certainly at the time there was still clearly a human in the loop, right?

8:05 These the humans were were making all the judgments.

8:07 These were not autonomously directed, right?

8:09 They weren't AI These systems weren't launching

8:12 drone strikes of their own volition, right?

8:15 There were humans at various checkpoints involved.

8:18 Yes, and that language of human in the loop

8:20 is really interesting because you have a lot

8:22 of military commanders then and since saying we

8:24 will always have a human in the loop.

8:26 Um it's not actually technically the policy of the of the Defense

8:30 Department or Department of War as they call themselves now.

8:34 Uh the first directive on autonomy came in, I think,

8:37 in 2012 and then it was updated in 23.

8:40 And the update that was given in 23 says uh appropriate levels

8:46 of he appropriate levels of human judgment over the use of force.

8:50 So, it implies something a little

8:52 closer to supervision rather than each decision what's appropriate?

8:57 Such a good question.

8:58 And some of the reporting that's come out on um the big fight between

9:02 Anthropic and Pen- the Pentagon has focused in on the word appropriate and said,

9:07 "In the end, we're just quibbling over

9:08 the word appropriate." But, from the Anthropic perspective,

9:13 what is appropriate could determine whether a human is involved or not.

9:17 At the time in 2018, 2019, 2020,

9:20 there's no sense at all that an algorithm is making these decisions.

9:23 But, of course, people are beginning to wonder,

9:25 "If I start relying on algorithms,

9:27 if I start trusting the outputs without being able to check myself,

9:32 at what point am I no longer asking

9:35 the analytical questions that are required for target engagement.

9:40 And again, they weren't at that stage at that point, but there was certainly,

9:43 I think, always concern,

9:46 discomfort from some of the people being asked to use it.

9:49 And so, it needed repetition, practice, workflows, all that kind of thing.

9:54 Initially, as you said,

9:54 this was about they were using the software to sort through drone footage.

9:59 But, was the idea, the plan always to develop tech,

10:03 scale it up, and deploy it across the entire military and defense department?

10:09 Was that a clear vision from the jump?

10:11 It really depends who you ask.

10:13 Um the memo itself that started Project Maven just talked about um uh drone

10:19 footage uh in the fight against ISIS

10:22 and potentially extending to other defense intelligence purposes.

10:27 When I did the research,

10:28 a really important question for me was to establish two things.

10:32 One was tell me how you thought about targeting

10:36 from the get-go uh in terms of this project,

10:38 because when we haven't got there yet,

10:40 but eventually Google protesters uh become very concerned about what

10:44 they're not discovering that their company was working on this.

10:47 Um so, that's partly an issue of transparency.

10:49 But, they were concerned that they could be involved in the business of war.

10:53 And Google said at the time, "It's only for non-offensive purposes." So,

10:57 I really wanted to check, was that true?

11:01 Was it always intended for non-offensive purposes?

11:04 Was Google correctly describing the project?

11:08 And Drew Kuko told me he always had targeting in mind from the outset.

11:16 That language is not in the memo.

11:18 Um others told me Drew Kuko would wince

11:21 if you said this was a a targeting project.

11:24 But, when I actually managed to speak to the to the man himself,

11:28 and when I went back and read his papers um his thesis,

11:30 he he believed in this idea of white

11:33 dots that you could look at a map essentially.

11:36 He He wrote this before we even had Google Maps,

11:39 but imagine just looking at Google Maps, clicking with your cursor,

11:42 and being able to pick up the the the precise

11:44 coordinate from your cursor and send a weapon to it.

11:48 So, he has his own He was a marine intelligence officer.

11:52 His own papers describing this very idea that then,

11:55 when he also suggested the idea for Project Maven and leads it,

11:59 he was very clear with me that he always had targeting in mind.

12:03 And he knew that he would be going up against um eons

12:08 of intelligence practice where there are specific

12:12 programs used to take um a coordinate.

12:15 There are specific ways of checking elevation.

12:17 There are specific things to do for georectification.

12:19 It's obviously a very very complex system.

12:21 Plus, then there are processes of you know, strike list,

12:24 but there that system was the system that he wanted to not blow apart,

12:32 but he knew he was going to be bulldozing through a part of it.

12:37 He uh Colonel Kuklinski,

12:38 he's a very interesting character um in the book and in this history, really.

12:44 Um And you know, you you quote him in the book

12:48 saying that the the problem with war is the humans.

12:51 They're materially corrupt, inefficient, and they get tired.

12:56 You know, I I'm familiar with this type of military officer.

13:00 You know, very often when they um like rail

13:04 against um the bureaucracy and that sort of thing,

13:08 they're they're really protesting uh all those pesky rules

13:11 of engagement that make it harder to kill people.

13:15 And you know, I served with people like this, and I'm

13:18 not saying they're villains or bad people at all.

13:21 Um I just think sometimes well-intentioned people inside the war

13:26 machine have a very hard time appreciating the importance of guardrails.

13:32 And in their defense, it's not their job to do that, right?

13:35 Their job is to prosecute wars.

13:38 Um but in to me that quote suggest

13:41 that what they're really looking for is easier ways.

13:43 Obviously, they want to save lives, right?

13:45 The particularly their their own troops,

13:48 but they're looking for ways to make it easier and more efficient to kill.

13:52 And that's a very dangerous game.

13:55 I think he's I mean, he is a very interesting person.

13:57 I think he's very aware of that um read.

14:03 He comes the way he presents himself certainly is that he

14:07 comes from a very moral place about the consideration of war.

14:11 So, yes, he does say those things that humans are the problem

14:15 with war and he can sound um cold in that sense,

14:20 but he never suggested the rules of engagement um should be diminished.

14:26 And the main reason he puts forward

14:28 in his conversations to commercial entities for why they

14:31 should come on board with the Pentagon was

14:34 always um we could save civilians this way.

14:37 We could make sure we don't hurt our own.

14:40 So, I think for him he had been sent to Afghanistan in 2001

14:49 uh the month after 9/11 and was one of those first targeting officers,

14:55 intelligence officers, who was having to suggest targets.

14:59 And of course, very quickly US military

15:01 personnel were being hit by improvised explosive devices.

15:07 Yep.

15:08 And he was having to put together the the packages,

15:11 where should we go, who should be hit, who was the enemy.

15:15 And was frustrated that there was so little information to protect US personnel.

15:25 And has also talked about other moments where he was frustrated the US

15:30 couldn't intervene in support of civilians

15:33 because they didn't have the information.

15:35 So the way he's always framed it, he's done the first part of what you said.

15:38 He's he's is almost brutal or brusque certainly.

15:42 Um, but he's never suggested he wanted to kill more people.

15:47 He doesn't use rhetoric like that.

15:49 Other people will do and other people on the team said to me anonymously,

15:54 "Great, let's get AI.

15:55 Now we can kill people all the time." Uh,

15:59 so there are there are definitely people who considered that AI would

16:02 help speed up and scale killing of the people perceived as America's enemies.

16:09 He himself has a slightly different filter on it.

16:14 Support for the show comes from Bombas.

16:16 If your sock drawer could use a little love, you can upgrade to Bombas.

16:20 They design their socks with a keen eye for detail,

16:22 offering everything from dress socks to sport socks.

16:25 The latter is made with a cushioned,

16:27 sweat-wicking design that stops them from sliding down your foot,

16:30 which I hate, while you stay active this spring.

16:34 I wear my Bombas socks pretty much every

16:36 day and I especially wear them when I'm running, which I do all the time,

16:40 and we're now entering the summer down here on the Gulf Coast,

16:44 which is essentially a giant sunny sauna.

16:48 And the only socks I wear at this point are my Bombas socks.

16:52 Everything else gets too hot or too gross, but when I take off my shoes,

16:55 when I get back from a long run,

16:57 my Bombas socks are still dry, still cool, as are my feet.

17:01 I cannot recommend them enough.

17:02 They really are great.

17:04 They have more socks, too, with breathable,

17:06 soft, high-quality basics, including underwear and t-shirts.

17:10 You can go to bombas.com/grayarea and use

17:13 code grayarea for 20% off your first purchase.

17:17 That's bombass.com/grayarea code gray area at checkout.

17:26 What does the chain of decision-making look like?

17:30 How much do we really know about that?

17:33 So CENTCOM has told me that they're using a variety of AI tools.

17:38 Um I've separately reported that that includes Maven Smart System,

17:42 which is the system that Palantir helped

17:44 develop for the algorithms to feed into.

17:47 So it's almost like the digital display

17:50 that you'd have in a headquarters or maybe

17:52 on a a handheld device so that you can look at the battlefield digitally.

17:57 And then uh more than 150 different data feeds feed into it.

18:03 And you can crunch through that using AI.

18:06 Um so they've got the computer vision,

18:08 but they've also now got large language models, specifically Claude,

18:12 um which is the Anthropic model that is cleared to work on classified cloud.

18:17 And the US fights its war wars on classified networks.

18:20 So I was told in 20 last year in summer, I went to visit NGA,

18:28 which is the combat support agency that supports um the Defense Department,

18:32 but is also member of the intelligence

18:33 community National Geospatial Intelligence Agency.

18:36 So last summer they told me that with the help of AI,

18:40 Maven Smart System can now get through a thousand targets a day.

18:44 A thousand?

18:45 Um in the first 24 hours of the US operations in Iran,

18:50 they went through a thousand targets.

18:53 And with the help of LLMs, uh really using that to speed up the processes,

18:59 the kind of admin processes involved in building a targeting package,

19:02 getting permission for it, still from a human,

19:04 still from a commander, still with legal review, but sped up.

19:08 Um they told me, one person uh official told me they could

19:13 now get to 5,000 targets in a day if if they wanted to.

19:18 That's a lot.

19:21 Yeah.

19:21 So, take something like drones.

19:23 Obviously, drones are such a big part of modern warfare.

19:27 Um we're using them.

19:28 Everybody seems to be using them.

19:30 Um are humans still piloting our drones

19:33 or are these mostly autonomously controlled now,

19:37 even if there still is somewhere on the back

19:39 end a human in the loop green lighting strikes?

19:43 Ukraine has a lot of drones and Russia has a lot of drones,

19:45 but the US is not producing that many.

19:47 Uh the US is desperately trying to now take those lessons

19:51 on board and and produce um and compete them against each other.

19:56 They are almost entirely not autonomous.

20:00 So, autonomy is the hope.

20:02 And under the Biden administration um the hope,

20:05 especially for something like Hellscape,

20:06 which is the Indo-Pacific Command's idea of how they could defend

20:10 Taiwan from an invasion by China if China decided to do that.

20:15 And the admiral there, Admiral Paparo,

20:18 talks about um using autonomous uh weapons to buy him a month.

20:23 So, just make it impossible for China to take

20:25 Taiwan and then send in the larger US platforms.

20:31 So, under the Biden administration,

20:33 I think in '22 or '23, they launched something called Replicator,

20:37 which is to bring in cheap they would have to chase basically,

20:40 you don't need to use it again.

20:42 Um Yeah.

20:44 drones.

20:44 And those are meant to be autonomous.

20:45 So, so they've been trying to develop the software.

20:47 They've been competing with different companies to do that.

20:50 And through the course of my reporting,

20:53 I discovered that the idea was to take um

20:55 some of that those algorithms that Maven had produced,

20:59 um train them on data from the Indo-Pacific, uh really at that boat level.

21:04 So, uh boat drone cameras, aerial drone cameras, infrared,

21:11 anything that might be looking at a Chinese vessel, capture those pictures,

21:17 train the algorithms, sit them on the drone now instead of having it

21:21 on a digital uh at a digital platform at headquarters level,

21:25 and have that AI on the drone automatically detect the target,

21:30 and then be able to have the drone go and take the target out.

21:35 It was very tough going those experiments,

21:37 even before the Trump administration got in.

21:40 They were making progress.

21:41 They also wanted to do something very ambitious,

21:43 which was to link up drones in the sky, drones on the water,

21:48 and drones under the water into one big autonomous swarming mesh.

21:52 It sort of boggles boggles the mind.

21:54 Um Then they even part of it wasn't working.

21:58 So, they had the best data stores, I'm told.

22:01 So, the algorithms were potentially the best,

22:05 but they couldn't integrate the algorithms onto the platform.

22:07 And so much of AI isn't the specific piece of tech itself,

22:10 it's can you make all these platforms talk to each other?

22:14 Can you make an operator believe in this platform?

22:16 Can you workflow it?

22:17 Can you start operating as one continuous ecosystem?

22:20 And the answer is not without a huge

22:22 amount of prac- practice and trial and error.

22:24 And and maybe just no.

22:26 But it's just a matter of time, right?

22:28 Autonomy might be a hope, but it's also inevitable, right?

22:30 It's just a question of the tech getting there.

22:32 And it's moving in one direction, right?

22:34 Like this That's where this is going.

22:36 Maybe not tomorrow, maybe not next week, but I mean,

22:39 look at the progress in the last 12-18 months alone, right?

22:42 I mean, that's where this is going.

22:44 I'm always wary of the word inevitable.

22:45 But as a as a as a history student, I was taught nothing is ever inevitable.

22:50 and I am not a reporter in any sense of the the term, so fair enough.

22:54 Okay.

22:54 My opinion.

22:55 in in support of your point, almost in support of your point,

22:57 um the Trump administration came in, tore up Replicator a little bit,

23:03 um, changed the name, just a repackaging of the name DOG,

23:07 uh, the Defense Autonomous Warfare Group.

23:09 So, autonomy's in the name and is warfare.

23:11 So, all of those concerns where the Pentagon was too nervous to say we want

23:15 to put AI autonomy and death together because

23:17 everyone was outraged about it back in 2018,

23:21 the language now is so much more permissible.

23:23 The Pentagon is simply saying it.

23:25 The fight now with Anthropic is over, not just autonomy,

23:29 but fully autonomous weapon systems, and they're trying again.

23:33 So, they have this new project that I've reported recently on.

23:37 It was launched in January.

23:38 Uh, it's a it's a $100 million prize challenge.

23:42 Same ideas of Maven to a certain

23:44 extent of competing the companies against each other.

23:47 And SpaceX and xAI are one of the contenders,

23:50 um, Palantir I reported as a contender,

23:54 OpenAI is named as the second on two other contenders,

23:58 um, Cap Gemini was I think I reported.

24:01 Uh, and they're all trying

24:02 to make voice-controlled autonomous drone swarming tech.

24:07 So, you could have an operator on a beach,

24:09 let's say, saying, "Move left." And the drones would move left.

24:13 And you have to hope they could identify the target.

24:15 That's wild.

24:17 So, there is a quote in the book uh,

24:19 that I really wanted to mention, and it's from Jane Pennerlis.

24:21 I hope I'm saying her name correctly.

24:23 She was in charge of testing Maven in those early days, and she said,

24:28 and now I'm quoting, "If the US military wanted to use AI-enabled systems,

24:32 it had to become more accepting of risk."

24:36 Based on the people you talked to, like,

24:38 what is the level of acceptable risk she has in mind there?

24:44 I don't have a number for it, um, but I think it's about this.

24:48 I think it means they know that AI is a black box technology that can go wrong,

24:57 and that it needs vetting.

24:59 But at a certain extent,

25:01 if you are going to put it in a system where you can't see behind the hood,

25:06 you're going to be relying on something that has inbuilt risk.

25:10 We know about hallucinations, bias.

25:12 Um she spoke extensively about algorithmic drift,

25:15 this tendency for an algorithm to get worse over time.

25:18 So, she wanted of course to hold standards high,

25:22 but she wanted to understand how AI will fail.

25:25 The risk element is often put to me this way.

25:28 If you use AI in an urban environment,

25:30 there's a huge chance that you could be getting um civilians.

25:36 If you're using AI in a war at sea in a China scenario, that box of operations,

25:44 you're going to have already cleared There won't

25:45 be civilians walking around cuz it's a sea.

25:48 The commercial boats will no longer go have thought,

25:51 "I'm not going to go in that area or it's banned." And so,

25:54 all you really have are targets at sea.

25:59 And the risk then for the US becomes are

26:02 they going to shoot their own targets by accident?

26:04 And are they definitely shooting at uh Chinese military

26:07 vessels who are legal targets under the law of war?

26:10 But it's the idea that if you go wrong with AI at sea,

26:12 you're just getting water.

26:14 And so, they might not be as accurate,

26:17 even though the claim for AI is often accuracy,

26:19 uh but if it does go wrong, the risk of harming civilians is much lower.

26:24 Are Are we watching that in real time?

26:25 So, on the first day of this conflict in Iran,

26:30 American weapons bombed a school in Iran

26:36 that killed lots of people, lots of children.

26:40 And that was on the first day of the conflict.

26:42 And based on the reporting of the Washington Post,

26:45 at least, and maybe others by now,

26:47 these AI systems may have been powered by Claude

26:50 was involved in identifying hundreds of targets before

26:54 that conflict started and presumably many of those targets

26:59 were the ones that we hit on that first day.

27:02 Do you know anything about that?

27:03 Do we Do we Do we know if that was in fact an AI identified target that a human

27:10 in the loop failed to to realize that it

27:12 was based on I believe like decade-old intel?

27:16 Bunch of caveats first, which is that the US says it's investigating

27:18 and they haven't said they they did it themselves.

27:22 Um the reports uh that are out in Alice not not my own.

27:27 Um have suggested the US did it.

27:32 This is uh there's no suggestion yet that AI is involved,

27:38 no confirmed suggestion.

27:40 This is what I would say.

27:41 The the US builds its targeting lists based on a stored data.

27:52 If something is a valid target or not, it's kept in a list.

27:57 What the AI can do is identify a specific object,

28:03 a specific threat, or something moving often.

28:07 If AI is pulling on an existing targeting list,

28:13 if that school turns out to be on a military intelligence database when it

28:21 should have been on the the restricted target list, no AI can fix that.

28:28 So, a key question is was that school on a targeting list by mistake?

28:33 Was that target list kept updated when

28:35 the school peeled away from being you know,

28:38 was an IRGC facility and then suddenly it had a bunch of kids in it.

28:42 Did they update the targets?

28:45 Could they have been using AI to check against open

28:49 source information if the school was listed on Google Maps?

28:54 What on earth is the point of AI if you're not checking that?

28:57 As the US military becomes better at checking open source information,

29:01 and that lesson was really learned in the US support to Ukraine,

29:05 they were drawing on social media feeds.

29:07 They were pulling Twitter posts, um so that Maven smart system could analyze

29:13 it for is there a yellow flag tied bench?

29:16 Does that mean this town supports Ukraine?

29:18 Does this mean this town actually supports, you know, or has a Russian presence?

29:22 Uh has something just exploded over there?

29:25 If you can pull from social media and use

29:28 that to inform your understanding of the battlefield,

29:32 can you pull from Google Maps?

29:34 Now, my understanding is any system, even if it's open source,

29:37 needs to be an authorized system on US kind of networks.

29:41 So, where is the gap, if there is one,

29:43 on being able to pull open source information and cross-check?

29:47 It should be extremely easy for AI to cross-check if there's a girl's school.

29:52 Um it should happen before uh there's a blink.

29:57 But, the question is I I we just don't know yet,

30:01 and they may choose to put out a public report.

30:05 Journalists may have to sue for it.

30:06 You know, that information will come out,

30:08 but we know from previous errors that um the Beijing uh embassy uh attack.

30:18 There's a really good one from 1999, I think.

30:22 Um the US hit um the the Chinese embassy in Europe,

30:33 and it was two or 300 yd off from the target they were meant to hit.

30:40 And they didn't have it labeled right.

30:42 Now, with AI, all of that should be much easier.

30:45 There is an argument to be made that sort your systems out,

30:48 but if someone doesn't care sufficiently about protecting civilian lives,

30:52 or someone isn't uh forcing AI into the bits of the system that will

30:57 protect people as opposed to speed up the death cycle or the kill chain,

31:02 all of that becomes a really big problem.

31:04 And if AI has been involved in any way in this hit,

31:06 of course it's a Well, any which way,

31:09 it's a it's a it's it's not just a tragedy,

31:11 it's a it's a very powerful mistake that um To go back to Anthropic,

31:16 but what do you make of the the very

31:18 public fight between Anthropic and the government, right?

31:21 I saw, you know, Anthropic, from what I understand, set a couple of red lines.

31:27 Um no mass domestic surveillance and no

31:29 fully autonomous weapons without human oversight.

31:33 Those were their red lines,

31:34 and apparently they could not come to an agreement with the Defense Department.

31:39 Just what do you make of that and the consequences?

31:41 I think by the time you have a frontier AI company

31:49 that is the first to put its model on classified cloud,

31:58 you have a company that's leaning in in a way

31:59 that is is not reminiscent of of Google back in 2017, 2018.

32:05 Anthropic was on the very systems where there

32:09 are lethal operations and clearly comfortable with that.

32:14 Um if you read Dario's two big essays, he has Dario is the the CEO of Anthropic.

32:23 Yeah.

32:23 He He has these two big essays that he wrote,

32:25 um making his case for why his company should be involved in national security.

32:33 You know, grappling with that thing

32:34 that everyone in AI is worried about existentialism,

32:37 will do what what it what whatever it is about um

32:41 whether AI comes to kill us all or not takes over.

32:44 He's grappled with that, too,

32:45 and he has found peace with the idea that you can do

32:48 national security work and still be {quote} "the good guy." The good guy.

32:52 Um he talks about a real fear of um robot swarms.

33:02 And his position which I think emerged in greater clarity only he part

33:08 way through this fight is not even

33:11 that he's against fully autonomous weapon systems,

33:13 is that he's against fully autonomous weapon systems now.

33:18 And it raises questions about what was actually under discussion.

33:23 Was there a system that he was being asked to put AI onto that he didn't

33:28 want to, or was he just worried

33:29 that they weren't doing the testing and evaluation right?

33:32 Because Anthropic did submit I reported to this $100

33:36 million prize challenge to create voice control drone swarming tech.

33:41 Um so that's leaning really far forward

33:45 uh for a company that's concerned about autonomy.

33:47 They were prepared to take part in the creation

33:51 of lethal autonomy uh or parts of it.

33:55 The there's clearly a political dimension because

33:57 the president himself called um them left-wing not jobs.

34:02 Uh there there is clearly I think you have to take the Pentagon at its word.

34:06 They're genuinely worried that a company could dictate policy to them.

34:09 Or they're certainly genuinely um annoyed at that prospect.

34:16 And the the castigation of the company as a supply chain risk,

34:21 then taking it to court, then having Microsoft file the you know

34:24 the the the the amicus brief shows that once again

34:30 the ability for the Pentagon to get to the tools

34:34 that it thinks it needs is somehow at risk.

34:38 Even when it has a company that was leaning really far into it.

34:42 And whether they can get up um xAI and OpenAI onto classified

34:46 cloud and into Maven smart system in a way that works as well

34:50 as Claude in those 6 months of transition time while the US

34:54 is using Claude in Iran is a really unexpected turn of events,

34:58 I think, to have it uh collapse

35:02 so spectacularly that relationship just at the point

35:05 that the US decides to test it the most it's ever been tested.

35:10 Uh it it it it really is genuinely surprising.

35:14 It's not my original analogy,

35:15 but I I did hear someone say um that allowing a handful

35:21 of private companies to control AI is kind of like leaving Amazon,

35:26 Google, OpenAI, whoever in charge of the Manhattan Project.

35:30 And then also allowing them to control and profit from the bombs.

35:35 Of course, like in order for that to land,

35:37 you have to accept the premise that AI is

35:39 as revolutionary and transformative attack and as powerful as nuclear weapons.

35:46 But if you do accept that premise, and I'm certainly open to it it is startling.

35:53 So, what is What is your moral position

35:54 on it given your kind of own national security background?

35:58 On Well, I I served in the military.

35:59 I wouldn't say I have a national

36:01 background um beyond just having been one soldier.

36:05 But what is my position on what exactly?

36:08 On the morality of AI in these national security uses.

36:11 When you When you When you were

36:12 talking about the moral position of the companies.

36:15 I'm extremely uncomfortable with it.

36:17 Extremely uncomfortable with it.

36:19 I understand the utility.

36:20 I understand all the potential applications,

36:22 and I can see the case for all the lives

36:24 it might save and all the good it might do.

36:27 Um But the tail risk really alarm me.

36:33 And my personal view is that war and killing

36:39 people should be very hard and very costly.

36:42 And anything that makes it easier and faster

36:47 and cheaper and less expensive in terms of human life

36:50 when you can just pull pull a lever or push

36:52 a button thing that just makes killing people easier.

36:57 And I just it makes me very uncomfortable.

36:59 And then that's just even beyond all that I'm not entirely sure

37:04 that AI is a technology that we are going to be able to control.

37:08 A lot of these conversations presume that we'll be

37:11 able to control what these systems do and don't do.

37:15 And I'm not sure that's the case.

37:17 Which scares me even more.

37:19 So, I I I know I I don't know how clear a position that is,

37:22 but I I'm just that's really all I can

37:24 say right now that I'm very uncomfortable with it.

37:27 Very uncomfortable with it.

37:28 And I don't I don't really trust anybody to make

37:33 the with that much power to make those sorts of decisions.

37:37 Um I don't know.

37:40 What do you think about that?

37:42 I think um you know,

37:46 the director of Project Maven who who is no longer uh Jack Shanahan has spoken

37:53 up during this crisis of the Anthropic Pentagon fault line not in favor of AI.

38:01 Even though he was the director of Project Maven to say

38:04 no LLM should be anywhere near an autonomous weapon at this stage.

38:08 And you know, the other red line

38:10 that Anthropic raised was uh mass domestic surveillance.

38:15 Now, I don't quite know what they think the Pentagon

38:20 had in mind because the Pentagon's position is you know,

38:24 we have rules around that and we follow them.

38:27 Um but the volume of data points available

38:32 on any given individual not with traditional intelligence,

38:37 just with commercially available information from your phone or your route

38:43 that your vehicle takes um or your shopping habits or you know,

38:46 all of these things and and that discussion where you had people like Elon

38:52 Musk signing on to letter saying we

38:55 shouldn't have technologists involved in AI because

39:00 of the risk of we shouldn't be creating new weapons of war and it's

39:05 his company that signed on to make

39:09 these voice-controlled uh drone autonomous drone swarming tech.

39:14 The changing comfort levels about what technologists are prepared to accept is

39:21 the sort of massive tribute to the Pentagon's ability to change people's minds.

39:27 Support for the show comes from DeleteMe.

39:30 DeleteMe makes it easy, quick,

39:31 and safe to remove your personal data online at a time

39:35 when surveillance and data breaches are

39:36 common enough to make everyone vulnerable.

39:39 The reality is that we're all

39:41 susceptible to having our private information stolen,

39:44 public figures and private citizens alike.

39:46 DeleteMe can help protect you and your family's personal privacy or privacy

39:50 of your business from doxing attacks before sensitive info can be exploited.

39:56 Our colleague Claire White recently tried DeleteMe.

39:59 DeleteMe is a tool that anyone online should have in their back pocket.

40:02 DeleteMe has saved me not only hours of removing my data from online,

40:07 but has saved me hours of worrying about who has

40:11 their hands on my data and what they're doing with it.

40:14 Take control of your data and keep your private life private

40:17 by signing up for DeleteMe now at a special discount for our listeners.

40:22 You get 20% off your DeleteMe plan when you

40:25 go to joindeleteme.com/vox and use promo code vox at checkout.

40:30 The only way to get 20% off is

40:32 to go to joindeleteme.com/vox and enter code vox at checkout.

40:37 That's joindeleteme.com/vox code vox.

40:47 Support for the show comes from Shopify.

40:49 Every thriving business starts with a series of what if questions.

40:53 There's what if nobody likes me, but also what if they really really do?

40:57 There's only one way to find out and you

40:59 can make it happen with help from Shopify.

41:02 They say millions of businesses around the world rely on Shopify

41:05 for e-commerce from businesses just getting started to household name brands.

41:09 It can help you with everything

41:10 from payment processing to analytics to website design.

41:14 You can choose from hundreds of templates to create a great looking website.

41:18 Their email and marketing tools can help you get

41:20 your name out there and stay connected with customers.

41:23 And if you ever need help,

41:24 Shopify's 24/7 award-winning customer support has your back.

41:29 You can turn those what ifs into a thriving business with Shopify today.

41:33 You can sign up for your $1 per month trial period today at shopify.com/vox.

41:38 You can go to shopify.com/vox.

41:41 That's shopify.com/vox.

41:48 I just happen to think that we are not

41:49 even close to really stepping back and and wrestling

41:52 with how profoundly all of this tech is

41:53 going to change our society and our institutions.

41:57 Did you get the sense that there were like serious discussions going

42:00 on about about how these tools might

42:02 migrate from the battlefield to American cities?

42:04 How they might end up in the hands of police departments across

42:08 the country using it for for surveillance and and and God knows what else.

42:13 is that Yeah, I know you to that idea is exactly what has animated so many

42:17 of the protesters of Project Maven and campaigners

42:21 against the development of these AI tools.

42:23 And I do think it's interesting that the twin things that Anthropic

42:27 has raised is not just uh fully autonomous weapons against presumably an enemy,

42:33 but also domestic surveillance.

42:35 Um and it is because the overlay of data

42:39 and knitting up of systems presents such a potentially powerful tool.

42:45 And that will come down to policy choice

42:50 and law because the technology is now possible.

42:55 Uh still hard.

42:57 Uh but the data points are out there.

42:59 You just need to suck them up.

43:02 Do you think policy makers in DC are are taking this seriously?

43:07 Are are they paying enough attention to the care?

43:10 I was put to me the I mean Congress has spent a long time looking at AI,

43:14 but there's there's no regulation.

43:16 And one of the things that the Trump administration

43:18 has has really focused on is setting AI free.

43:23 So, the way the Europeans are regulating

43:26 on on data specifically, never mind also AI,

43:29 but on data um the US is taking a different approach and and that's

43:34 to do with uh the champions would say uh the innovative US spirit,

43:41 that entrepreneurial ability to go fast and um

43:44 and make things and maybe break things as well.

43:47 Uh so, I think in this debate between Anthropic and or over the fault

43:53 line between Anthropic and the Pentagon uh

43:55 several of the expert voices have said,

43:57 "Where's Where's Congress in this?" Um and it

44:03 it stops short of regulation at the moment.

44:05 I don't know, Katrina.

44:06 I mean, I I think at some point war ceases to be a human activity.

44:13 You know, it will still impose enormous human cost,

44:15 but the actual war fighting um will just

44:18 be a technological affair for the most part.

44:22 And that's a very different world, you know,

44:23 and you can only change the character of war

44:25 so much before you change the nature of it entirely.

44:28 And I think that's where we where we are.

44:30 I appreciate you writing this.

44:31 It's so important.

44:33 It's so important.

44:34 Um and it's so well reported.

44:36 Uh I feel like I understand the world,

44:39 this world, better than I did before I opened it.

44:42 So, um thank you for writing it and thank you for for coming on the show.

44:47 Thank you.

44:48 Thanks for the discussion.

44:50 Once again, the the book is called Project Maven.

44:54 If people want to check out your your other your reporting

44:56 for Bloomberg or or any of your other work, where can they go?

45:00 Bloomberg, yeah.

45:01 Just my name on Bloomberg will do it, but Bloomberg's plenty.

45:04 Thanks for watching.

45:05 Every week, we bring honest and nuanced

45:07 conversations about what's happening in culture,

45:09 tech, and the world of ideas to your video and audio feeds.

45:13 Episodes of The Gray Area drop every Monday on YouTube,

45:16 Apple Podcasts, Spotify, or your favorite listening app.

45:21 Comment below and let me know what you thought of this conversation.

45:24 I promise I won't be offended.

45:25 You can also send us an email

45:27 at the grayarea@vox.com or leave us a voicemail at 1-800-214-574-9.

45:34 And if you enjoy what we do,

45:36 please help support Vox by joining our community on Patreon at patreon.com/vox.

45:42 Thanks again.