The Internet Was Weeks Away From Disaster And No One Knew

0:00 (suspenseful music)- [Derek] In 2021, a hacker uncovered a fatal weakness

0:04 in the world's most important operating system.

0:07 What would you do with a key that gets you into any server on the internet?

0:12 Is this live to the public right now?

0:14 Yeah, it's live on the server.

0:15 Look, I'm not pleased.

0:16 I would like you to change it back.

0:19 [Narrator] At the time,

0:20 just about everyone believed that hacking this system was impossible,

0:23 but they were wrong.

0:24 Well, I can tell you how many systems would have been compromised,

0:27 which would have been millions.

0:28 Actually, I'm still surprised the mainstream news

0:31 outlets haven't really covered this very much.

0:33 How close did we come?

0:35 We were weeks away from millions of internet

0:38 servers being accessible to whoever crafted the backdoor.

0:41 Anything from spying, to ransom, to taking down entire countries,

0:47 you could have done it with this backdoor.

0:50 This hacker had realized the entire operating system rested on a single part,

0:54 maintained by a single person, and that by compromising that one part,

0:58 they could infect almost any server on the internet.

1:02 So, how could we ever let ourselves get this vulnerable?

1:07 Well, the story begins with a jammed printer.

1:11 (suspenseful music) (upbeat music)- [Narrator] The AI lab was buzzing.

1:21 They had just installed the Xerox 9700.

1:24 It was one of the first ever commercial laser printers.

1:27 It was a pretty big deal.

1:29 The only problem was it kept jamming.

1:34 [Stallman] You'd wait an hour figuring, I know it's gonna be jammed,

1:36 I'll wait an hour and go collect my printout,

1:39 and then you'd see that it'd been jammed the whole time.

1:42 Frustration up the wazoo.

1:46 Richard Stallman, a researcher at the lab, thought that he had a solution.

1:50 Years earlier, he had solved a similar problem by coding

1:52 a simple program that sent an alert whenever there was a jam.

1:55 Now, it didn't fix the problem mechanically,

1:57 but it did make sure that a jam wouldn't go unnoticed.

2:00 He thought he could do a similar thing now.

2:02 The only problem was that Xerox hadn't

2:04 provided them the source code for the printer,

2:06 and without it, Stallman couldn't write his code.

2:08 So he tracked down the original developer.

2:11 [Stallman] And I said, "Hi, I'm from MIT.

2:12 Could I have a copy of the printer source code?" And he said,

2:17 "No, I promised not to give you a copy." I was stunned.

2:21 I was angry.

2:23 All I could think of was to turn around on my heel and walk out of his room.

2:30 Maybe I slammed the door.

2:33 And I thought about it later on because I realized that I was seeing not just

2:38 an isolated jerk but a social phenomenon

2:41 that was important and affected a lot of people.

2:46 [Henry] This social phenomenon had slowly

2:48 invaded the world of computer research.

2:50 In the late 60s, engineers at AT&T's

2:52 Bell Labs invented an operating system called Unix,

2:56 which they shared widely across universities and research labs.

2:59 This was a time of freedom.

3:01 But by the 80s, AT&T started going

3:04 after Unix clone developers for copyright infringement.

3:07 Later, they even sued the University of California at Berkeley.

3:10 The tech landscape had shifted.

3:12 They wanted to close off software development.

3:15 Companies were now making their employees sign non-disclosure agreements,

3:18 prohibiting them from ever sharing their code with other programmers.

3:23 [Stallman] See, this was my first encounter with a non-disclosure agreement,

3:26 and I was the victim.

3:28 And the lesson it taught me was that non-disclosure agreements have victims.

3:32 They're not innocent, they're not harmless.

3:36 [Henry] Stallman wondered, maybe he could adapt to this new world.

3:40 [Stallman] But I realized that that way I

3:41 could have fun coding and I could make money.

3:44 But at the end, I'd have to look back at my career and say,

3:48 "I have spent my life building walls to divide

3:51 people." and I would've been ashamed of my life.

3:56 So Stallman chose a different path.

3:58 He quit his job at MIT and in 1985 established the Free Software Foundation,

4:03 and it worked to promote four basic freedoms.

4:06 You should be free to run software for any purpose,

4:08 free to study it, free to change it, and free to share it.

4:12 Now, to ensure those freedoms, he created a legal license that developers could

4:15 attach to their code called the General Public License.

4:19 And to stick it to AT&T,

4:20 he started to work on a project based on Unix but built from the ground up,

4:24 so AT&T couldn't sue.

4:27 He called the project GNU, a recursive acronym for GNU is Not Unix.

4:33 Now, to replicate a Unix system,

4:34 the GNU Project had to recreate three layers of functionality.

4:38 They needed the utilities,

4:39 which were the everyday tools and commands, the shell,

4:42 which is the terminal that people use to interact with the machine,

4:45 and finally, the kernel,

4:46 which is the core that talks to the hardware and manages memory.

4:50 Now, over the next seven years, the GNU Project made much of that from scratch.

4:54 They created the GCC code compiler, the Bash shell,

4:57 and a host of other core utilities.

4:59 But they were always missing one key component.

5:02 The kernel.

5:05 That changed in the fall of 1991 when Stallman visited

5:08 the University of Helsinki to give a talk promoting the project.

5:12 In the audience was a young computer science student who

5:15 just happened to be building his own kernel from scratch.

5:19 His version wasn't free, but after hearing Stallman speak,

5:23 the student changed his mind and adopted the General Public License.

5:27 At first, he wanted to call it Free Unix, or Freax,

5:31 but his friend thought that sounded terrible,

5:34 so he renamed it after the student himself, Linus Torvalds.

5:37 Linus, Unix.

5:39 Well, that's how he got Linux.

5:43 That kernel, combined with the other components from the GNU Project,

5:46 became a full operating system.

5:48 Now, technically, Linux only refers to that kernel,

5:51 but a lot of people use it to refer to the whole operating system,

5:54 so GNU and Linux and whatever else.

5:57 Because the code was open and free and the projects built on it were too,

6:01 a new model of software development took hold.

6:03 Anyone could inspect the code, improve it, fix flaws,

6:07 and generally just push development forward for everyone.

6:09 So, software split into two competing ideologies.

6:13 Proprietary closed source systems controlled by companies,

6:16 and open source projects where the code was free.

6:19 It's free in two ways.

6:21 It's free as in you don't have to pay for it,

6:23 but it's all free to change it in any way you want,

6:26 and that seems to be the much more important aspect.

6:29 People are happy to pay for technology,

6:31 but so often do they run into some roadblocks where

6:34 you have to file a support ticket with some large company,

6:38 they may or may not get the help they need,

6:40 and engineers are just itching to just fix it themselves.

6:45 Developers could take that basic code which was freely available

6:48 and then add on their own features relevant to their specific device.

6:52 They didn't have to reinvent the wheel every time.

6:55 So that's why Linux spread into all sorts of different applications.

6:59 Hello, I'm a Mac.

7:00 And I'm a PC.

7:01 No one else.

7:02 No one.

7:02 (woman clears throat)- Hi, I'm Linux.

7:05 There are an estimated 30 million Linux users out there.

7:09 How long you been standing there?

7:10 A long time.

7:12 And it's not even just limited to computers.

7:15 Your electronic vacuum is definitely Linux.

7:18 Your camera is definitely Linux.

7:20 Most TVs, most electronics are Linux.

7:24 Linux even runs some of the most sensitive machines on the planet.

7:28 You can assume that Linux is pretty much used in anything of high-security need,

7:32 not necessarily because Microsoft, for instance,

7:36 couldn't build something equally secure,

7:38 but because usually there's secrecy involved in building,

7:42 let's say, a new weapon system,

7:44 and you don't necessarily want to have to work with some tech company.

7:47 You don't want to involve more people than absolutely necessary.

7:52 [Henry] Of the top 500 supercomputers in the world, every single one runs Linux.

7:56 It's used in the Pentagon and on US nuclear submarines.

8:01 Every bank you can think of really, manufacturers, hospitals, governments,

8:07 defense organizations and things like that, they're all running Linux servers.

8:11 Today, Linux is everywhere, and most people are familiar with Windows and macOS,

8:15 but they are not the most popular operating systems in the world.

8:18 No, they are dwarfed by systems running a Linux kernel.

8:22 Android, with over 3 billion devices, is built on Linux.

8:26 And it also powers the majority of internet servers in the world.

8:30 There is no one company that could have imagined all

8:32 the different cases where computers are used these days, and Linux,

8:37 thanks to its adaptability where everyone can just tweak it

8:40 in little ways to make it fit their use case, now covers all the use cases.

8:47 But all of this, it all relies on one key assumption.

8:52 That the code is secure.

8:53 Now, there's a good reason to feel this way.

8:56 Because there are so many people looking at the code,

8:58 there's this idea that bugs, either intentional or unintentional,

9:02 won't be too deep to catch.

9:04 It's known simply as Linus's Law.

9:06 That with enough eyeballs, all bugs are shallow.

9:09 But there's a big problem with this assumption.

9:12 The open source movement isn't one big project.

9:15 It's an ecosystem.

9:16 You need thousands of small tools and libraries each doing a different job,

9:20 like networking, security, or compression.

9:23 Now, a lot of these projects start because

9:25 one person wants to fix a specific problem, so they build it themselves.

9:29 They're often unpaid, coding on nights and weekends just to make the tool work.

9:33 If it's useful, one open source project adopts it, then another,

9:37 and suddenly you have millions of machines

9:39 all relying on one person's passion project.

9:42 That's how the entire ecosystem can end up quietly

9:45 resting on a project maintained by a single volunteer.

9:48 There's a famous XKCD comic that captures this idea perfectly.

9:53 But what happens when that block is compromised?

9:58 In our story, our person isn't from Nebraska.

10:01 No, Lasse Collin is from Finland,

10:03 and he's been working on a small data compression tool called XZ since 2005.

10:08 XZ is so good at compression that it's

10:11 now used in almost every major Linux distribution.

10:14 For the past 20 years, almost all of the work of keeping

10:17 the tool compatible with ever-evolving hardware, it's all fallen on Lasse.

10:22 He's never been paid for it, but up till now, he's been okay with that.

10:26 Recently, though, he's been under more and more pressure.

10:30 "Over one month and no closer to being merged.

10:33 Not a surprise." "Progress will not happen until there is a new maintainer.

10:37 Submitting patches here has no purpose these days.

10:40 The current maintainer lost interest or doesn't

10:43 care to maintain anymore." Lasse responds, "I haven't lost interest,

10:47 but my ability to care has been fairly limited,

10:50 mostly due to long-term mental health issues, but also due to some other things.

10:54 It's also good to keep in mind that this is

10:57 an unpaid hobby project." But it's not enough.

11:00 "I'm sorry about your mental health issues,

11:02 but it's important to be aware of your own limits.

11:05 The community desires more.

11:07 You ignore the many patches bit rotting away on this mailing list.

11:10 Right now, you choke your repo." Lasse is burning out.

11:15 But just when he thinks he can't handle it anymore...

11:19 "Nice job to both of you for getting this feature as far as it is already.

11:22 Just trying to do my part as a helper elf." Signed, Jia Tan.

11:27 For months, Jia has been taking some of the load off Lasse.

11:30 He's been incredibly helpful.

11:32 Now he offers to step up and take over as maintainer of the project.

11:36 To Lasse, it sounds almost too good to be true.

11:39 "As I've hinted in earlier emails,

11:41 Jia Tan may have a bigger role in the project in the future." Finally,

11:45 Lasse can step back and breathe after 20 years of hard work.

11:49 But Jia is not who he appears to be.

11:53 And he's identified Lasse Collin's XZ project

11:56 as a weak link in the Linux ecosystem,

11:58 one that could give him access to almost every computer on the internet.

12:02 (suspenseful music) Today we take secure remote logins for granted.

12:11 I mean, they've worked reliably for over 30 years.

12:14 But it all started in 1995

12:16 at the Helsinki University of Technology when a hacker

12:19 captured thousands of usernames and passwords sent

12:22 over the campus network in a sniffing attack.

12:25 In hindsight, the problem's obvious.

12:27 These login requests were being sent totally in plain text,

12:30 so anyone who intercepted the data could just read it.

12:34 (suspenseful music) When Tatu Ylonen,

12:37 a computer researcher at the university, learned of the attack,

12:40 he made it his mission to ensure that it would never happen again.

12:44 [Tatu] Password sniffing was perhaps the most

12:47 serious security issue on the internet back then.

12:52 To do this, his solution needed to ensure two things.

12:55 First, machines had to establish a secure connection.

12:58 If both computers could agree on a shared secret

13:00 code that they would use to scramble their data,

13:02 then even if they were overheard,

13:04 anyone without that secret code would just get gibberish.

13:07 Now, you could agree on that shared secret ahead of time in person.

13:11 Password.

13:12 But on the internet, that's rarely practical.

13:14 No, you have to agree on that shared secret ahead of time

13:17 without ever having met and also with someone listening in the entire time.

13:21 It sounds really tricky, but there is a way to do it,

13:24 and I can show you how using this jar of paint.

13:27 Say I'm trying to send a message to Gregor over there.

13:30 First step is we agree on a shared public color.

13:33 Let's pick this red.

13:35 This is no secret, anyone can see this.

13:38 Now we each pick our own private color.

13:40 I'm gonna pick yellow, and he can pick whatever he wants.

13:44 So we take our private color, and then I'm gonna mix that with the public color.

13:51 It's worth saying now that these mixtures are assumed to be impossible to unmix,

13:54 so even if you know this orange and you know this red,

13:57 you can't exactly deduce the exact shade of yellow we used to create it,

14:00 and this is important for the actual computer example later.

14:03 Okay, so I'm gonna send this over to Gregor.

14:06 So, I mixed in my secret color with the public,

14:07 and I'm gonna pass this to Henry.

14:10 So, Gregor sent me this, which looks like a sort of dark green sort of color.

14:15 And what we're gonna do now is we're

14:17 gonna mix it with my original private color.

14:21 Okay, now that I have Henry's secret color mixed in with the public,

14:24 I'm gonna add some of my own.

14:28 So we end up with this sort of distinct olive color.

14:33 There's my yellow in there, I can see, and whatever Gregor had in his side.

14:37 And the thing is because each set of paints went through the same process,

14:40 they both end up with this same olive green,

14:43 even though we never shared our secret colors.

14:46 So we end up with this shared secret color at the end that no one else can get,

14:50 and that means that we can use it as our secret code when sending information.

14:54 Now, in the real exchange, we use big public numbers instead of colors,

14:57 but the idea is the exact same.

14:59 Each side mixes in their own private number using

15:02 some math that, when you try to reverse it, leads to a discreet log problem,

15:06 which makes it practically impossible to unmix them.

15:08 That way, we solve the first problem.

15:11 But there is another threat that's unaccounted for.

15:14 Say a hacker, like Casper here, tries to sit in between us.

15:18 Now we can create a legitimate connection,

15:21 so we end up with a shared secret code,

15:23 and Casper could do the exact same thing with Gregor.

15:27 Now, whenever I send a message, he can relay that to Gregor,

15:30 he can change and modify it and send his response back.

15:33 And to each of us, the connection looks legitimate,

15:35 but Casper's sitting between us the whole time.

15:38 He's a man in the middle.

15:40 So, I need a way of authenticating that Gregor is really who he says he is.

15:44 Now, we could do this again by agreeing on a password ahead of time in person,

15:48 but we need a practical way to do it over the internet.

15:50 This was the second problem that Tatu had to solve.

15:53 To make that happen,

15:54 Gregor can take two really big prime numbers, which he keeps secret.

15:58 He then multiplies them together to get an even bigger number,

16:01 which he then makes public.

16:03 Now, when I want to send Gregor a message,

16:05 I just take that big public number and I scramble it in a way that only Gregor,

16:09 who knows the two prime factors that make up that big public number,

16:12 can successfully unscramble.

16:14 For anyone else, getting those two prime factors is practically impossible.

16:17 So, as long as I know that that big public number actually belongs to Gregor,

16:21 I know that anything encrypted to that key can only be read by him.

16:24 This is called RSA encryption,

16:26 and it means that if I know the certificate is valid,

16:29 then I accept the connection.

16:30 And by authenticating Gregor, it foils our man in the middle, Casper Devious.

16:36 All right.

16:38 Tatu Ylonen combined these two steps,

16:40 securing the channel and authenticating the user,

16:42 into a protocol for remote logins between machines.

16:46 It gave you the same simple text shell people

16:48 were used to, a plain terminal where you type commands,

16:51 but now the connection was encrypted.

16:53 He called it Secure Shell, or SSH.

16:57 And it was immediately useful.

16:59 Many Linux machines don't even have keyboards or monitors,

17:02 especially not servers,

17:03 so you wanna be able to log in and control them remotely.

17:06 So SSH was soon adopted on almost every machine that ran Linux.

17:11 And as Linux spread, so too did SSH.

17:14 Today, when you control a machine remotely,

17:16 there's a good chance you're using SSH.

17:18 SSH is literally the maintenance backbone of the entire internet.

17:23 And the most widely used open source SSH implementation is called OpenSSH.

17:29 And because it's so popular, it's heavily protected.

17:33 I mean, OpenSSH is probably one of the most closely examined projects out

17:39 there because it's just so vitally

17:42 important to the security of servers everywhere.

17:44 Having a way to bypass the authentication in secure

17:48 shell is like having the master key to the hotel.

17:51 It lets you into every room.

17:54 (suspenseful music)- [Henry] This is why Jia Tan wants a way into OpenSSH,

17:59 but trying to hack it directly would be almost impossible.

18:03 Lucky for Jia, the open source model doesn't just

18:05 mean that operating systems are stitched together from many programs,

18:09 but that each of those programs is itself stitched together from other programs.

18:14 Those are called dependencies.

18:16 OpenSSH is one of the most scrutinized software packages,

18:19 but that doesn't extend to all of its dependencies.

18:24 Jia believes that if he can compromise a dependency of OpenSSH,

18:27 he can sneak an exploit into the main project.

18:30 And it just so happens that Lasse Collin's compression

18:33 tool XZ is linked through a chain of these dependencies.

18:38 (suspenseful music) Now, Lasse's original goal with XZ was to find

18:46 a better way to compress data on Linux.

18:48 That data could be anything.

18:49 Code, an image, text.

18:51 But what was important to Lasse was

18:53 that once you compressed and decompressed it,

18:56 it had to come back exactly the same.

18:58 The method had to be lossless.

19:00 Let me give you an example.

19:01 We're gonna take the lyrics to Rick Astley's hit "Never

19:04 Gonna Give You Up" and we're gonna try to compress it.

19:07 Now, say we take this and we represent it as a stream of characters,

19:11 and each one gets a fixed-width 8-bit code.

19:14 Now, that works, but it's inefficient.

19:17 If we go through this stream and just count up how often each symbol appears,

19:21 you'll notice there's a pattern.

19:22 Some appear more frequently, like N with 430 uses,

19:27 and some, barely at all, like J with one use.

19:30 To save space, why don't we give

19:32 the ones that appear more frequently shorter codes,

19:34 and the rarer ones, well, they can afford to be long.

19:37 But how do we do that?

19:38 So, let's start by counting up how often each symbol

19:41 appears and sorting that from most frequent to least frequent.

19:45 We take the two least frequent symbols and join them together into a pair.

19:48 We then treat that pair as a new combined symbol

19:51 whose frequency is the sum of the two it represents.

19:54 We can then reinsert that back into the list.

19:57 Then we do it again.

19:58 We take the two least frequent items, combine them,

20:01 and then reinsert them back into the list.

20:03 And we do that over and over again until

20:05 we get this massive structure called a Huffman tree.

20:09 Now, to get our codes, we just walk the tree.

20:12 A step right is a 1, a step left is a 0.

20:15 So, for example, to get R, we just go right,

20:18 left, left, right, so the code is 1001.

20:21 So what you'll notice is the more commonly occurring

20:24 symbols naturally appear at the top of the tree, so they get shorter codes,

20:27 while the ones that appear less frequently are at the bottom of the tree.

20:30 The system works well, but it also has a weakness.

20:33 In our "Never Gonna Give You Up" example, it always encodes N-E-V-E-R space.

20:39 It doesn't realize that this whole chunk repeats.

20:42 So, what if instead of looking at symbols, we looked at those chunks?

20:46 Now, they don't have to be words, they can be parts of words or even longer.

20:49 They just have to be patterns that repeat.

20:51 So let's scan through the text but keep

20:53 a rolling dictionary of what we've just seen.

20:55 Then, as we move forward,

20:56 we can check whether the next chunk has already appeared.

20:59 And if it has, we don't need to write that chunk again.

21:02 We just write a code with two numbers,

21:03 how far back to look, and how many characters to copy.

21:07 Now, when we decompress,

21:08 we can just read along and whenever we hit one of these codes,

21:11 we jump back, copy the matching chunk, and paste it into place.

21:15 Two scientists, Lempel and Ziv, published this algorithm in 1977,

21:20 so it became known as LZ77.

21:24 But some of these symbols and pointers show up more often than others.

21:28 They actually have their own frequencies.

21:30 So we can feed that whole stream into another

21:32 Huffman tree to get a second layer of compression.

21:35 And in our demo, it actually gets the file down 85% smaller than the original.

21:40 This might look new, but you've almost certainly used it yourself.

21:43 It's called deflate, but it's better known for the files it creates, .zip.

21:49 If you ever clicked Close on this before, you've definitely used it.

21:52 But Huffman only uses the overall frequency of a chunk repeating.

21:58 Real data isn't just random chunks.

22:01 In our example, after "Never gonna", you might get "give you up",

22:05 "let you down", or "run around and desert you".

22:08 You might get "make you cry",

22:10 you might get "say goodbye" or "tell a lie and hurt you".

22:13 Each one has its own probability, and you can represent these probabilities

22:17 with a mathematical tool called a Markov chain.

22:20 The algorithm can then encode the stream of data so that the more

22:25 probable next chunks cost few bits and the less probable ones cost more.

22:30 If you combine that with a much bigger search

22:32 window so it can point much further back in memory,

22:34 then you get the Lempel Ziv Markov chain algorithm, or LZMA.

22:39 LZMA was developed by Igor Pavlov around 1998,

22:42 and it often beats much more familiar methods.

22:45 In many cases, it can shrink files to about 70% of the size of a typical .zip.

22:50 Lasse took this elegant compression algorithm and made it work on Linux,

22:54 and he called it XZ not because it stood for anything,

22:57 but just because it sounded cool.

23:00 I'm using XZ quite a lot.

23:01 I think XZ is a wonderful project.

23:03 There are lots of different ways of compressing data.

23:06 Some of them are fast but they don't compress very well,

23:09 and some of them are slow but they get extremely good compression.

23:15 But across Linux, projects are constantly shipping the same files

23:18 and updates to millions of machines, so XZ is perfect.

23:22 You compress something once, then you get a smaller file to download forever.

23:26 Lasse released XZ in 2009, and over the next decade and a half,

23:30 it went from a niche tool to the common

23:32 choice whenever a project needed effective lossless compression.

23:35 So, XZ quietly spread everywhere, eventually becoming a dependency of OpenSSH.

23:41 (suspenseful music)- So,

23:47 it was at some point in about February 2024 and Jia Tan, he emails me.

23:53 He's got all these new features in the new version of XZ.

23:57 [Henry] He wins Rich over almost immediately.

24:01 So I get to talk to hundreds of contributors all the time,

24:03 and I do get a feel for them.

24:06 I feel, you know, are they good coders, which is what I really care about.

24:11 Are they conscientious people, are they helpful?

24:14 Do they respond to bug reports quickly?

24:18 And in all of the dimensions,

24:20 Jia Tan would be a very good contributor because he's obviously a good coder.

24:26 He's very responsive, he's very keen, and I love all that.

24:30 All indications are that Jia is a great contributor,

24:33 and this puts Rich at ease, so he lets his guard down.

24:37 And that's often where the problems start on the internet.

24:39 You can't keep your guard up forever.

24:41 But lucky for us, with today's sponsor, NordVPN, you don't have to.

24:45 NordVPN's Threat Protection Pro blocks dangerous websites before they load.

24:49 It stops malicious downloads and it

24:52 strips out trackers and intrusive ads automatically.

24:54 And it works even when you're not connected to the VPN,

24:57 so a lot of these attacks never get the chance to start in the first place.

25:00 I use NordVPN whenever I'm traveling or working on public wifi because

25:04 it means that I don't have to think about who's running the network.

25:07 It's just one click and it's so fast that I often forget that it's on.

25:10 Not just that, if there's a show that's no longer

25:13 available in my region or a sports team that's blacked out,

25:16 like I'm often watching international football and they

25:19 don't quite have it where I'm going, well,

25:20 in that case, I can just switch my server

25:22 location with one click to unlock the content.

25:25 Apparently you can even use it to find better deals

25:27 on plane tickets by changing your IP address to another country.

25:31 I haven't tried it yet, but that sounds fascinating.

25:33 So, if you wanna try it,

25:35 you can get the best deal by going to nordvpn.com/veritasium.

25:39 When you use that link or this QR code, you'll get a huge discount.

25:43 Also, you get a 30-day money back guarantee through Nord.

25:47 It's a no brainer.

25:47 So again, that's nordvpn.com/veritasium or you can

25:52 click the link in the description below.

25:54 Thanks so much to Nord,

25:55 and let's get back to Jia and the prize he's got his eyes on.

26:00 At this point, we were preparing RHEL 10.

26:04 [Henry] See, Red Hat ships two major flavors of Linux.

26:07 Fedora, which is free and publicly available, and Red Hat Enterprise Linux,

26:12 or RHEL, which is available through a paid subscription.

26:15 This one has to be stable and secure

26:17 because it's widely used on the most important machines,

26:20 like in governments and hospitals.

26:22 Jia wants his code in RHEL,

26:24 but RHEL only has a new major release about once every three years.

26:28 So, there's definitely a deadline,

26:30 and that deadline was around sort of March, April in 2024.

26:35 Jia has to act fast.

26:36 He wants complete control of any compromised machine.

26:39 And to pull it off, he has three steps in his plan.

26:42 Step one, the Trojan horse.

26:47 The code for XZ lives on a website called GitHub,

26:50 which tracks all edits to XZ's code using a tool called Git,

26:53 which was also developed by Linus Torvalds.

26:56 So, Jia starts by making small changes.

26:58 He changes the primary contact for bug reports to his own email.

27:02 He tweaks small tools that will help him later.

27:04 But he can't sneak in the payload this way.

27:07 I mean, it'd be too obvious.

27:08 So he needs a way to sneak it in without

27:10 it ever appearing as normal source code on GitHub.

27:14 So, when you're writing compression software,

27:16 it's very often the case that your software is full of these binary blobs,

27:21 as we call them, so just lumps of binary which are

27:24 used to test the compression or the decompression is still working.

27:28 Nobody reads these test blobs.

27:30 They're included without ever appearing in the human readable source code.

27:33 They're assumed to be garbage data.

27:36 But for Jia, this is the perfect place to hide his payload,

27:40 inside something that at first glance looks harmless.

27:44 But in reality, it's a Trojan horse.

27:48 But with a Trojan horse inside of XZ,

27:50 it's still just a lump of data in a binary blob.

27:54 He has to unpack it.

27:56 So, in the code that builds the project,

27:58 he slips in a small easy-to-miss change.

28:00 It hides among all the automatically

28:02 generated code and quietly unpacks his payload,

28:05 inserting it into the XZ library.

28:08 But now that it's inside of XZ, it still has to pick the right time to act.

28:13 On to step two, Goldilocks.

28:18 Jia's end goal is to compromise a very

28:20 specific part of the SSH connection process, the RSA authentication step.

28:25 He realizes that if he can slip a small malicious component in there,

28:29 let's call it the payload,

28:30 then every time SSH checks for a key, his code will run first.

28:34 It will quietly look for a special master key that only he knows,

28:38 and if it sees that key, it'll let him straight in.

28:40 If it doesn't, it'll call the real code and no one's the wiser.

28:44 So, he will have his backdoor entrance to OpenSSH.

28:47 But he can't just go in and rewrite RSA Decrypt,

28:50 the function that verifies the client's identity during the login.

28:53 It's not that easy.

28:56 See, when you build an application, you could take all the code you need

28:59 from different libraries and bundle it into your application.

29:02 But there's a big drawback to this approach.

29:05 If 10 different applications on a system all bundle the same library,

29:09 you end up with 10 separate copies on your machine, so it's redundant.

29:13 That's why modern systems mostly use shared libraries.

29:17 When an application starts, the linker fills in a table of addresses.

29:20 These addresses point to the functions and variables

29:23 it needs from the libraries it links to.

29:25 That table is called the Global Offset Table, or GOT.

29:30 Now, when it wants to use something from a shared library,

29:32 it just checks the GOT and jumps to the right spot in memory.

29:36 RSA Decrypt doesn't belong to OpenSSH at all.

29:39 It comes from a shared crypto library.

29:41 So to hijack authentication,

29:43 Jia can overwrite the GOT entry that tells SSH where it is.

29:48 And to do that, he can use a little known tool called an IFUNC resolver.

29:52 The IFUNC is used where let's say you wanna optimize

29:56 your code to run on Intel's hardware and AMD hardware.

30:00 Now, you could write the software just for Intel,

30:02 and it would run very fast on Intel

30:04 and it probably would run very badly on AMD hardware.

30:07 [Henry] Instead, you keep multiple versions of the same function

30:10 and the IFUNC resolver picks the right one for the hardware you're on.

30:13 At first glance, that sounds like a way for Jia to trick the system

30:16 into thinking it's running hardware that needs

30:19 his own compromised version of RSA Decrypt.

30:21 But there is a catch.

30:23 A library can only define IFUNC resolvers for its own functions.

30:26 And since RSA Decrypt doesn't belong to XZ,

30:29 it can't use an IFUNC resolver to override it.

30:32 But IFUNC can still help him.

30:34 So it will, very, very early on in the running of the program

30:37 it will do this sort of determination of what hardware is available,

30:40 and crucially, it does let you run your own code in the library very early on.

30:48 Now, at this early stage, from within an IFUNC resolver,

30:51 Jia could try to directly rewrite the GOT entry for RSA Decrypt.

30:56 But at this point, the system is still filling in the GOT,

30:59 so even if Jia changes the RSA Decrypt slot,

31:02 the loader will come along later and write

31:04 the real address back in, wiping out his change.

31:07 And there's a limit on the other side as well.

31:09 To make this sort of hijacking harder,

31:11 once every entry is filled on the GOT, the system marks the table Read Only.

31:16 That means that if Jia waits too long, the RSA Decrypt entry is frozen.

31:21 So he has to slip it in at a very precise moment.

31:24 After the RSA Decrypt entry is filled in legitimately,

31:27 but before the table gets marked Read Only.

31:30 And that tiny window is the Goldilocks zone.

31:34 And to hit it, he's gonna need another tool.

31:37 So, linking shared libraries in the GOT often leads to bugs,

31:41 so Linux has a special debugging feature that tracks what the system's doing.

31:45 It lets you run code whenever the linker writes a symbol's address into the GOT.

31:50 It's called a dynamic audit hook,

31:52 and normally you'd use it to profile performance.

31:54 But crucially for Jia, there are no real guardrails.

31:58 The hook can run any code he wants.

32:00 And this is where IFUNC finally pays off.

32:03 Jia uses an IFUNC resolver to set the audit hook early.

32:06 Then, when the linker writes in the real RSA Decrypt address,

32:10 the hook fires and swaps in his payload.

32:13 Right in the middle of the Goldilocks zone.

32:17 There is one final complication, though.

32:19 Audit hooks are normally configured by the system, not by libraries like XZ.

32:23 So when Jia is first looking for the audit

32:25 hook variable that he's supposed to rewrite,

32:27 it's actually hidden from him, so he first has to find it.

32:31 Within the IFUNC, he scans a small region of binary code,

32:35 hunting for signs of the hook.

32:37 But it's just raw bites, so he writes a tiny decoder to turn

32:40 them back into instructions that he can read.

32:43 Now Jia can find where the hook lives in memory and finally plant his code.

32:47 Then, when RSA Decrypt gets called legitimately,

32:49 it triggers the payload and he's in.

32:52 But now that he's in, what does he do?

32:54 And how does he get out of there cleanly?

32:56 Step three, the cat burglar.

33:00 With Jia's exploit in place,

33:02 SSH isn't just checking for a legitimate login anymore.

33:05 It's also listening for a hidden master key.

33:08 And Jia is careful, he doesn't want anyone else stumbling onto the backdoor,

33:11 so that master key isn't just a simple password.

33:14 It's actually a mini cryptographic exchange of its own.

33:18 First, the backdoor code checks for a shared secret,

33:21 and then, second, it authenticates the user.

33:23 And only if both checks pass does the payload run.

33:26 In effect, it's like the backdoor is running a miniature

33:28 version of the encryption from SSH inside of SSH.

33:32 But in SSH, it uses that encryption to keep the attackers out.

33:36 In this case, the backdoor is using that encryption to make

33:39 sure that it's only the attackers that can get in.

33:41 But he's still careful.

33:43 One of the main ways defenders catch intrusions is through SSH logging.

33:47 So, to cover his tracks, he wipes evidence of the backdoor ever firing.

33:51 And this is on top of the numerous safety checks that he's inserted throughout

33:55 the process to make sure the system supports

33:57 the backdoor and doesn't crash and draw attention.

34:00 And this is the genius of Jia's trap.

34:02 It's cautious and meticulous,

34:05 designed to slip through only where it will run invisibly.

34:08 With all three of these steps complete,

34:10 he can finally control the machine undetected.

34:13 All he needs to do now is get his updated XZ implemented in the next release.

34:18 But just as Jia is completing his backdoor,

34:21 an open source developer requests to remove

34:23 the dependency that links XZ to OpenSSH.

34:26 This would spell disaster for Jia Tan.

34:29 He becomes frantic, pushing harder and harder

34:32 to get his compromised XZ into major Linux releases.

34:35 He gets it into an early experimental build of Debian.

34:38 He files a request to have it added to Ubuntu.

34:40 He's trying to land the backdoor everywhere

34:42 he can before anyone realizes what's going on.

34:45 And it's then that Rich gets his first message from Jia.

34:49 Over the next few weeks, he gets more and more insistent,

34:52 urging Rich to add the updated XZ into the next release of Fedora.

34:55 I'm always very keen to talk to keen upstream contributors,

35:00 contributors who are really excited about new things in their software,

35:05 who are really willing to help us get stuff into Fedora.

35:08 So, you know, that's great, love it.

35:10 That kind of makes my day, it's my happy place.

35:13 Eventually, Jia gets what he wants.

35:15 Rich adds the updated XZ to a pre-release version of Fedora.

35:19 Jia has succeeded.

35:21 Except there's a bug.

35:24 In low-level code like the backdoor, things you normally take for granted,

35:28 like memory management, are not done automatically.

35:31 If a function grabs a bit of memory,

35:33 it also has to give that memory back when it's done.

35:36 And if it doesn't, then every time the function runs,

35:38 it grabs more and more memory and then never releases it.

35:41 Over time, the program just keeps growing.

35:44 That's called a memory leak.

35:45 And to catch problems like this, developers use a tool called Valgrind.

35:49 It runs the program more slowly

35:51 but watches every memory operation for anything suspicious.

35:55 Valgrind is raising hell on Jia's code.

35:59 We put XZ, this version, 560, into Fedora 40.

36:04 We get a bug report initially.

36:07 And the backdoor in XZ specifically is generating invalid writes errors.

36:12 Well, the logic was written by hand, bypassing the compiler's safety checks,

36:16 and so they accidentally wrote outside the memory stack.

36:19 Now, lucky for Jia, all this isn't immediately obvious.

36:22 Rich still hasn't noticed what's happening.

36:25 New software has bugs, right?

36:26 It's the state of nature of software.

36:28 Software is absolutely full of bugs all the time.

36:32 [Henry] Now, the real problem is inside the malicious code in the test file.

36:35 But Jia can't just go and fix that, that would completely expose the backdoor.

36:39 So he invents a cover story.

36:41 He claims that the random data he used to generate the original test files,

36:45 well, it's not reproducible, so he's replacing it.

36:47 And in this updated code, he fixes the memory error.

36:51 It's a very convincing and plausible explanation

36:54 for why this test blob has to be updated.

36:56 But of course, it's not the real reason.

36:59 All right, so now the real fix is in, but if the bug just magically went away,

37:03 it would look a bit suspicious.

37:04 So he has to find a way to cover it up.

37:06 So what he then does is he changes the IFUNC code in a way

37:12 where he adds like a whole bunch of comments and changes to the code

37:17 around it that doesn't actually change the code but is plausible enough

37:22 to look like he's changing how the IFUNC works to fix the Valgrind bug.

37:26 It does, listening to it and I'm like I

37:27 know that this is the evil hacker Jia Tan, but I'm like, ooh, that's clever.

37:32 You know?

37:33 Yeah, I mean, look, the guy is obviously not an idiot, right?

37:37 But none of this is suspicious.

37:41 This is what we expect from compression software.

37:44 And as a packager, it's not really my job to fix every bug in upstream software.

37:51 As soon as it gets to a certain level of difficulty, my thought here is, well,

37:57 Jia Tan has actually been writing this software, right?

37:59 So he's got it all in his head, he knows how it works.

38:02 It's easier for me to just give him the problem.

38:05 And I send the bug over to him and like a day later he sends the fix back.

38:09 From my point of view, it's problem solved.

38:11 It worked, system worked, right?

38:13 I made the right call.

38:14 I don't see, at that point, knowing what I know then,

38:19 I don't see that there's any problem.

38:22 So we downloaded Jia Tan's version of XZ,

38:24 which was available on Fedora publicly, but we made a slight modification.

38:29 Instead of using Jia's secret code, we're using our own,

38:31 and that means that we can take advantage of Jia's backdoor.

38:35 In this case, we're targeting the veritasium.com website.

38:39 And once we get control of it, I got a little trick in store for Derek.

38:43 Now, to make sure I don't mess with any real traffic too bad and lose my job,

38:47 we actually cloned the Veritasium website and put it on a very similar URL,

38:51 but it will work the same.

38:52 Of course, Derek doesn't know that I've covered my bases.

38:56 Oh no.

38:58 Man, when you guys do these things, I just,

39:01 I start to get more and more scared now.

39:03 I want it to work for the video,

39:04 but I also don't want it to work 'cause I don't wanna screw stuff up, so.

39:08 Yeah, it's the risk you take, I guess, letting us run rampant.

39:11 It is a concern.

39:13 I'm gonna execute a script here, which is gonna open up.

39:16 It's opening up a port on the Veritasium server.

39:20 And then on this side I'm gonna execute a little script.

39:26 Uh-oh.

39:27 (Henry laughs) Henrytasium.

39:30 Who is this goof?

39:32 On the main photo, you spent time getting all suited up there.

39:37 Of course.

39:38 Looking sharp, sir.

39:39 Thank you, thank you.

39:41 [Derek] "Videos Derek would never approve of." Uh-oh.

39:44 The concept was over the years that we've worked together,

39:47 you've said no to a bunch of my ideas,

39:49 and I figured now with control of the website it's about time the world saw it.

39:53 "Surviving 7 days living underwater.

39:56 How do saturation divers live at -1,000 feet?" I mean,

40:01 you wouldn't be outside, right?

40:02 So I don't know why you need goggles

40:04 there and like a respirator but you're not underwater.

40:08 "Why it's almost impossible to shoot 4,000 meters." It's a sniper video.

40:14 Yeah.

40:15 "The CIA lied: exposing how the CIA lied about torture." I

40:19 feel like that still goes into a tough territory for us.

40:22 "How xenon gas replaced oxygen.

40:25 I attempted to climb Mount Everest on xenon

40:28 gas." That sounds like a terrible idea.

40:30 This is what this whole video is about,

40:32 this whole video is just about trying to get me to green light your projects.

40:37 You know, if people like these video ideas,

40:39 they can feel free to let us know in the comments and we can actually make them.

40:43 The top upvoted comment one, I will green light happily.

40:48 Let's go!

40:49 Is this live to the public right now?

40:51 It is live, yeah, it's live on the server, yeah.

40:53 If anyone's on the website right now, that would be very strange for them.

40:57 Look, I'm not pleased, I would like you to change it back.

41:02 It doesn't seem like this should be possible on a Linux server.

41:06 So the big question is, how did you do it?

41:09 The address is the server, the seed is our code to get in, and then

41:13 the command is what we're doing to essentially open up,

41:16 in this case nc, which is like opening up a port

41:18 on the machine that we can then access from this second terminal.

41:21 Then what we're doing is on this side we're running

41:24 a script that's connecting to that port that's just been opened up,

41:28 copying our files and then by the end

41:30 we're gonna have root access on the server.

41:33 That means that it thinks that we own the thing.

41:36 That's so crazy.

41:37 This is a very scary hack.

41:40 I do not like it.

41:42 Another thing is that this is a very obvious way of demonstrating this attack.

41:47 Like I've changed everything on the website,

41:48 you immediately know that I've gone in and hacked the server.

41:51 If we were doing this for real, we would do it a lot sneakier.

41:54 I mean, as you say, right?

41:56 The thing to do would not be

41:57 to totally rework someone's website so everyone notices,

42:00 but to change it subtly so nobody notices so you can skim data or, yeah,

42:05 like get credit card details or take

42:08 payments to a different location, stuff like that.

42:11 So you can copy anything you want,

42:13 you can change anything you want, you can delete anything you want.

42:15 So if there's any interesting documents or crypto tokens,

42:19 any files you're interested in, those are yours now.

42:22 If there's secret communications going across these, and let's keep

42:26 in mind all of our communication networks are also built around Linux,

42:30 those communication streams are yours now.

42:33 If you wanted to encrypt something and ask for ransom, that's possible now.

42:38 [Henry] The possibilities really are endless.

42:40 After two and a half years of hard work,

42:42 slowly infiltrating the XZ Project and weaving

42:45 in this ingenious backdoor, Jia's done it.

42:49 He now has free rein on any machine that installs the new Fedora pre-release.

42:53 And he also gets the same access

42:55 on Debian testing and Ubuntu's pre-release environments.

42:58 And with RHEL 10 coming up,

42:59 his code could infect some of the most important computers in the world.

43:02 Now he should be able to relax,

43:04 wait for the release, and he's got his backdoor key.

43:07 But just when he thinks everything's going right...

43:11 (suspenseful music) Andres Freund is a German programmer.

43:19 He's not a security researcher, he's not a hacker.

43:23 He's just an employee at Microsoft working

43:25 on an open source project called Postgres.

43:28 One day in March 2024, he tries out the unstable release of Debian

43:31 to make sure that Postgres will run smoothly.

43:34 But while checking the server connection times, he notices something odd.

43:38 A slowdown.

43:40 It's not much.

43:40 In the worst case, it's only half a second,

43:43 but it's enough to make Andres suspicious.

43:45 We tested the connection times ourselves on our own version

43:48 of the XZ hack and we found the exact same thing.

43:50 Consistent slowdowns of about 400 to 500 milliseconds.

43:54 Andres had already seen the problems with XZ and Valgrind

43:57 weeks earlier and this only makes him more suspicious, so he digs in deeper.

44:01 He looks at recent additions to OpenSSH and traces

44:04 the delay back to an update in XZ.

44:07 He sees the binary test files but notices that they were never used in a test.

44:11 It's even stranger.

44:12 Andres tries to get back to work, but he can't stop thinking about it.

44:16 [Andres] I remember sitting in a bunch of meetings and like

44:18 not really being able to concentrate because it feels like,

44:23 I need to continue looking into this.

44:25 Eventually, Andres sees it.

44:26 This isn't some bug, this is a backdoor.

44:30 And this backdoor is meticulous.

44:32 It hunts through memory to find the audit hook,

44:35 it implements a decoder to read those raw bites,

44:37 and then it wraps everything in custom encryption and safety checks

44:41 so that it only triggers on the right kind of connection.

44:44 I mean, it even garble its own strings so that it won't be detected.

44:48 It's incredibly cautious.

44:50 But all of that takes time,

44:51 and in the end, that's what grabs Andres's attention.

44:55 If they had done less obfuscation,

44:56 I probably would not have noticed that anything was wrong.

44:59 [Henry] Now, XZ's security contact is Jia Tan,

45:02 so Andres can't exactly report it through the usual channels.

45:06 Instead, he emails the Debian security team directly and posts

45:09 a detailed report to a public security mailing list.

45:13 Then, all hell breaks loose.

45:16 I'm called up on I think it was a Friday evening, in fact,

45:22 I'm sure it was a Friday evening, to join a internal Red Hat meeting.

45:28 It's immediately obvious that this is not a normal

45:31 meeting because like our head of security is there.

45:35 It's explained to me that it's been found

45:39 by somebody in the community that XZ has a backdoor,

45:42 and immediately I'm like, WTF?

45:45 How did this happen?

45:47 To cover their bases,

45:48 Red Hat quickly rolls Fedora back and tells all their users to revert,

45:52 and the whole open source community starts digging

45:54 into the project to understand what went wrong.

45:58 One thing is clear, though.

46:00 Andres is a hero.

46:02 Now, the fact that this was discovered

46:04 in a different test at all, that was lucky.

46:06 But then what are the chances that someone who

46:09 isn't looking for a security bug spends days investigating this?

46:13 So, big kudos to the researcher, and yeah,

46:17 saved us all from possibly a doomsday on the internet.

46:22 I think that Andres did a brilliant job because he did what I should have done,

46:27 actually, which is I should have looked at the, you know,

46:29 I should have looked at the bug when I saw it and I should have gone there,

46:35 you know, like a crazy hound sort of sniffing

46:38 around trying to find out what's going on.

46:40 [Henry] Andres even gets a shout out from the CEO of Microsoft.

46:43 But when the story breaks, the mainstream response is surprisingly muted.

46:48 Actually, I'm still surprised now that the mainstream

46:52 news outlets haven't really covered this very much.

46:55 Well, I can tell you how many systems would have been compromised,

46:58 which would have been millions,- Anything from spying,

47:02 to ransom, to just taking down entire countries,

47:07 you could have done it with this backdoor.

47:10 [Henry] I guess the big question is, who is Jia Tan?

47:14 That's the question, isn't it?

47:15 Okay, so my feeling is that Jia Tan,

47:18 the person that I talked to I believe is one person,

47:22 but I also believe that behind him must be a group of people.

47:27 And they worked for quite a while.

47:30 I mean, they were at this for perhaps two and a half years that we know about.

47:35 If you look back at the accounts pressuring Lasse, they share some similarities.

47:39 They use free email addresses and they have

47:42 almost no footprint outside of the XZ threads.

47:45 These were very likely sock puppet accounts,

47:47 identities manufactured to apply pressure as part

47:51 of a multi-stage social engineering campaign.

47:54 Now, who spends a million dollars and takes two and a half years

47:57 to attempt to break into every hotel room on the internet with a master key?

48:02 (suspenseful music) I think it's not

48:05 a criminal organization because I don't think

48:07 a criminal organization would have that patience

48:10 to spend that time without any real return.

48:13 So I think it has to be a nation state actor, here.

48:20 A lot of the aliases, like Jia Tan, they sound like Asian names,

48:24 and the published changes are all timestamped in UTC+8, Beijing time.

48:29 So the signs point to China.

48:32 And that's why it's probably not China.

48:35 I mean, why would they make it that obvious?

48:37 Every other part of the operation has been so meticulous, so cautious.

48:41 And they also worked on Chinese New Year, but not on Christmas.

48:45 And over the years, there were nine changes

48:47 that fall outside of the Beijing time into UTC+2,

48:51 which is a time zone that includes Israel and parts of Western Russia.

48:56 That's why some experts have speculated that this could be the work of APT29,

49:01 a Russian-state-backed hacker group also known as Cozy Bear.

49:05 But again, do we know?

49:07 No, of course we don't know who it is, and we likely will never know.

49:10 Jia Tan himself just disappeared as soon as this exploit

49:14 became publicly known and never heard from again.

49:18 In a sense it doesn't matter whether this was Russian or Chinese or Iranian.

49:22 We need to protect from these types

49:24 of backdoors no matter where they're coming from.

49:26 I see this as like, you know, the canary in the coal mine of what's

49:30 gonna be happening as attackers get more sophisticated,

49:35 they make fewer mistakes.

49:37 You know, the gloves are off in a way.

49:39 I don't think that the Linux community is fully,

49:44 you know, is fully ready for this yet.

49:49 In the aftermath of XZ, the open source community poured over countless

49:52 small similar projects looking for similar campaigns,

49:56 but they found almost nothing.

49:59 I'm worried that we didn't find other backdoors.

50:00 The incentives are just too clear.

50:02 There are state-sponsored parts of either governments,

50:07 militaries or even private contractors working for states

50:11 that are all preparing for the next cyber escalation,

50:15 some kind of a war, some kind of a geopolitical conflict,

50:18 and where are all of those backdoors?

50:20 There's just too many people incentivized to put

50:22 backdoors for the few backdoors that we're actually discovering.

50:27 Now, some experts have argued this reveals

50:29 a fundamental flaw in the open source model, but not everyone agrees.

50:33 Closed source software would be no better here.

50:35 In fact, who's to say that there aren't already state spies working as paid

50:41 software engineers at some of the larger

50:44 companies putting in exactly backdoors like this?

50:46 But then there would be no community member

50:49 running free testing and detecting this by chance.

50:52 This backdoor, if anything, underlines the ethos of open source.

50:57 I mean, just think of what it took to get this done in public.

51:01 There was a multiple-year social engineering campaign,

51:04 there were all these layers of misdirection,

51:05 and then there was code that was designed to withstand constant scrutiny.

51:09 Compare that now with a closed source hack.

51:12 Sometimes all it takes to get a backdoor installed there is a court order,

51:16 or you have a public company that can just brush a breach under the rug.

51:21 I actually used to work as an open

51:23 source researcher myself at the Japanese telecom giant NTT,

51:26 and my perspective is that it's only because this is

51:29 an open source project that it's been picked apart,

51:32 analyzed, and turned into a conversation about security at all.

51:35 One that focuses on the fundamental vulnerability.

51:39 It's not the code, it's the people.

51:42 Now, the system has not supported them enough.

51:46 I feel for Lasse that he's given this beautiful

51:51 gift to the whole world and, you know,

51:56 what have we, what has humanity done back to him, right?

52:00 We've poisoned his gift.

52:02 And then I think implicitly a little bit,

52:06 not everyone's saying this, but implicitly we're blaming him

52:11 for not being there to maintain this stuff for free forever.

52:17 But why are we demanding that Lasse do

52:21 anything when he's not being paid for this stuff?

52:24 And that's, in my opinion, quite unfair.

52:28 On this Saturday evening, we were working together on a workaround for this bug

52:35 in RHEL 9 that he's added to XZ,

52:38 and he absolutely could have told us to get lost, and didn't.

52:43 What a brilliant guy.

52:48 (electronic beeping) (music fades out)