FFmpeg: The Incredible Technology Behind Video on the Internet | Lex Fridman Podcast #496

0:00 The important is, is your code good?

0:02 We care about excellent code.

0:04 We don't care who you are.

0:05 Like maybe you're a dog.

0:07 I don't care, right?

0:08 I don't care where you come from.

0:09 I need to look at your code.

0:11 Oh, yeah, but I'm an engineer at this very large company in Italy,

0:15 in Germany, in the US.

0:17 We don't care.

0:18 We care about the quality of your code because

0:21 this is what defines our community and which means

0:24 that we have a lot of people who contribute

0:26 who are some very different backgrounds and very introverted.

0:29 Sure.

0:29 But that's okay, right?

0:32 FFmpeg is probably one of the biggest CPU users in the world.

0:34 Everything we've just said in the past couple of minutes,

0:38 every sentence is someone's lifetime's work.

0:40 There are books about every sentence.

0:41 So the level of complexity in many cases is inordinate.

0:46 FFmpeg has one hundred thousand lines of assembly for all the codecs.

0:50 For all codecs.

0:50 Mm-hmm.

0:51 And just this one has two hundred and forty thousand.

0:54 Every cycle matters.

0:55 We are talking about probably three billion devices

0:59 which are going to decode video nonstop because,

1:01 for example, thirty percent of the video from Netflix is now in AV1,

1:06 fifty percent of YouTube.

1:08 This is what peak video codecs should look like.

1:11 Seventy-nine point nine percent assembly, nineteen point six percent C,

1:15 and zero point five percent other.

1:19 And what's incredible is with those tweets, which is factual, people get crazy.

1:25 For the last two years, they go crazy.

1:27 No, intrinsics is fine.

1:28 The compiler-- You can optimize your compiler.

1:30 Auto-vectorization, it's your fault.

1:32 You don't understand.

1:33 And we've tried that forever, right?

1:35 For two years, and two years later,

1:38 showing hundreds of examples of handwritten assembly.

1:40 No, no, no, you're doing it wrong.

1:42 The compiler can do this.

1:43 The intelligence agencies tried to, like, say,

1:46 "Can you put a backdoor in VLC?"- Yes.

1:48 Two of them.

1:50 Well, what did you say?

1:51 No.

1:52 Well, I was a lot less polite.

1:54 Basically saying, "Hell no."- Like,

1:56 if we had to compromise our software, we would shut it down.

1:59 This is clear.

2:00 Any tweets Kieran, you regret?

2:04 Tweets I regret?

2:05 Or is it like that, how does the French song go?

2:08 Regret nothing.

2:09 Don't regret anything.

2:10 No, it's because regrets are attacks on your mind.

2:17 The following is a conversation all about FFmpeg

2:20 and VLC with Jean-Baptiste Kempf and Kieran Kunhya.

2:25 FFmpeg is an open source software system

2:28 that is the invisible backbone behind YouTube, Netflix, Chrome, VLC, Discord,

2:34 and basically every platform that touches video or audio on the internet.

2:41 It can decode, encode, transcode, stream,

2:44 and play almost any video or audio format ever created.

2:49 To me, it is one of the most incredible software systems ever developed,

2:54 and it's all done by volunteers.

2:59 VLC is also a legendary piece of software.

3:02 It is an open source media player that plays basically anything you throw at it,

3:07 any format, any platform, no ads, no tracking.

3:11 It has been downloaded over six billion times, and again, for me,

3:15 it has been one of my favorite pieces of software ever,

3:19 with the most legendary logo, which I, of course,

3:23 had to honor in this conversation by wearing

3:26 the VLC traffic cone hat the whole time.

3:31 So again, above all else,

3:33 thank you to the incredible volunteer engineers who put their heart and soul

3:37 into this code that has been used and loved by billions of people.

3:42 Thank you.

3:43 And about the two great engineers and human

3:46 beings I'm talking to in this episode,

3:50 Jean-Baptiste is the president of VideoLAN and is

3:54 a key figure behind VLC and FFmpeg.

3:58 Kieran is a longtime codec engineer, FFmpeg contributor,

4:02 and the man behind the now infamous FFmpeg

4:06 account on Twitter/X that I recommend everybody follow

4:10 for the memes and for the unapologetic celebration

4:14 of open source and great low-level software engineering.

4:20 Let me also say that it's inspiring and humbling that so much of modern

4:25 civilization rests on software built by people

4:28 who are not chasing fame or money,

4:30 but are obsessed with the craft of engineering.

4:34 We live in a world where billions of people consume video

4:37 every day without ever thinking about the invisible machinery underneath it.

4:41 But that machinery matters.

4:43 Open source infrastructure matters.

4:46 It is one of the great examples of human

4:49 beings quietly collaborating across borders to build something useful,

4:53 durable, and elegant for the rest of us.

4:57 And so this conversation is not just about codecs and media pipelines.

5:02 It is also about the deeper spirit of engineering

5:05 and generosity that makes projects like FFmpeg possible.

5:11 Again, I can never say it enough.

5:14 Thank you.

5:16 This is the Lex Fridman Podcast.

5:18 To support it, please check out our sponsors in the description,

5:21 where you can also find links to contact me,

5:25 ask questions, give feedback, and so on.

5:28 And now, dear friends, here's Jean-Baptiste Kempf and Kieran Kunhya.

5:35 So the legend goes VLC can open everything.

5:38 What's the weirdest thing that you know that it can open?

5:43 You know, there is a ton of people

5:45 who are using VLC to record VHS videos, right?

5:47 Like, it's just like you plug it with a capture

5:49 card and you can basically record VHS video.

5:52 Well, how does that work?

5:53 Basically, it's, you know,

5:54 those type of capture card where you can put a Peritel in or-...

5:58 or RCA, and you put that, and actually VLC can play those type of cards,

6:02 and there is a module which allows

6:04 to control directly some of those VCR camcorders.

6:06 We support DVD audios lately, right?

6:09 We spent the summer working on DVD-Audio support,

6:13 and like there is no, no one's making any DVD audio support.

6:16 There is a custom encryption schemes.

6:19 What about Lucasfilm?

6:20 Oh, yeah, and there is of course all the weird codecs support,

6:23 game codecs supported by FFmpeg.

6:25 The one Star Wars video game, the first ten- second opening sequence,

6:28 someone has gone and implemented that and made sure that's bit exact on one

6:32 disc that existed at one time of one little sequence in the game.

6:36 And then funnily, there was a...

6:38 At one VideoLAN conference,

6:39 we made a competition to make the weirdest and most horrible file ever...

6:44 and see if VLC could play it.

6:46 What did it end up being?

6:47 What's the file?

6:48 It was an MKV file made by Derek-...

6:52 which each of the frame was changing resolution, aspect ratio-...

6:56 rotation and it was like-- Did it work?

7:00 Yes.

7:00 And there was another one where

7:03 the whole video was actually animated subtitles, right?

7:06 SSA, right?

7:07 So-- Yeah.

7:08 I remember that, yeah-...

7:09 each, this one was- And so each frame was a black frame,

7:12 but on top of that there was a, a subtitle that was animated for each frame.

7:16 There was a file that's a valid ZIP and a valid

7:18 MP3 at the same time or something like that, so.

7:21 So yeah, we'd made a competition of stupid files.

7:24 And it worked.

7:25 It opened all of the stupid files.

7:27 Yes.

7:27 By the way, For people who are not familiar, I am wearing a hat.

7:31 Would it be fair to say this is the best worst logo of all time, the cone?

7:37 Yeah, by far, right?

7:38 The logo of VLC is so iconic, right?

7:40 Like we are a team with a small

7:42 number of people and the icon is known everywhere.

7:45 I go to middle of nowhere in India or in China, people know the cone, right?

7:50 And 25% of the website traffic that comes

7:53 to our main website is cone player, right?

7:56 So, so many people don't know VLC, right?

7:58 They know the cone player.

7:59 That's the thing they Google for is cone player.

8:02 Yeah.

8:02 They go on Google and they put cone player and they download VLC, right?

8:05 So that's iconic.

8:06 And once we tried to change it as a joke, right?

8:08 We said it was going to be a type of uh,

8:12 caterpillar construction and we said that during April 1st-...

8:18 and we had around 10,000 emails saying, "No,

8:20 don't change the logo," and so on, right?

8:21 So it's so iconic, right?

8:23 It's so distinctive, right?

8:24 If you want to do a video player,

8:26 you're going to put a play button on a TV, right?

8:28 And that's a YouTube, YouTube logo, right?

8:31 It's unoriginal.

8:31 This one is orange, right?

8:33 Yeah.

8:34 It's very bright and it's weird.

8:37 And it's ridiculous and it's absurd and it's hilarious.

8:39 It becomes meme and meme becomes culture.

8:41 Yeah.

8:41 And you keep it and you know about it and you know that in 20 years,

8:45 like you still have, going to have the cones and remember,

8:47 oh yeah, that was a video player.

8:49 Yeah.

8:50 And we'll talk about, you know,

8:52 the mission of FFmpeg being a kinda the archival aspect of it.

8:55 So you can think about 1,000 years from now

8:57 we'll have all these videos that only VLC can open.

9:01 Humans, human civilization has already destroyed itself multiple times

9:04 and the only thing that will remain is this like,

9:07 you know, the cockroaches will be crawling around and it'll be the VLC logo-...

9:11 with some of the archival footage that VLC can open.

9:14 And the aliens will show up and they'll press

9:17 play and they'll get to see it all 'cause-- Well, really, really hope so, right?

9:19 But there is also so many memes where people say, "Well,

9:21 I'm sure I can put a pancake inside my DVD

9:23 drive and VLC will play it." Like-- Can they?

9:26 No, we tried.

9:27 It doesn't.

9:27 Um-- Doesn't.

9:28 but we actually have a video of us trying that.

9:30 Didn't work.

9:31 A codec for physical reality, I don't know what that would even look like.

9:34 There was a guy who did that, right?

9:36 He printed a small cone, right?

9:38 Like the ones we distribute as goodies and inside he put

9:41 an RFID chip which was his way of playing a movie, right?

9:45 And so he-...

9:47 put this on a RFID player and when he put

9:49 that it was playing like The Last Star Wars and so on.

9:52 So instead of having like DVD boxes, he had like VLC cones all around and he

9:56 plugged that and that was like physical objects.

10:00 So the thing that we're talking about is everything around video codecs,

10:03 video encoding, video decoding,

10:05 video streaming, video player client that I'm wearing on my head,

10:10 the entire ecosystem enabling free media.

10:12 We'll talk about FFmpeg, we'll talk about VideoLAN,

10:15 VLC and all the other incredible video technology

10:19 that is used probably by billions of people.

10:23 So JB, you're the lead developer behind the legendary VLC player.

10:29 Kieran, amongst many other things,

10:31 you're lead developer behind the legendary FFmpeg handle on Twitter.

10:35 And both of you have spicy opinions I would say.

10:40 So today we wanna talk about FFmpeg and VLC.

10:45 For context for people who are not aware and I'm sure basically everybody

10:50 listening to this have used these two

10:53 technologies probably regularly without knowing it.

10:57 So FFmpeg underlies basically most video on the internet including YouTube,

11:02 Netflix, Chrome, Firefox, of course VLC and countless other video platforms.

11:08 It is estimated that over 90% of video

11:11 processing workflows online and offline involve FFmpeg.

11:14 VLC has been downloaded at least 6.5 billion times.

11:21 But likely that number,

11:23 'cause it's impossible to really count the number is much higher than that.

11:28 Virtually any operating system supports virtually any media format.

11:35 The limitation being it can't open pancakes.

11:38 So, Can we just lay out some of the basics

11:42 to help people understand what's involved in all of this?

11:45 So when we press play on a video player like VLC, what happens?

11:51 What— How does it go from the file or the stream

11:55 to the pixels on the screen and the sound on the speaker?

11:58 What are the big stages to be aware of?

12:01 So there are several stages, right?

12:03 The first stage is to get from an address, right,

12:06 which is the type of URL, to give you a byte of streams, right?

12:11 So this would be, for example, HTTP, file, DVD, right?

12:14 You give the path to the media, and it gives you a stream of data.

12:19 The stream needs to be cut up by what's known as the container,

12:22 the demultiplexer or demux.

12:24 We'll try and keep the jargon light throughout this, but it

12:27 needs to go and start demarcating video and audio frames.

12:29 So it just gets data from the operating system blocks at a time

12:33 and needs to start cutting these frames up into compressed data.

12:36 It then needs to start doing simple parsing of the video frames-...

12:41 mainly to figure out whether that codec is

12:44 GPU decodable or needs to fall back to software.

12:47 We're very sort of used to assuming the GPU will play all of these things.

12:51 There'll be hardware acceleration.

12:52 I think it's up to forty- five percent of files are not GPU decodable.

12:56 So these need to be probed.

12:57 They need to be detected.

12:59 There can be variants of a given codec, some of which are decodable on the GPU.

13:03 Different vendors of GPU might have different capabilities,

13:07 so those need to be detected.

13:08 So if it's GPU capable, you pass it through to the GPU black box.

13:12 So now if there's a software fallback,

13:15 that means in the beginning is to first do deentropy coding,

13:19 so removing the mathematical coding of the bit stream.

13:22 So this uses capabilities such as Huffman coding or arithmetic

13:25 coding to actually decompress the mathematical layer of the bit stream.

13:30 We then need to start reading the syntax elements for intra prediction.

13:34 So intra prediction are like still images of the video, so your I-frames.

13:40 So this works and operates in the spatial domain.

13:42 So you do your intra prediction in spatial domain.

13:44 You have a residual because your prediction

13:46 isn't quite matching that of reality.

13:48 So you've made a prediction, but then there's a little bit left,

13:51 and that's what's known as the residual.

13:53 This is stored in the frequency domain,

13:55 and these are quantized to decompact their space.

13:59 We then need to do the inverse transform to bring

14:02 them back to the spatial domain and apply these residuals.

14:06 So a lot of the process of the decoding is this thing is compressed.

14:10 And you have to predict the highest quality thing that's supposed to go there.

14:15 I-frame-...

14:16 is the best representation you have spatially.

14:19 And then there's a lot of temporal

14:21 compression that can happen depending on the codec, and then you're predicting.

14:25 You're predicting what the reality that was captured in this rawest form.

14:30 Yeah, because what people don't realize is that the compression

14:33 on video and audio is one hundred times, right?

14:37 Like, people don't realize how compressed we, we do, right?

14:40 For audio, you move, you compress by, when you go from normal audio to MP3,

14:45 you compress by ten times, right?

14:47 When, when you move to video,

14:48 you need one hundred times, two hundred times, right?

14:51 So you need to remove all the details,

14:53 but that you don't care about because all the compressions that we do,

14:57 and that's very important,

14:59 people forget about that, is to be viewed by humans, right?

15:02 So all the codecs, either for audio, mimic basically how your ear works, right?

15:07 And a lot of things about, like,

15:09 the response on the ear and same for your eyes, right?

15:13 And so, for example, on video, we don't work on RGB, right?

15:17 Everyone expects to work in RGB.

15:19 We don't, right?

15:20 We move to YUV, which is basically one is luminance,

15:23 brightness, and the other are colors.

15:26 And this matches your eyes,

15:27 where inside your eyes you have the cones and the buttons, right?

15:30 With some of them look on brightness and more on, on the other on colors, right?

15:34 So we need to compress a lot, and so we need to degrade.

15:37 But in order to degrade, we need to match the human perception,

15:40 and this is why it's so difficult.

15:42 And then we need to use the maximum power,

15:46 mathematical power, very complex technologies.

15:48 We move to the frequency domain, as Kieran said.

15:51 We do a ton of dequantizing,

15:53 in order to get the best compression, but it still looks good.

15:59 You're trying to compress in order to maximize

16:02 the highest quality thing for human perception.

16:05 That is correct.

16:06 And this is very important, right?

16:07 Compression is not like a ZIP, right?

16:09 A ZIP, you have data in, you get data out, right?

16:13 And you try with all the ZIP compression to arrive with the limit.

16:17 Here we are degrading the signal, right?

16:19 And so we need to degrade both the audio

16:22 and the video signal in the best way possible.

16:25 And we can do that, but it involves, first,

16:28 a lot of theoretical knowledge about how the eye works,

16:32 but it, a lot of mathematical change, a lot of mathematical tricks, right?

16:36 For example, when you move to RGB and you do go to YUV, for example,

16:41 what we do very often is that we scale

16:44 down the resolution of the color compared to the brightness.

16:48 And most of the time, just this without compression,

16:51 it divides the size by two, but most people don't see it, right?

16:56 And so on and so on, right?

16:58 And then you go to very complex mathematical change.

17:02 So of course Fourier transforms, which de facto are not Fourier transforms,

17:06 they are like discrete cosine transform, but that's the same idea.

17:10 So frequency domain we split the video by blocks, right?

17:13 So that's why when it's wrongly decoded, you see those blocks and badly encoded,

17:18 you see those blocks,

17:19 and so on, to arrive to compression states that are insanely high, right?

17:24 And each generation of the codec is like thirty percent less-...

17:29 for the same quality, right?

17:31 And this requires amount of power of computational power that is huge.

17:36 No, no, but you should elaborate.

17:37 It's thirty percent better, but an order of magnitude,

17:39 perhaps, perhaps even two orders of magnitude more compression power.

17:43 That's the big difference.

17:45 What do you mean by compression power?

17:47 Sorry, CPU power to achieve that level of compression.

17:49 Oh, yeah.

17:49 So you have to be able to leverage

17:51 the CPU and sometimes GPU, like you mentioned.

17:53 And then we should mention that a lot

17:56 of this programming is done at the lowest possible-...

18:01 stack, whether it's C and of course, as the legendary—...

18:05 Twitter handle re-emphasizes over and over, a lot of assembly.

18:10 So what happens globally is that you have an address, right?

18:12 Which gives you with the operating system,

18:14 a stream of bytes, a stream of data, right?

18:16 And this is the first step.

18:18 And the second step arises with demuxing, where you're going to separate audio,

18:22 video, subtitle in type of different tracks.

18:24 And then on each of those tracks, you're going to decompress them, decode them,

18:28 either audio with an audio codec,

18:30 video to video codec, and subtitle to subtitle codec.

18:34 And once you've decompressed those type of things, you have raw images, raw,

18:38 and then you're going to talk with your graphics

18:40 card and your screen and display that.

18:42 And same for the audio, you're going to talk to your audio card,

18:46 which then is going to go in analog to your audio speakers.

18:50 And everything we've just said in the past couple of minutes,

18:53 every sentence is someone's lifetime's work.

18:55 There are books about-...

18:56 every sentence.

18:57 So the level of complexity in many cases is inordinate.

19:01 You know, it's, it's...

19:02 Every sentence has thousands of people working on this-...

19:06 in industry as a whole, books written about it.

19:10 So there's a lot of detail, there's a lot of subtleties,

19:13 there's a lot of both academic and practical realities, both of which matter.

19:20 Uh, we mentioned codecs, but I don't think you mentioned containers.

19:24 So what, what are the actual containers

19:28 for some of the stuff we're talking about?

19:30 So people are familiar with MP4, uh, MOV, MKV.

19:34 So anyway, what are containers versus the thing that goes inside?

19:41 So the container is what we call also the muxer, right?

19:43 When I say demuxing, it means decontainerizing, right?

19:46 So actually, if you look, mux means multiplexer and demultiplexer, right?

19:53 Mux and demux are those.

19:55 And same, a codec is actually coder, decoder, right?

19:58 Um, and so containers are this collection of multiple tracks, right?

20:03 So it's a, what normal people call the file format,

20:07 but it's a bit more, um, subtle than that.

20:10 But the most known one, of course, is MP4,

20:13 but when I started, it was AVI, right?

20:15 AVI was the, the video format from-...

20:18 from Microsoft, and MOV, M-O-V, which became MP4, was a format from Apple.

20:23 In the open source community one of the person that is

20:27 still active on VideoLAN is called Steve Lhomme and started,

20:29 This Matroska format, which is, like, a bit more complex and more future-proof.

20:34 And there are so many others.

20:38 So, I mean, there's a, it's a pretty common thing,

20:40 and maybe it'll even happen in this conversation,

20:43 that people confuse container and the codec, right?

20:47 So confuse MP4 and H.264, for example.

20:49 Is that a horrible violation?

20:52 No, it's not, because technically the name of H.264 is MPEG-4 Part 10.

20:58 Because MPEG-4 is actually a meta specification

21:02 which has several things in it, right?

21:05 There is the Part 2.

21:07 so there is, like, audio codecs, right?

21:09 AAC de facto is MP4 audio-...

21:12 something.

21:12 There are actually several video codecs, right, inside the MPEG-4 specification.

21:16 One of them is MPEG-4 Part 10, called also AVC, called also H.264.

21:22 Right?

21:23 So it's completely the fault of the industry

21:26 to make things difficult to understand.

21:28 So that's very difficult so that people then don't

21:32 understand why sometimes you talk about MPEG-4 Part 10,

21:35 where you mean H.264, and why it's not MP4.

21:38 So you can technically shove in all kinds

21:40 of different codecs inside containers and horribly so.

21:44 But broadly speaking, though,

21:46 MP4 is understood to generally be H.264 plus AAC audio.

21:50 99% of the time that's that, and that, the rest are de minimis,

21:56 the small effects, you know, edge effects really compared to that.

21:58 So it's not the end of the world.

22:00 There, there are people who do get annoyed by that.

22:03 But also in reality, something like VLC, just to point out,

22:06 the file may say .MP4, but it may be something completely different,

22:10 and that's one of the challenges both FFmpeg and VLC have is

22:13 the real world is a completely different place to a three- letter file format.

22:17 And this is very important to say, right?

22:19 Like, for example, in VLC and in FFmpeg, we discard the file format, right?

22:24 We look into the file to understand what's in it because so many people,

22:29 like, they say, "Oh, it's a video, it should must be MP4," but technically

22:32 it's an MOV or maybe it's a MKV, right?

22:35 So we analyze in real time everything that we have, and we don't trust-...

22:42 the format.

22:43 So what information does the fact that it's .MP4 give you?

22:46 It helps, right?

22:47 It gives you a hint, right?

22:48 Just like, oh, it's finished by .MP4.

22:50 I will start first by opening,

22:54 probing it with the MP4 container demuxer to see, well, it should be that.

22:59 But I don't trust it, and if I'm lost, I say, "Okay,

23:02 maybe I'm going to try it." So it bumps the priority of the module.

23:06 So how do you get to...

23:08 just to take a bit of a tangent there.

23:11 You know, the dumb thing is if you try the MP4,

23:15 but it turns out it's a different codec than you would have expected,

23:19 Most players just break there.

23:21 Yes.

23:22 Yes.

23:22 And so how do you not break?

23:25 There's just philosophically,

23:25 I'm sure there's a bunch of stumbling blocks along the way

23:29 where it's easy to just break and stop, freak out.

23:32 That's it.

23:33 How does VLC not?

23:34 This is why VLC is popular.

23:36 But the reason is because actually VLC was,

23:40 is just a client of a streaming solution called VideoLAN

23:44 from a very long time ago, from the late '90s.

23:48 And when you're playing video which are on UDP,

23:52 right, in network, they might be damaged, right?

23:54 So you don't trust your inputs, and this is very important into the security

23:57 is that you don't trust your inputs.

24:00 So everything in VLC is prepared to work with broken files.

24:06 And it's a philosophical idea from the beginning,

24:09 and everything is engineered into that.

24:12 And it's a culture, right?

24:13 And so, for example...

24:15 And VLC became very popular on that because

24:17 a long time ago when people were pirating content

24:20 which they do a lot less today-- And none of us ever have-- No, of course not.

24:26 Um—...

24:27 the metadata to play some files like AVI is at, at the end of the file, right?

24:32 And when you're downloading, you don't have that, right?

24:34 So VLC was just like, "Hey, this file is broken,

24:36 but I'm still going to try to interpret it," and this was very useful.

24:41 We hinted at the awesomeness of the various different stages.

24:45 We hinted at the awesomeness of codecs,

24:47 the depth and the richness and the complexity of everything involved there.

24:51 What— Let's try to define what is a video codec?

24:55 What's involved there?

24:56 What does it mean to compress something?

24:58 You already started to hint at it—...

24:59 but can we elaborate a little bit more?

25:02 So there's a huge amount of redundancy in any video both spatial and temporal,

25:06 and the point of any video codec is to remove this redundant data,

25:10 use mathematical properties as part of this reduction process.

25:13 So more often than not,

25:15 using several orders of magnitude more compute to compress because that's

25:19 more costly versus both costly both financially and in CPU resources—...

25:24 versus the decompression.

25:25 So it's asymmetric in that respect.

25:27 Uh, often the case because compression is done once,

25:29 but there could be lots of viewers of another file.

25:32 So to take that information and compress it by 100x, 200x,

25:37 removing redundant information and using

25:40 mathematical properties to make that small,

25:42 but also have properties such as error resilience.

25:45 So as, as JB suggested, VLC in the beginning was,

25:49 was used to play UDP network feeds, and UDP network feeds lose packets.

25:52 And so some of the design goals of a codec is also to be recoverable.

25:58 You need to actually be able to join a stream.

26:00 It's not necessarily a file.

26:01 You need to join, get on the decoding process, and start decoding.

26:06 And, and to give a more image to people who are not familiar, right?

26:11 Like, when you're going to see any type of movie, right?

26:14 You're going to see the camera is going to pan, right, and travel.

26:17 And you realize that, for example,

26:19 all the background is the same from, for, like, a minute, right?

26:23 Or—...

26:24 thirty seconds, right?

26:24 So you can reuse the cloud that you see uh,

26:27 on the background, you can reuse that from a frame to another, right?

26:31 And so it's, gets the more, the more memory you have,

26:35 the more power, the more comparisons you can make, right?

26:38 And so the more compressed you can be.

26:41 And most of the modern codecs are basically doing that.

26:44 So just to make it even more explicit.

26:47 So what is video?

26:49 Video is a bunch of pixels often RGB.

26:54 You have three values, and you have a grid of pixels, and you have, let's say,

26:59 twenty-four or thirty or sixty, frames a second,

27:03 and you just have all these pixels repeating and showing different stuff-...

27:08 thirty times a second.

27:10 And so the question, the philosophical, the technical question is,

27:14 how can I compress all of that, store all of that at 100x?

27:20 Yep.

27:21 Or 1,000x, right?

27:22 1,000x.

27:23 The target is 1,000x, right?

27:24 And the goal is when you say redundancy, what is redundant?

27:30 Meaning stuff at best that humans wouldn't notice if it was missing.

27:35 So for example, you have a picture of a cloud, right?

27:38 And from the next frame,

27:39 they're still going to be the same cloud, so it's redundant.

27:41 You could just put it once and not do it, right?

27:44 Or you have a black background behind me, for example.

27:47 The black is the same on the whole picture, right?

27:49 So you can say, "Well, you know, in this picture,

27:51 take the pixels that you have on the top left and the one on the top right.

27:55 I'm not going to give the value.

27:56 I'm just going to tell you it's the same at the top left." And then

27:59 you can say for frame one reuse

28:02 something from the previous frame or the previous,

28:04 previous frame, and so on and so on, right?

28:07 So you could...

28:09 Basically, it's unlimited,

28:10 but then it's limited in terms of memory or in terms of compute power.

28:15 Because, for example, if you need to compare pixels on two

28:18 hundred frames in the past on 4K resolutions, it's a huge amount of compute.

28:26 And then when you're showing it, you have to do the decompress of all of that.

28:30 So is it the codec, the, has the encoding

28:34 and the decoding is a coupled process that you're developing?

28:37 Yes, exactly, right.

28:39 And those are two different trade-offs, right?

28:42 Are you going to compress more?

28:43 But then it might be more difficult to decode.

28:47 Are you going to comp- to make it a codec

28:50 that is more complex to encode and easier to decode?

28:53 Are you going to make a codec that is

28:55 easier to encode because you need to be fast,

28:57 but then the, the client side, the, the player is going to spend more time?

29:00 That's why you have so many different type of codecs,

29:03 is that it's not always easy.

29:05 And to make it even more complex,

29:08 modern codecs like AV1, AV2, or VVC are actually not codecs.

29:12 They are a collection of tools, right?

29:14 There are multiple tools,

29:16 multiple codecs in the same codec to, depending on the image,

29:19 get the more compression.

29:22 So just to elaborate, codecs like AV1,

29:24 VVC have a much wider, have a wide audience.

29:28 It could be a screen share content, it could be video, it could be animation.

29:33 All of these require different coding tools.

29:37 So what happens these days is a collection of tools

29:40 are put in and called AV1 and called AV2,

29:43 called VVC to allow for different use cases.

29:46 So you may be on Zoom and sharing your PowerPoint,

29:49 and then you need to show the audience a video.

29:52 That codec needs to start changing its tool set

29:55 depending on the content to compress in a different way.

29:59 And like you said, there's a bunch of incredible engineers behind each part

30:03 of that, each part of the tools that make up AV1, for example.

30:06 Uh, so we've kind of danced around it.

30:09 We talked about VLC, the logo, the hat.

30:12 Let's talk about FFmpeg.

30:14 What, what is FFmpeg exactly?

30:18 FFmpeg is basically the low-level libraries for codecs,

30:24 so compressions and decompression, muxers and demuxers, and filters.

30:29 It's— The core is this, and then you have several tools which allow

30:33 you to create a type of pipeline to process any type of video files.

30:38 And it's used as a library absolutely inside

30:41 everything from VLC to Chrome to your smart TVs,

30:45 to basically any video that you see online you usually use FFmpeg.

30:50 And FFmpeg in it has all those type of tools,

30:54 and sometimes depend on other libraries like x264, libvpx, and others, right?

31:00 So it's really now the de facto tool to process images.

31:06 From a philosophical level, I think it's incredible that your home videos,

31:10 your grandmother's home videos and trillion-dollar corporations effectively are

31:16 on a level playing field using the same technology stack.

31:21 It's— it wouldn't be a surprise, you know,

31:23 these big companies just have three thousand-line FFmpeg commands.

31:27 There are some that use the API,

31:29 but there are some that just have long command lines.

31:32 So yeah, there's a bunch of tools,

31:34 like literally command line tool, FFmpeg, of course, FFprobe.

31:38 There's libraries, libavcodec, libavformat, libavfilter.

31:43 But the FFmpeg on the command line— is, like,

31:48 legendary because you can cut— Like, there's so many parameters.

31:51 You can customize everything to hell.

31:53 It's a language.

31:54 It's an actual language.

31:55 It's an actual— yeah, you could think of it as a programming language.

31:58 Yeah, of course, I'm sure.

31:59 Because— so most of the people, they're going to take FFmpeg,

32:02 file in, file out, and specify the format, right?

32:04 But you can— We've seen thousands of characters, and we've seen also,

32:09 like, people, like, doing programming generation

32:12 of command lines to make FFmpeg.

32:15 There is a ton of people who are using AI to generate

32:18 command lines for FFmpeg because you have no idea what it is.

32:21 But you can do- specify so many filters right on the command line, right?

32:25 So FFmpeg is this collection of toolbox

32:30 for multimedia processing that everyone uses.

32:34 And everyone that is watching your videos is also using it, right?

32:37 You're on YouTube.

32:38 Well, it's FFmpeg on the client side.

32:40 Well, the your server side, on the server side.

32:43 The client side is probably Chrome.

32:44 Well, you're using FFmpeg also.

32:46 And you're using OBS to record.

32:48 Well, it's FFmpeg, right?

32:49 You're using a ton of important, like, big box, professional boxes.

32:53 Well, it's very possible that inside some part of FFmpeg is running.

32:58 I mean, there's like so many, just to give people an idea,

33:00 like I use FFmpeg a lot on, on everything.

33:03 Just trivial stuff like, Take a video, add an intro video and an outro video,

33:10 and fade one into the other like what is it called?

33:14 Dip to black, like where it dips and then shows

33:17 the next video and does the same thing with audio.

33:21 There's like a cross dissolve of the audio.

33:23 It's quiet, it quiets the audio and makes it loud again.

33:26 And then there's a bunch of stuff like showing the captions on screen card,

33:32 like baking the captions in.

33:34 You can customize the font.

33:36 You can do all kinds of layering of audio and video.

33:39 There's a million things and of course,

33:42 all of that works like magically with basically any codec.

33:49 Like anything you can shove in on the audio and the video side, it works.

33:53 But it's like if you look at, for example,

33:56 you can do things that you would do with Adobe After Effects-...

34:00 in command line on FFmpeg, right?

34:01 It's, and it's very interesting because, for example,

34:04 for imaging, there is not such tool.

34:07 There is a few tools, but not with the breadth of FFmpeg.

34:11 So ImageMagick has a similar kind of-- Yes, but you will not--...

34:14 spirit, but it--...

34:15 do some filters, complex filters.

34:16 You don't have the equivalent of Photoshop- in command line, right?

34:19 But for video, you have FFmpeg in command line.

34:23 Yeah.

34:23 It's incredible.

34:23 I mean, it's like an example of a thing when

34:26 a bunch of great people get together and they get a vision,

34:29 and they stick by that vision for many years, which is incredible.

34:34 And the vision behind, and the same for VLC and FFmpeg,

34:37 is that we make everything that is very complex

34:41 easy to use for the normal people, for everyone.

34:45 Right?

34:45 Our goal is to make something that is insanely

34:48 complex technically and make it easy to use, right?

34:51 And people, they use VLC, they drop a file.

34:54 They don't realize how complex the file is, but they play it.

34:57 Or, or people put any type of thing inside FFmpeg with complex filters,

35:01 and it just works like magically, right?

35:03 And people- And this is our mission, right?

35:06 Make very complex things.

35:08 We wouldn't be here and you wouldn't be here if this required,

35:12 you know, a traditional television studio setup.

35:16 It's tools like FFmpeg that democratize this.

35:18 The podcast and streaming revolution, the YouTube revolution- was caused.

35:23 You know, FFmpeg was a big player in that because

35:25 it democratized this technology that was once in the nineties,

35:29 for example, you needed equipment that cost

35:32 hundreds of thousands of dollars to do compression.

35:34 It was the size of a car,

35:37 and now everybody has that at almost an exact level playing field,

35:40 and that's something that's so remarkable.

35:42 It gave voice to a lot of people.

35:44 And just to clarify, we say you,

35:46 you wouldn't be here, not the human, but the podcast.

35:49 The podcast.

35:49 Oh, sorry.

35:50 You as a...

35:50 Sorry.

35:51 I would still...

35:51 VLC did not have anything to do on a biological level-...

35:55 at creating me as a human.

35:57 But, but it's like you realize also everything moved

35:59 from text to images and images to video, right?

36:02 Look at social networks.

36:03 Video is everywhere.

36:04 It's the most powerful medium there is, right?

36:08 And when you see shorts and, and in Reels and in TikTok, right?

36:12 It's amazingly powerful to give...

36:15 Video is amazing for that, right?

36:18 But the complexity is important.

36:20 This is what people don't realize.

36:22 I mean, this is really it gave power to the individual all across the world.

36:27 That's real freedom.

36:28 And I think, I can't believe it, but we still haven't mentioned the actual

36:32 obvious thing for people who are not familiar, which it's open source,

36:37 and there's a open source community of users and developers behind it.

36:43 So it's really, it's a movement.

36:45 So, like, we'll talk a bunch in a bunch

36:47 of different ways about the community behind it.

36:49 But can you speak to the open source element?

36:52 So when we say what is FFmpeg, it's an open source project.

36:57 Yeah.

36:58 So FFmpeg, VLC, x264, VideoLAN, everything we do is fully open source.

37:02 And for the people who don't understand how open source is,

37:05 my usual analogy is about a chocolate cheesecake, right?

37:08 Um, usually for you, when you want to buy your cheesecake,

37:11 you go to a bakery, they give you the cheesecake.

37:13 The other one way of having a cheesecake is have

37:16 your grandma give you a recipe of how to make that.

37:19 When we do open source, we give you the chocolate cake,

37:22 and we give you the recipe to actually remake the same cake,

37:26 but at the same time tell you how to build the oven and also

37:30 how you're allowed to modify the recipe and resell it to someone else.

37:35 And this is because software is just a very long recipe of small instruction.

37:39 Computers are not very clever.

37:41 They go very, very fast.

37:43 So a normal program has tens of billions of instructions

37:46 instead of the tens when you have your chocolate recipe.

37:49 So a lot of the software industry was about selling software,

37:53 like where you just have like the final cheesecake.

37:57 In open source, we give you everything,

38:00 and that managed to get a lot of people work together, right?

38:04 Because then you decide that you're going to make the best program,

38:07 the best recipe for video, and you create communities.

38:10 In FFmpeg, since the beginning of FFmpeg,

38:14 probably two thousand to three thousand-- In the thousands, yeah-...

38:17 people have contributed from the beginning, right?

38:19 And then it's exactly like the Linux kernel, right?

38:22 The Linux kernel has probably ten thousand people contributing everywhere,

38:25 and they get together, well, mostly online, right?

38:29 So they virtually get together to create the best tool for something.

38:33 And on FFmpeg and VLC, it's just like, well,

38:38 this codec doesn't work, so I'm going to work on the codec,

38:41 and I'm going to add the support for this file inside FFmpeg,

38:45 so it will be beneficial to everyone.

38:47 Because again, we work for the greater good.

38:49 We work for everyone, and that is what open source is.

38:54 And we should mention, depending on the licensing,

38:57 You could probably build a billion-dollar,

39:00 maybe even a trillion-dollar company around basic...

39:05 as a wrapper to...

39:07 Well, yes-...

39:08 people do.

39:09 People do, right?

39:10 There was a lot of problems with mostly,

39:13 Cloud providers who are basically running some open source tools,

39:17 In the cloud and just give you the API to access to that.

39:22 And there was a lot of um,

39:25 databases like Mongo or Elastic who changed their license

39:28 in order to avoid those type of scenarios.

39:31 This is a question we get a lot in FFmpeg is,

39:34 "Why don't you do that?" And you can't.

39:36 We have, we have thousands of contributors,

39:38 some of whom aren't even alive anymore.

39:40 It would need all of their agreement to do that, and JB will go maybe

39:43 a bit later and talk about how challenging

39:45 that process was in VLC to do the re-licensing.

39:49 The license is a social contract in terms of Rousseau de facto of the community.

39:55 The community does not agree on much beside the license.

40:00 People go around, discuss around because of the license,

40:04 and that also allow those license fork, right?

40:07 Sometimes the community splits,

40:08 but it's possible because of the license and to merge back.

40:12 And we've seen that so many times, right?

40:14 GCC and GC, And EGCS in the past.

40:17 We have seen, for example, all the web browsers, right?

40:20 They started as web, like KHTML,

40:22 which becomes WebKit and then which becomes Blink, right?

40:25 So open source license is like the core of the community

40:30 and people are coming from all around the world,

40:35 very different type of religion, Political borders.

40:39 They work in the same way on a project to solve a specific problem,

40:45 and the specific problem we're working

40:47 on is to make multimedia easy for everyone.

40:50 Uh, looking it up on Perplexity here,

40:52 looking at the different open source licenses.

40:54 Most major open source licenses fall into two buckets:

40:58 permissive, very few conditions,

41:00 and copyleft, share-alike requirements for derivatives.

41:03 Below is a brief practical summary of the main ones you'll see in the wild.

41:11 MIT license, BSD, ISC, Apache, GNU GPL, GNU AGPL.

41:20 Where's LGPL?

41:21 Yeah, LGPL.

41:22 Let's see.

41:23 There's the Mozilla Public License.

41:25 There's Eclipse Public License.

41:27 It goes on.

41:28 There's a lot of variety.

41:29 I mean, I think the really popular ones is MIT, GPL, LGPL-- Yeah.

41:35 And BSD.

41:36 BSD-...

41:37 and BSD, Apache.

41:37 Sometimes you'll see-- Apache as well-...

41:39 Apache.

41:39 Unlicense, that's an option.

41:40 Attempts to dedicate code to the public

41:41 domain with a fallback permissive license.

41:44 There are many licenses for many different things.

41:46 What people don't understand that public domain

41:48 is something that doesn't exist worldwide, right?

41:51 So it's all the open source licensing use the copyright law,

41:56 right, the international copyright law,

41:58 in order to give rights on how you use the software or how you modify.

42:03 It's de facto a copyright license contract that you

42:07 give to the end user or to the developer.

42:10 And so you have like the first one,

42:12 which are basically very permissive, MIT, BSD.

42:14 You give the code and basically you do whatever you want, right?

42:19 You take it, you want, you modify, you do what you want.

42:22 And this is popular for JavaScript and the type of BSD operating system.

42:29 So some of them, one of the parameters is whether they require attribution,

42:33 meaning if you use the code, you have to say-- Yes.

42:36 So in those types of permissive licenses, some you need to say if you use it,

42:40 which is called attribution, and some you don't.

42:42 And then there is the other part of license which are copyleft,

42:46 where you need to give back to the community

42:50 your modifications and with different strings attached.

42:53 some weak copyleft license, like the Mozilla Public License,

42:58 to some which are a bit stronger like a GPL, or even very strong like AGPL.

43:03 So all of those are different types of licensing that depends

43:08 on what your goals are and how you want to structure your community,

43:11 which is why I spoke about social contract,

43:13 because this is very important to understand.

43:16 FFmpeg and VLC are mostly GPL or LGPL.

43:21 The Linux kernel is GPL but Android is Apache.

43:26 A ton of JavaScript frameworks that are using are mostly MIT.

43:32 All the BSD kernels, OpenBSD, NetBSD are of course BSD.

43:37 And so it's philosophical change on how you want people to contribute back-...

43:44 basically.

43:45 So I think you talked about that, you've moved at one

43:50 point from GPL to LGPL on certain parts of the project.

43:54 What...

43:54 Can you describe the difference between the two,

43:56 and what does it take to move to, I guess, a more permissive...

44:00 So that direction is more permissive.

44:01 LGPL is more permissive than GPL.

44:04 Yeah.

44:04 So you have to realize that you can

44:06 always go from more permissive to less permissive, right?

44:10 Because of course, those licenses are basically statements,

44:14 and so if you restrict, you can always restrict more, right?

44:18 So in a GPL project, you can take MIT code,

44:21 but you cannot do the opposite, right?

44:23 Because they are more constrained to match.

44:25 Indeed, in fact, I changed the core of libVLC, which is the engine of VLC-...

44:34 from GPL to LGPL.

44:37 And there were two reasons to do that.

44:39 The first one is that so people can use the VLC engine,

44:44 libVLC, into third-party applications.

44:45 So a lot of applications which are playing video on your phone

44:49 or on your tablet are actually VLC engine in it-...

44:52 which is calling FFmpeg in it.

44:55 Um, so that was one of the ways to create one of the companies I created,

45:00 which is doing consulting and integration

45:02 of those types of applications where you

45:04 integrate VLC into third-party solutions like

45:07 inside game engines or stuff like that.

45:10 With GPL, you couldn't do that because

45:11 that means you needed to open source everything,

45:13 and those are for a lot of, like, commercial companies who don't want that.

45:18 So you can create a company with LGPL, you can create a company around it.

45:22 Yes.

45:22 You can do a commercial thing.

45:23 You don't have to open source it.

45:24 So that's a big, big leap.

45:25 So you could play video in your game.

45:27 Yes.

45:27 The problem is I'm a game developer, and I want to play some videos-...

45:32 and I don't want to be forced to open

45:34 source the entire game just to play those videos.

45:36 So that's where the consulting business, the libVLC LGPL-...

45:40 allows you to do that.

45:41 The LGPL, the library GPL as it used to be known, allows you to do that.

45:45 And FFmpeg is exactly the same.

45:47 It force...

45:48 LGPL forces you to give back what you change on this component, this-...

45:52 library, which is why it's library GPL.

45:55 And so you can use FFmpeg as LGPL into, like, any type of application,

46:00 even non-open source,

46:01 but you need to give back the modification you did on FFmpeg.

46:04 Same on libVLC.

46:06 Is it limiting from an open source perspective to go GPL?

46:10 Because if you— if your library, if your code is GPL, it means you're not...

46:17 You're basically discouraging companies from building a business-- Yes-...

46:22 around it, right?

46:23 Is that, is that fair to say?

46:24 It depends on the company,

46:25 but the company whose business model requires the source,

46:28 the application to be closed source, yes, it's limited.

46:32 So that's why, for example, I moved to LGPL.

46:34 The second reason is a, a bit more obscure,

46:37 is that the terms and conditions of the, App Store,

46:40 the Apple App Store for iOS makes it very complex to have GPL application on it,

46:46 while it's easier to have LGPL applications on it.

46:49 So VLC on Windows and on Mac, And on Linux is GPL.

46:53 The core is LGPL.

46:55 But on iOS the iPhone version and the Apple TV

46:59 version is a type of different license called the MPL.

47:03 And yes, I went and changed the license and it was a long story.

47:08 Yeah.

47:08 So I think basically to change the license

47:10 you have to contact all the contributors.

47:13 Yes.

47:13 It's very important to understand that open source projects are

47:16 what we call in the US copyright law joint work,

47:21 or in civil law collective works or collaborative works,

47:25 is that you work all together in terms of the same goal,

47:29 and then you create one software, which is one release.

47:32 But the copyright is kept by all the individuals.

47:36 Some open source projects don't do that.

47:37 They force copyright assignment, but this is not what we do.

47:40 We're communities.

47:41 So everyone has basically copyright on what they changed.

47:45 And this copyright stays even if at the end your contribution was

47:50 deleted because the new contribution was based on your previous one, right?

47:53 So if you want to properly re-license, you need to find all the contributors.

47:58 And at that time, I had to contact more than three hundred and fifty people.

48:02 And sometimes, well, they're just an email, right?

48:04 So it's...

48:05 you need to actually track down.

48:07 I actually, like, travel to some place to go someone that I was like, sorry,

48:11 that I'd found online to see a— to go to their job and say,

48:16 "Well, you licensed that.

48:17 Can you— do you want to change from GPL

48:19 to LGPL?" Most of the times they don't even care.

48:22 They wanted to help VLC.

48:23 But also it brought me to very complex situation.

48:26 I arrived to the work of a person who was a factory worker.

48:30 Um, and I said, "Well, I need to you to sign that," because it was

48:36 his son who died who actually wrote the code, right?

48:40 So I had to explain all those type of open source meaning, and no,

48:44 I was not a company trying to rip out

48:46 the two line or five line that that guy did-...

48:49 but was useful, and the whole community agreed

48:51 on that, and he had no idea I was a factory worker.

48:55 This com— And I was a lot younger, right?

48:57 Like it was fourteen years ago, and like, like I was almost in tears, right?

49:02 It's very difficult, right?

49:03 We are talking about lives of people and he explaining,

49:05 and we went talk about the photo of this guy, right?

49:09 So it's important to do it right and to do it correctly.

49:13 But yes, that means tracking down everything because every contribution works.

49:18 There are some project who don't respect

49:20 that, and we do re-licensing a bit, like, aggressively.

49:23 But as I said, it destroyed the whole heart of the community

49:26 because it's— we only agree on the, on the license, so that's important.

49:31 I would emphasize the community is such a wide-ranging group of people.

49:35 There's people in the Syrian war zone with electricity part-time.

49:40 There's, there's all people from all walks of life-...

49:45 rich, poor, young, old.

49:47 So it's quite remarkable to get, you know,

49:51 a group of people aligned on something.

49:53 And that's an achievement in itself.

49:56 Yeah.

49:56 It's incredible.

49:57 And a lot of them are introverts, so you coming to find them and getting them

50:01 and getting them to answer an email might be quite difficult.

50:05 Most of us are introverts, right?

50:07 You need to be more precise.

50:08 You have extremely introverts, extremely,

50:10 extremely introverts and introverts, right?

50:12 It's just like a whole spectrum of different people.

50:15 It doesn't matter.

50:16 The important is, is your code good?

50:19 Is your code great?

50:20 Is your technology great?

50:22 We care about excellent code.

50:23 We don't care who you are.

50:25 Sorry, it's just like we have no idea to check.

50:28 We cannot check, right?

50:29 Like, maybe you're a dog.

50:30 I don't care, right?

50:32 I don't care where you come from.

50:33 I need to look at your code.

50:35 And this is important because people don't understand that, and they

50:37 come to the community and send them some patches,

50:40 and they get rejected, and they don't like that because,

50:44 I mean, you're just like, "Sorry, it's not up to our standards." "Oh, yeah,

50:47 but I'm engineer at this very large company in Italy,

50:51 in Germany, in the US." We don't care.

50:54 We care about the quality of your code

50:56 because this is what defines our community,

50:58 and which means that we have a lot of people who

51:01 contribute who are some very different

51:03 backgrounds and, and very introverts, sure.

51:05 But that's okay, right?

51:08 So one of the legends of the community is of course, Linus Torvalds,

51:12 who created Linux and is a longtime maintainer of the Linux kernel.

51:18 As the legend goes, he can be pretty harsh on this meritocratic

51:22 process of reviewing the code and saying it's not good enough.

51:25 Can you just speak to the legend of Linus Torvalds?

51:29 Linus is one of a kind, right?

51:32 And I would even go and say that what he did

51:36 on Git is more interesting than what he did on the Linux kernel.

51:39 He's very harsh, but what people don't see is usually when he's

51:43 harsh to, it's people who are maintainer of part of the kernel, right?

51:48 So they know him, right?

51:49 So he's not very harsh like that to everyone.

51:52 The thing is, what he created in his room

51:55 is basically powering every server online, right?

51:58 Even at Microsoft cloud called Azure, I'm quite sure seventy,

52:02 eighty percent of the servers are running Linux.

52:04 All your Android phones are running Linux.

52:07 What he did with the power of open source, sure, is amazing.

52:12 And yes, the quality of the Linux kernel is very high,

52:17 and yes, it's difficult, but we cannot compromise on that.

52:22 We cannot compromise on quality because in the end,

52:25 and you have to understand that, is the core community of VLC is five people.

52:30 The core community of FFmpeg is ten to fifteen,

52:34 and we are the ones who are going to maintain your code, right?

52:38 Because one thousand contributors in the timeline and just ten staying,

52:41 it's one percent chance that someone comes and stays.

52:45 One percent.

52:46 So you will have change of job, change of wives,

52:49 you have children, you have accident in life.

52:51 You're going to change jobs, whatever.

52:53 You're not going to come back.

52:55 It's most likely.

52:56 So we are the one going to maintain your code.

53:00 It needs to be maintainable.

53:02 It needs to be excellent.

53:05 And yes, sometimes that means that you need

53:07 to rework your work because it was good, but it's not excellent,

53:10 and we need excellence because we are very few

53:13 to maintain something that is critical for the whole.

53:16 But we should also mention that there is some spiciness,

53:18 some harshness to the language that's sometimes used

53:21 when you're keeping this high bar of excellence.

53:25 Is there something to say to that?

53:27 It's, it's true, right?

53:28 It's also the fact that, for example, what we're doing is low level.

53:32 It's extremely technical.

53:33 You get into this community.

53:35 The tone gets very like a type of— It's a subculture, right?

53:41 So people who arrive from the external

53:43 are basically not known to the subculture.

53:45 Most of those people around FFmpeg and VLC,

53:49 we do VideoLAN DevDays, VDD every, every year.

53:52 They are so fun in real life, and they love it.

53:54 But it's true that you're online and sometimes,

53:57 like, the tone, you don't realize how it is.

54:00 But that's okay.

54:01 It's a culture.

54:02 I mean, you get this in the gaming culture.

54:04 There's pretty harsh, intense, the way people communicate,

54:06 and it's— everyone understands that the way you show

54:09 love and respect just looks different in different communities.

54:12 Sometimes people...

54:13 It depends.

54:14 If it's a book club, usually people are going to be much sweeter.

54:18 If it's an open source project that's very high stakes and used

54:23 by millions of people-- But it's very not often insults that you see,

54:27 for example, in the gaming, right?

54:29 And so Linus' tone is a bit unusual even for the open source community.

54:34 It's more like it's more harsh on the results, saying, "No, this is not good.

54:37 This is crap." Those type of things that you will see.

54:40 Try not to make it about the person, make it about the code.

54:42 Yes.

54:43 It's very, very matter of fact,

54:44 and I think you've got to look at it in terms of, you know,

54:47 the famous FFmpeg is developed almost entirely by volunteers, and that's true,

54:50 and you've got to imagine someone's done a hard day's work at their day job.

54:54 They come home.

54:56 You know, terseness might be a thing, you know, it...

54:58 And that's not something to take personally.

55:02 You're tired, you're busy, but you still care about this open source stuff.

55:05 But you may not be able to explain and handhold someone on every subtle detail.

55:10 And also you have to realize that most

55:13 people don't speak English as native language.

55:16 And this is especially for open source projects like FFmpeg and VLC,

55:21 which are mostly centered out of Europe.

55:24 Sometimes like people who are from the US

55:28 or, or just like are very not happy about the tone,

55:31 but most of the time it's also like they don't know better, right?

55:34 It's difficult.

55:34 The language is— English is a difficult language.

55:36 There is so many subtleties and tone and so on that you don't have, right?

55:40 So often it's also difficult in those type

55:43 of community about like different cultures and languages.

55:46 So as the legend goes, JB,

55:49 you repeatedly turned down millions of dollars to keep

55:54 VLC open source free for everyone without ads.

56:00 So take me through the reasoning behind that decision

56:03 of leaving millions of dollars on the table.

56:06 Yeah, that's like almost a meme, right,

56:08 on Reddit or-- There literally is a meme on Reddit.

56:11 9GAG and yeah, yeah.

56:13 See, there's-- You looking like a wizard in the, in the VLC hat on Reddit.

56:20 This is JB, the creator of VLC media player.

56:23 He refused tens of millions of dollars in order to keep VLC ads free.

56:28 Thanks, Jean-Baptiste Kempf.

56:30 You can even summon him on Reddit.

56:33 Yeah.

56:34 And usually if you see, right, it's usually like people tag me, right?

56:37 And, and then there is me, and then like I say,

56:39 "Good morning." I got twenty-four K upvotes, which is great, right?

56:42 My karma on Reddit is amazing, at least on that account.

56:46 So the question is, needs to be answered first,

56:50 what is the story about VLC, right?

56:53 Because yes, this is true,

56:55 I refuse dozens of millions of dollars, yes, several times.

56:59 Yes, I could be a multimillionaire and be somewhere on the beach.

57:03 Um, but I did not do it because I thought it

57:06 was not moral and it was not the right thing to do.

57:09 And this is very important for myself,

57:11 is to be like, I work for the greater good,

57:15 I work for people, and I don't want— It's not just by myself.

57:18 But the reason is also because I did not feel that I'm

57:22 completely legitimate to do that, and let me explain you why.

57:26 VLC story is a very weird story.

57:29 In France, we have university and we have a type

57:33 of top colleges and those top of excellency schools are engineering schools,

57:38 business schools, and basically lawyers and medical, right?

57:43 But they're outside of university,

57:44 and in order to enter those, you spend two years working like crazy math,

57:49 physics to enter those best engineering schools.

57:53 One of the schools is called the École Centrale Paris.

57:55 It has changed name since, but it was called the École Centrale Paris.

57:58 And because it was Centrale, they had to move it because it was too small

58:03 after the World War II and, and they moved it,

58:05 they wanted to move it to the center

58:07 of France in a place called Clermont-Ferrand.

58:09 And the alumni decided that this was not okay, right?

58:12 It is a, the school that Eiffel, right,

58:14 the, the one who did the Eiffel Tower, attended to, right?

58:17 So they said, "No, no, we are amazing, great school.

58:20 We cannot do that." And so they bought

58:22 a piece of land south of Paris very near Paris.

58:24 And it was a campus managed by a nonprofit of the alumni, okay?

58:32 Because of that, everything on the campus was managed by students.

58:35 The university did nothing, right?

58:39 So radio, TV, supermarket, library defining who was going into which rooms.

58:46 Everything was managed by the students.

58:48 That's amazing.

58:49 That's an amazing experiment, that it all, all didn't go to hell quickly.

58:54 It somehow flourished.

58:56 It worked great, and I learned so much

58:58 in my life doing those side activities, right?

59:01 Because you're twenty-two and you need to run your campus,

59:03 else you don't have electricity, right?

59:05 So you care about that, right?

59:07 But anyway, in the '80s they did a full experiment

59:11 of deploying a network mostly sponsored by IBM and 3Com,

59:14 which was a token ring network.

59:17 So token ring is something that probably almost no one knows about anymore.

59:22 It's a networking technology where you don't have routers, right?

59:26 Everyone is linked.

59:27 It's type, like really a ring, and when you want to send a message,

59:31 you talk to your neighbor who's going to put the message to the next one,

59:34 who's going to put the things to the next one, in terms of ring.

59:38 The issue with token ring is, of course,

59:40 is that it's very slow because every computer on the network

59:44 needs to open the message, see if it's okay.

59:46 Is it for me?

59:47 No, it's not, and then send it back,

59:49 like a token which is traveling around the ring.

59:53 In the '80s, you're doing some Telnet and sending mails as university.

59:57 That's okay, right?

59:59 But starts the '90s, and the '90s and start video games,

1:00:03 and when you have high latency in video games, basically you die, right?

1:00:07 So in nineteen ninety-four, nineteen ninety-five,

1:00:09 around Doom and Duke Nukem coming around, they want a faster network.

1:00:13 So the students go and see the university and say, "You know what?

1:00:16 We want a faster network.

1:00:18 We need to work," which, and also play video games.

1:00:21 And the university tells them that basically, "Oh, I'm sorry,

1:00:25 we cannot help you because you understand the campus is not ours.

1:00:29 You manage it, so do something.

1:00:32 And you should see some basically partners

1:00:34 of this university and basically go away." And they go,

1:00:38 and they actually go and see the CIO of Bouygues,

1:00:43 which is a large French company and who's doing some TVs in France.

1:00:47 And he says, "Well, you know what?

1:00:49 The future of video is satellite." Well, today we know it's not,

1:00:53 but at least it was a good idea.

1:00:55 In nineteen ninety-five, the first satellite dish,

1:00:57 and he says that instead of having like one satellite

1:01:01 dish and a big decoder for each of the students,

1:01:04 which are one thousand and five hundred, what about you build,

1:01:08 like you put an enormous dish and only one decoder,

1:01:12 and you send the video directly on the network.

1:01:15 And that required a very fast network.

1:01:17 Today, it's obvious, but at the time was, like, the first to do video streaming.

1:01:22 So they built this project, which was called Network 2000.

1:01:26 Of course, right, we are in the '90s, right?

1:01:28 Everything is-...

1:01:29 futuristic is called 2000, like—- Yeah, 2000, yeah.

1:01:32 And so they do the Network 2000 project.

1:01:35 It's completely hacked.

1:01:36 It crashes after 45 seconds.

1:01:37 That's okay.

1:01:38 The demo is 40 seconds.

1:01:40 It leaks memory.

1:01:41 That's okay.

1:01:41 They put 64 megabytes of RAM instead of the 8 or 16 you have,

1:01:45 and the demo should have stopped there.

1:01:47 And that was the Network 2000 project by the students.

1:01:49 What was the format of the video that they had to work with?

1:01:52 MPEG-2 because satellite is MPEG-2 TS for transport,

1:01:55 MPEG-2 video, and MPEG-2 audio at that time.

1:01:59 And the project should have stopped there.

1:02:01 Everyone was happy.

1:02:02 They had, like, amazing ATM network at 155 megabits per second.

1:02:07 They had probably one of the best network in Europe at that time,

1:02:10 and they stopped the project.

1:02:12 Six months or a year later, two students arrive and say, "Well, you know what?

1:02:16 Maybe other people care about video streamed on a local

1:02:19 network," and they create the VideoLAN project, VideoLAN.

1:02:23 And one of them is called Christophe Massiot,

1:02:25 that is a good friend of both Kieran and me, and they start the project.

1:02:30 It's not even open source yet, and they spend around three years to get

1:02:35 the school to agree to make it open source.

1:02:38 Because the university wanted to get some— because

1:02:40 of the IP and copyright of the students,

1:02:43 wanted to basically monetize these MPEG-2 decoders.

1:02:47 Just to be clear, so what was the main application,

1:02:49 streaming on a local network?

1:02:51 It was streaming on a local network.

1:02:52 By the way, that's just, like, to state the obvious.

1:02:54 This is before YouTube.

1:02:55 This is before-- Ten years before YouTube.

1:02:57 You have a Pentium 60 or 75, right?

1:03:00 You, the main machine was 486DX at 33 megahertz, right?

1:03:04 Bear in mind, television was the main form of video at the time.

1:03:07 You could get new channels.

1:03:09 In the '90s, having even one new channel when you grew up with four channels,

1:03:12 having a fifth or a sixth was a big deal,

1:03:15 and so having this satellite service with, you know,

1:03:19 dozens, even hundreds of channels was so groundbreaking.

1:03:22 Especially because this is university where you

1:03:23 had a ton of different nationalities, right?

1:03:26 So there was a ton of people who wanted...

1:03:28 So the— in the end, they had, like,

1:03:30 several dishes on different types of satellites, right?

1:03:31 Because, for example,

1:03:32 a lot of people were coming from the Maghreb or the Middle East and they,

1:03:36 so they went to different types of satellites.

1:03:40 Anyway, the solution worked great, and they started the VideoLAN project.

1:03:45 The VideoLAN project has several and some are completely crazy solutions,

1:03:50 like one how to create multicast on a unicast network,

1:03:53 but let's not come to that.

1:03:56 It's too, too complex.

1:03:58 But VideoLAN client part is what became VLC.

1:04:03 Actually, they basically strong-armed the university to force it

1:04:06 to open source because university did not understand that.

1:04:08 And in 2001, it's still early.

1:04:11 But basically, yes, the university agreed early 2001 to make it open source.

1:04:16 I joined the project in 2003 because that's when I joined the university.

1:04:20 So the first thing is I'm not the one

1:04:23 who created VLC because actually no one did, right?

1:04:27 Just kind of naturally emerged from the VideoLAN project.

1:04:29 And we should mention that, like, again, you said it just, but to make it clear,

1:04:34 VideoLAN, As what it became was at the time

1:04:37 was a set of technologies around video, and the VLC, what you called the client,

1:04:42 that's the thing that most normies, uh-- That is correct, and-...

1:04:48 think of, like, as the thing, which is, like,

1:04:50 the thing that pops up when you click on a video and you play it.

1:04:53 So I arrive in 2003,

1:04:55 and then I will create the open source nonprofit organization called VideoLAN,

1:05:01 and I took everything out of the university

1:05:04 to create a nonprofit project and something sustainable.

1:05:06 It's, yes, it's true that I spent more time than anyone on VLC and VideoLAN.

1:05:12 That is sure.

1:05:13 but it's a continuity of a previous project, VideoLAN, the student project,

1:05:17 which is a continuity of the Network 2000 project,

1:05:20 which is a continuity of that and that.

1:05:23 I'm sure there's moments along the way there you were thinking of, like,

1:05:26 what is the future of this from an open source perspective?

1:05:29 'Cause as, as the internet is blowing up, and there is companies...

1:05:34 I mean, for people who don't remember,

1:05:36 like, there's companies making huge amounts of money.

1:05:39 And I can tell you that in 2005,

1:05:42 the project should have died, And I made it to continue the project.

1:05:48 At some point, we were only two active developers.

1:05:52 and I thought it was great technology and was useful,

1:05:56 and it will be useful and I made that my life and my, my time.

1:06:01 And I made that grow from a few hundreds of thousands of users,

1:06:06 millions of users to what we have now, which is probably billions of version

1:06:10 of VLC around the world and used everywhere.

1:06:14 So that's a bit the story of VLC.

1:06:17 There is ton of very funny stories around that.

1:06:20 Many people from around the world working on it,

1:06:24 like you said, in Syria or middle of nowhere in India.

1:06:27 But along the way, I got several

1:06:29 offers which were either to bundle toolbars, right?

1:06:33 You remember those horrible toolbars-...

1:06:36 which were basically spyware,

1:06:37 or changing your web browser or your search engine or even,

1:06:42 like, advertisement inside VLC.

1:06:44 And I didn't like that, right?

1:06:47 I am— and people don't understand that.

1:06:49 It's not— I'm not against money, right?

1:06:51 I'm very happy to make money.

1:06:53 I created several startups and one I hope that is going to work very well.

1:06:58 It's the fact that I believe that you need to win money ethically.

1:07:03 There is a right way of doing that, and doing

1:07:06 sneaky advertisement or stealing data is not the correct way, right?

1:07:10 For example, if Netflix arrived at some point and say, "Well,

1:07:13 we want to put Netflix inside VLC,"

1:07:14 probably the story would have been different, right?

1:07:16 But they didn't.

1:07:17 The only people who came to us were shady ads company.

1:07:22 And if I do that, right, I would have a ton of money, right?

1:07:25 And then three years later, project is gone, right?

1:07:28 Someone forks it and something else happens.

1:07:32 So it's not even necessarily ads or any of that, it's the shadiness of the-...

1:07:36 dishonesty of the— So you had a good radar,

1:07:38 you had a good threshold of like, "No,

1:07:41 this compromises the spirit of what this is

1:07:44 supposed to represent."- But also it's for me, right?

1:07:48 I'm like very selfishly,

1:07:49 I need to go to bed at night and be happy about what I've done, right?

1:07:53 Maybe it's my upbringing, maybe it's my parents' fault or whatever, right?

1:07:57 But I believe there is right and wrong, right?

1:08:01 And this was the right decision at the time.

1:08:05 It still is.

1:08:05 I want to be proud of what I've been doing.

1:08:08 And like, if I had sold out,

1:08:11 I would have betrayed so many other people who work here.

1:08:14 Yeah, well, I should say me and most

1:08:16 of the internet thank you for that decision.

1:08:20 It's inspiring for others,

1:08:23 I think that are pushing the open source movement forward,

1:08:28 that it's okay to do these kinds of huge sacrifices if you believe it's right.

1:08:33 And I think in that case it was right and it

1:08:35 was the reason that VLC became as successful as it was,

1:08:39 'cause it's an embodiment, it's a symbol of like,

1:08:43 you know, freedom and what the open source community can create.

1:08:46 Yeah, and be a service for so many people around the world,

1:08:49 and this is important.

1:08:50 We should emphasize in the 2000s it was really normal

1:08:52 to download a program and it secretly installs some spyware.

1:08:55 It was buried in very faint text or in the license

1:09:00 text box that nobody reads that at the bottom-...

1:09:03 "Oh, I will be installing this toolbar-...

1:09:05 and changing all these things," and it was very common to have to, you know,

1:09:08 you install a program to do something at the time of any sort.

1:09:11 To put yourself in the mind of a developer at that time, I think it's very easy,

1:09:16 to everybody listening to this, it's very easy at that time

1:09:19 to convince yourself to take a few thousand dollars-...

1:09:24 a few thousand dollars to do it.

1:09:27 To say no to much more money—...

1:09:30 takes guts and takes vision.

1:09:34 The last offer I had was obscene, and they say, "Yeah,

1:09:39 but imagine with all that money you could build something new,

1:09:42 open source," right?

1:09:43 It was like the mind trick was, it was difficult.

1:09:47 But for me it was just like, "No,

1:09:49 this doesn't work like that or this is not the right thing,

1:09:51 so I don't do it." and again, right,

1:09:55 it's not that I don't like money or whatever.

1:09:57 It's just like it wasn't right.

1:10:01 Well, once again, thank you from me and from the rest of the internet.

1:10:04 Let me talk a little bit more about the open source movement,

1:10:08 about the fact that, as you say over and over and over and over,

1:10:12 FFmpeg, is and many open source projects are built by volunteers.

1:10:16 So there's a bit of drama recently, uh, Kieran, on the interwebs, on Twitter.

1:10:25 You have a spicy style on Twitter that I think articulates

1:10:29 and celebrates all the incredible developers and development and the code,

1:10:35 especially assembly that's involved in building some

1:10:38 of these codecs and building some of this incredible technology.

1:10:40 But that brings us to the, a bit of a debacle that happened.

1:10:45 Tell me the full saga of what happened with the Google security engineers.

1:10:50 Just to be clear, Google are one

1:10:52 of the biggest supporters of open source out there.

1:10:54 They have been for a long time.

1:10:55 It's just I think some things kind of went a bit overboard this time.

1:11:00 So FFmpeg itself, and this is not like a secret,

1:11:03 it's on the homepage, you know, the, it processes untrusted data.

1:11:07 There can be security issues when you parse untrusted data.

1:11:10 That's very normal.

1:11:11 But recently what changed was Google started using AI

1:11:14 to create security reports on an open source project, FFmpeg.

1:11:19 Volunteers had to deal with that.

1:11:20 They did, they provided very limited funding,

1:11:22 and they even went to the media first announcing how

1:11:25 good their AI was before the issues could be fixed.

1:11:29 And this is in the public forum.

1:11:31 Yeah, this is all public.

1:11:32 So report, reporting an issue, using AI to find an issue in the code,

1:11:36 which is a security vulnerability,

1:11:37 and then reporting that publicly before you're able to fix it.

1:11:40 Yeah.

1:11:41 It's announcing how good their AI is,

1:11:42 that they provided a standard 90-day industry deadline

1:11:47 without without really understanding the nature of volunteer-driven development.

1:11:52 In addition, this vulnerability was on an obscure 1990s game codec.

1:12:00 the way— And let's look at it from their standpoint to begin with.

1:12:04 Let's you know—- Yeah.

1:12:05 Can you steer me in their case?

1:12:06 Yeah, sure.

1:12:07 They have substantial resources working on the security

1:12:09 of open source projects that, you know, are ubiquitous, and they've used,

1:12:13 you know, a lot of compute to do that and very

1:12:17 expensive and very capable security researchers, to do that.

1:12:20 And that's their viewpoint is they are contributing by doing that.

1:12:25 But I think that's where opinions differ.

1:12:30 it opened up a lot of interesting fissures I would say.

1:12:36 it does seem that there's a portion of the security community that look

1:12:41 at themselves a bit like building architects that never have to go to site.

1:12:44 You know, going to site is something that is a little bit beneath them,

1:12:48 the actual day-to-day construction.

1:12:49 They're there to do their security things and it's someone else's problem.

1:12:52 The security industry also kind of has a very aggressive tone towards things.

1:12:59 The, the language they use is extremely aggressive.

1:13:01 They use very strong language like,

1:13:03 "You will get popped." So and to, Joe Public,

1:13:06 get popped, you know, means something quite bad.

1:13:09 For them it means to get hacked.

1:13:11 The way I would look at it personally is

1:13:13 a little bit like the padlock on your home.

1:13:15 The padlock on your home or, you know, the lock on your home is there to protect

1:13:23 against the capabilities of what it's there to protect.

1:13:28 It's not there to protect nuclear secrets.

1:13:30 It's not there to protect Fort Knox.

1:13:32 And it could be looked at that they're using AI at a level

1:13:36 of scale to go and pick those locks and then say, "Hey, your lock's not secure.

1:13:41 You need to deal with this." Whereas actually they're

1:13:44 the ones with resources to be able to fix this.

1:13:48 But that seems to not be something either they'll contribute

1:13:51 to in terms of patches or in terms of financially.

1:13:54 And the scale of AI is kind of the issue.

1:13:56 The bug reports are very wordy.

1:13:59 They're very, very— It's almost a denial of service

1:14:03 by AI-generated bug reports on very niche codecs.

1:14:08 and the other issue the security community

1:14:10 has is everything is marked high priority.

1:14:12 You're going to, you know, "This is the most important thing in the world,

1:14:14 and you need to deal with this.

1:14:15 High, high, high, vulnerable, scary, scary,

1:14:17 scary," on a game codec used on one disk in 1993.

1:14:24 And that's where the dichotomy lies.

1:14:26 Going around telling everyone that their padlock's not safe,

1:14:30 well, that's a hobby project of somebody.

1:14:32 The safety of that codec is consummate to what that person thinks.

1:14:37 It's their hobby.

1:14:37 It's good that they're security analyzing it,

1:14:39 but it doesn't need a big scary warning,

1:14:42 "This is a critical vulnerability." We also may

1:14:46 recently also see that there was another quote-unquote vulnerability.

1:14:49 It wasn't at Google in this case,

1:14:51 but a filter could overflow and have an integer overflow,

1:14:55 and one of your pixels could be the wrong color.

1:14:59 And this was marked high, 7.5 severity in red.

1:15:05 And at some point, the security industry needs to realize you

1:15:08 can't keep crying wolf like this because this just leads to people,

1:15:11 you know, the equivalent thereof of putting password stickers on their PC.

1:15:15 You know, you can't just keep crying wolf every day.

1:15:17 And I appreciate, you know,

1:15:19 that's their modus operandi is to create as much scare and fear.

1:15:24 But from the Google standpoint, at the end of the day,

1:15:29 they need to contribute either financially or with patches.

1:15:32 Google uses FFmpeg at a scale probably you or I couldn't even contemplate,

1:15:38 millions of CPU cores.

1:15:40 And yes, they contribute in areas mostly

1:15:43 regarding their own products, so VP9, AV1.

1:15:47 But in a wider sense, there's a disproportionate level of contribution.

1:15:52 Yes, they fund students.

1:15:53 Yes, they fund Summer of Code.

1:15:55 And I think so Alex Strange is a former

1:15:59 FFmpeg developer I think posting in a personal capacity.

1:16:03 So he posted about security engineers on Hacker News.

1:16:07 His post reads, "The problem with security reports

1:16:11 in general is security people are rampant self-promoters,

1:16:14 in parentheses, Linus once called them something worse.

1:16:19 Imagine you're a humble volunteer open source developer.

1:16:23 If a security researcher finds a bug in your code,

1:16:28 they're going to make up a cute name for it, start a website with a logo.

1:16:33 Google is going to give them a million-dollar bounty.

1:16:36 They're going to go to DEF CON and get a prize,

1:16:39 and I assume go to some kind of secret security

1:16:43 people orgy where everyone is dressed like they're in The Matrix.

1:16:47 Nobody is going to do any of this for you when you fix it."

1:16:53 basically commenting on the sort of the incentives

1:16:58 for the different people involved and misaligned.

1:17:02 The problem here is the disproportion of means

1:17:06 on discovery compared to patching it, right?

1:17:09 And this is the biggest issue, right?

1:17:12 And after that debacle, Google did some changes.

1:17:15 They are now starting to send patches,

1:17:16 which is-- And they also now have reward tools for fixing issues.

1:17:20 So it has changed a bit because of that debacle.

1:17:23 So it's good, right?

1:17:25 But we've seen, and we talk about Google,

1:17:27 but we have seen like some other large companies saying,

1:17:30 "Oh, you need to fix this bug because it's

1:17:32 critical in our product."- Can you explain the XZ fiasco?

1:17:36 The FFmpeg tweet reads, "The XZ fiasco has shown how a dependence

1:17:43 on unpaid volunteers can cause major problems.

1:17:47 Trillion-dollar corporations expect free and urgent support from volunteers.

1:17:52 Microsoft, Microsoft Teams posted on a bug tracker

1:17:56 full of volunteers that their issue is high priority.

1:18:02 After politely requesting a support

1:18:03 contract from Microsoft for long-term maintenance,

1:18:06 they offered a one-time payment of a few thousand dollars instead.

1:18:09 This is unacceptable.

1:18:10 We didn't make it up.

1:18:12 This is what Microsoft Teams actually did." And then you give

1:18:17 the image and the details and all that kind of stuff,

1:18:20 showing that these trillion-dollar companies are not giving much money,

1:18:23 not giving much support.

1:18:24 They think an open source project is a traditional vendor that they have an SLA.

1:18:28 They think a public bug tracker is actually, you know,

1:18:32 a third-party vendor's Jira where you can do all of these things.

1:18:35 It's not.

1:18:36 It is there to report bugs.

1:18:38 I think the thing that made

1:18:40 this particularly heinous was the name-dropping of Microsoft,

1:18:43 the name-dropping that this is a visible product.

1:18:46 If this was just a general bug report,

1:18:48 I think that would have made it a lot better.

1:18:51 Yeah, so they literally said, like, "This is a big deal because a lot of people

1:18:56 are using it in Microsoft." I wonder what happens psychologically.

1:18:59 So I think what happens in these companies,

1:19:02 maybe you can correct me, is they— You're right.

1:19:06 They just think of FFmpeg as like a vendor

1:19:10 that Microsoft surely is paying a huge amount of money to.

1:19:14 They kind of assume that in their interaction,

1:19:17 and nobody anywhere on the stack is going like, "Wait a minute.

1:19:22 Shouldn't we be giving like millions of dollars to FFmpeg?"- And this is

1:19:26 a very big problem in large— Like we're talking about some companies,

1:19:29 but it's the same everywhere, right?

1:19:31 A lot of those companies.

1:19:33 Like the, when we talk to that person, right,

1:19:37 he was just like a manager on one project in Microsoft Teams, right?

1:19:40 He had never really discussed with open source community.

1:19:44 He had no idea, right?

1:19:46 It was like—but the problem is that usually there is what we call OSPOs, right?

1:19:52 Open source program offices in those type of companies,

1:19:54 and they are the ones who are supposed to discuss with open source vendors.

1:19:59 Um, or open source communities.

1:20:01 But like they often don't explain that correctly internally, right?

1:20:05 And here it's just like we are not your supplier.

1:20:09 If you want me to be your supplier, I'm very happy, right?

1:20:12 I will send you a contract and SLAs.

1:20:14 Like I created five companies who are doing

1:20:17 that around open source projects, so that's okay.

1:20:20 We should say that some of the spicy tweets that Kieran,

1:20:23 you're behind, and some of the debacle produced results.

1:20:28 Yes.

1:20:28 Positive results.

1:20:30 Donations have increased substantially.

1:20:31 They're still not enough to cover even a single full-time developer,

1:20:35 but on both a, you know, awareness level and a technical level,

1:20:41 there's substantially more technical awareness and sort of awareness

1:20:44 of the importance of FFmpeg as a result, as a result of X and what's happened.

1:20:49 I can say, you know, it solved its purpose.

1:20:52 People realize the level of importance FFmpeg has.

1:20:56 And on VideoLAN it's the same, right?

1:20:57 Like for example, a, a very simple example.

1:21:01 For more than a year,

1:21:03 we couldn't update VLC on Android because of a bug on the Play Store,

1:21:08 on Android Play Store, right?

1:21:09 The only way we got someone to answer was to put a very spicy,

1:21:15 as you say tweet saying that we are

1:21:19 going to stop distributing VLC for Android, right?

1:21:22 And we have around 100 million people using that.

1:21:26 And now then someone from Android actually came and discussed to us, right?

1:21:31 We had the same issue with Microsoft or, or like saying

1:21:35 that we were going to stop distributing VLC on the Windows Store.

1:21:39 And unfortunately, we are so small that the only

1:21:44 very strong power we have to solve

1:21:46 those issues is blaming on social network because

1:21:50 it snowballs and now they listen to us.

1:21:53 But so as large companies often have difficulty talking to us.

1:21:58 Like for example, VLC, right,

1:22:00 is probably one of the top 10 software used on Windows.

1:22:04 I am not part of Microsoft ISV programs, right?

1:22:09 I don't have a point of contact at Microsoft, right?

1:22:12 While I'm sure any other software, Adobe, Spotify, has a point of contact.

1:22:17 I don't have that, right?

1:22:18 So raising awareness works.

1:22:22 It's sometimes very spicy, lot of drama.

1:22:25 Well, X and Twitter are okay for that, but it's efficient.

1:22:31 Uh, so everybody listening to this should go follow FFmpeg on Twitter,

1:22:37 on X, follow VideoLAN on Twitter, on X.

1:22:41 Go donate.

1:22:42 Donate...

1:22:43 to FFmpeg.

1:22:45 And thank you, Lex.

1:22:46 Over the years, several years you've been a supporter of, you know,

1:22:49 FFmpeg and VideoLAN on X.

1:22:51 You know, giving us shout-outs, appreciating, you know, what we do.

1:22:56 FFmpeg for life.

1:22:57 And for example, like Tim Sweeney, Carmack, and a few others,

1:23:01 like very high-level people have raised also the awareness on our X accounts,

1:23:06 and that helped a lot also.

1:23:09 Karpathy as well.

1:23:10 Karpathy, yes.

1:23:10 Karpathy as well, yeah.

1:23:12 Yeah.

1:23:12 I mean, also, you know, outside of the fact that so many people use it,

1:23:15 it's so impactful on the world,

1:23:17 it's also a great representation of a great open source project.

1:23:20 Like the value of assembly and C and making sure

1:23:25 that like you take programming seriously for real world systems.

1:23:30 It's not just that.

1:23:31 We'll talk about assembly later I'm sure,

1:23:32 'cause that's its whole topic in itself,

1:23:34 but it's also celebrating people like Andreas Rheinhardt who do maintenance.

1:23:39 It is, I believe unpaid, as I believe as a volunteer.

1:23:43 He's doing massive refactorings.

1:23:44 Uh, Andreas Rheinhardt and Anton Khirnov rewriting ffmpeg.c with threading.

1:23:49 Celebrating those guys,

1:23:51 celebrating the untold labor that's gone into this that actually

1:23:55 doesn't change anything from the user standpoint.

1:23:57 The files are exactly the same, but wow,

1:23:59 the, the, the airplane has been rebuilt whilst it's in the air.

1:24:03 Christian Garcia said,

1:24:04 "As a teenager running this account," referring to the FFmpeg...

1:24:08 account, and you responded,

1:24:09 "Teenagers have written more assembly in FFmpeg than Google engineers." But also

1:24:15 just pointing out that there's a lot

1:24:17 of incredible contributors who are teenagers.

1:24:20 Like JB said, we don't care who you are, where you're from, what you do.

1:24:24 Teenagers have written thousands of lines of assembly, Over the years.

1:24:29 Give a shout-out back in the days to Daniel Kang.

1:24:33 So also highlighting the work of people like Ruikai Peng.

1:24:36 This is a 16-year-old, some of his first contributions to FFmpeg,

1:24:40 actually doing and putting some of these quote unquote security researchers

1:24:43 to shame by by actually finding issues and fixing them and being 16.

1:24:48 There's no barriers.

1:24:49 There's no barriers to you have to study

1:24:52 on, at college under this person and understand these.

1:24:54 It's you can learn C, and let's be honest, it's from, it's from the K&R book.

1:24:59 Learn C.

1:25:00 You can learn assembly.

1:25:01 We'll talk about that maybe a bit later.

1:25:03 You can contribute to world-class technologies.

1:25:07 In VLC one of the oldest contributors called Felix,

1:25:10 he's the one doing everything on Mac and iOS.

1:25:13 He's starting working on VLC.

1:25:15 He was 16.

1:25:16 We had a guy called Edward Wong, who used to be a Google Summer of Code

1:25:21 student who stayed for three years around VideoLAN.

1:25:24 He was 14, right?

1:25:26 And, and part of Google Summer of Code and Google Code-in,

1:25:30 which were programs where basically we have students or high school,

1:25:34 We wrote a ton of assembly for x264 and for VLC and for FFmpeg, right?

1:25:39 So everyone can contribute.

1:25:41 And he also did a good job because he

1:25:44 didn't play the alarmist CVE heist, create a CVE,

1:25:47 which is like, a public exposure of security

1:25:50 and do these big scary red 7.5 high priority.

1:25:54 He just fixed an issue in Git after three days and just fixed it.

1:25:58 He didn't need to go and play a big security drama about it.

1:26:01 And I think I posted, you know, the kids are all right.

1:26:05 Whereas- there's, you know, there is a por- I'm not saying all security people

1:26:09 do this, but there is a portion of the security community,

1:26:11 as Alex said, that likes to hype themselves up by creating drama.

1:26:15 They would have happily raised, "This is a high priority CVE 8.0" or whatever

1:26:21 on a issue that actually was in Git.

1:26:23 It wasn't even in a release,

1:26:24 it was in development, and three days later was fixed.

1:26:28 Well, I just want to put a little bit of love out there,

1:26:32 even to the bigger much love and respect to Google engineers.

1:26:37 Like you said, they're Some of the, the best software engineers in the world,

1:26:41 and they do contribute a lot-...

1:26:43 even on the security front.

1:26:44 And also, you know, I'm a big fan of Theo.

1:26:47 Much love to Theo.

1:26:48 He was part of this, Debacle and drama a little bit.

1:26:51 I think when you just zoom out on the grand arc of human history,

1:26:57 the drama contributed positively to everybody involved.

1:27:00 Donations went up.

1:27:02 It brought more attention to the topic,

1:27:04 allowed everybody to bicker in a way that ultimately

1:27:08 got them to figure out what FFmpeg is all about.

1:27:12 So the way we looked at this is like it's

1:27:13 a rap battle at the end of the day, you know?

1:27:16 No, but it is.

1:27:17 We say stuff, we say stuff-...

1:27:19 but we can, we can leave it on.

1:27:21 X is a perfect place for, you know, international rap battle.

1:27:23 You say stuff.

1:27:24 I say stuff about your mama, but it doesn't mean,

1:27:26 you know, I have an actual personal issue with her.

1:27:29 Uh, and that's what it looks like.

1:27:31 The Theo situation, you know, JB can maybe expand,

1:27:32 went a little bit too far and there was a little...

1:27:35 But, you know, it's just a bit of fun.

1:27:37 It's just a bit of rap battle.

1:27:39 It's a bit, it's WWE.

1:27:40 You know, everyone's having a bit of fun on X.

1:27:42 It doesn't need to be taken seriously.

1:27:44 You know, the teenagers thing, you know, that...

1:27:47 So that guy was a Google employee saying, "Hey, you know,

1:27:49 there are other ways to run an open source business." You know,

1:27:51 go and there's like, oh, man, just have a bit of fun, you know?

1:27:54 That's what the point of this account is.

1:27:56 And, and furthermore,

1:27:57 if you can teach people about the ways of open source projects,

1:28:00 assembly, et cetera, by doing that, I think there's a lot to be offered here.

1:28:04 It's not dunking on people for dunking's sake.

1:28:06 It's showing actually the story that I think X

1:28:09 learnt is these are not big corporate open source projects.

1:28:11 This is not Kubernetes where there's,

1:28:14 you know, hundreds, maybe thousands of people-...

1:28:16 paid to develop this stuff.

1:28:17 These are just people in their basements in their spare time,

1:28:20 and if you can address that topic in a fun and entertaining way-...

1:28:24 I think that's the good thing and that's,

1:28:26 that's the value of X and then the reach we have.

1:28:29 And to be honest, right, like even at Google,

1:28:32 Google is one entity, but so many different people, right?

1:28:36 And there is a ton of Google engineers we work with all the time,

1:28:43 and even like Google from YouTube to Chrome

1:28:45 to Chrome Media to the rest of Google,

1:28:47 those are very different types of entities.

1:28:49 But what we do is efficient.

1:28:52 And, for example for, for Theo, right?

1:28:55 It went a bit too far.

1:28:56 I had him...

1:28:57 Like I calmed everyone down.

1:28:58 I had him on the phone.

1:28:59 We said, "Okay, like this goes too far," and so on.

1:29:02 But in the end yeah, it's a rap battle, but it's positive for the project.

1:29:06 It, like the awareness we have on open source and, and I mean true open source

1:29:12 from communities right now is increased dramatically

1:29:15 in the last two years, and this is useful.

1:29:19 Uh, what do you think motivates all

1:29:21 the incredible contributors that we've been talking about?

1:29:23 Like, what's the engine?

1:29:24 It's so interesting to see.

1:29:26 So-- Like you said, they're sitting in the basement.

1:29:28 What's the driver?

1:29:28 What's the engine there?

1:29:30 There are many drivers,

1:29:31 but weirdly the main one is that what we do in multimedia plays videos,

1:29:36 and video is cool, right?

1:29:38 And, and for example, we have so many people in the community

1:29:42 who arrive because they loved watching anime, right?

1:29:45 And this is like the advice when people ask me,

1:29:50 "What should I work on in open source?

1:29:51 How do I start?" And my answer is always the same: work on something you love.

1:29:56 I am working on VLC because I love movies, right?

1:29:59 And I love watching the same movies over and over,

1:30:02 even if my wife hates me when I do that, right?

1:30:05 But because it's interesting, right?

1:30:06 Because it's a topic that you like, right?

1:30:09 The first, that's the first thing where

1:30:11 people come to usually to VLC and FFmpeg.

1:30:13 The second thing is that technically we,

1:30:16 because we search for excellence, this is the best school ever, right?

1:30:21 This is the best school ever of programming.

1:30:24 If you're good in C, in FFmpeg, if you know how to write assembly,

1:30:29 I assure you you're going to be one of the best programmers ever,

1:30:33 even if you're working on writing TypeScript,

1:30:35 because this is the most amazing thing to do.

1:30:39 And you will, like, have to get reviews

1:30:41 by some of the most seasoned programmers ever who

1:30:44 are going to look at every part of your code and tell you why it's not great.

1:30:49 It's like we are the best teachers that you've ever had in programming, right?

1:30:53 Andrew Kelley started Zig.

1:30:54 He was an FFmpeg developer and started Zig after his FFmpeg school.

1:30:58 I mean, it's the place to learn so

1:31:01 many aspects of programming in the real world,

1:31:04 in a thing used by billions of people.

1:31:06 You have nowhere to hide.

1:31:07 You have to be open and honest about

1:31:09 your flaws and how you can learn and be better.

1:31:13 And what is also interesting in multimedia is

1:31:15 that you have 16 milliseconds to display a frame.

1:31:18 It's not like a game engine where you can basically slow down and wait a frame.

1:31:23 Like, so it's, you need to be good, right?

1:31:25 There is no choice, else you don't have your video.

1:31:27 And because of how codecs, if you miss a frame,

1:31:30 you're going to destroy the look of the video, right?

1:31:33 So you need to be good.

1:31:35 You need to be perfect to have the right thing.

1:31:38 But also is that it's not just

1:31:41 pure programming in the mathematical sense, right?

1:31:43 A lot of people don't understand,

1:31:45 but in order to program correctly on the open source multimedia community,

1:31:51 you need to understand how computers work.

1:31:54 And when you write assembly, you need to understand about CPU pipelining, right?

1:31:58 You need to understand how SIMD works, how the ALU works, right?

1:32:02 You need to understand what, how IO works, right?

1:32:05 And this is what I think that is missing

1:32:08 to a lot of engineers and software engineers today,

1:32:11 is understanding what we call computer architecture.

1:32:14 And, like, seriously, like some of the debates is like,

1:32:17 should we use this assembly call or this one?

1:32:19 And people say, "Well, no,

1:32:20 it's going to be like three cycles on this type of CPU

1:32:24 and this one," and has massive impact on the output, right?

1:32:27 We should expand.

1:32:28 FFmpeg is probably one of the biggest CPU users in the world.

1:32:31 There's it's probably running—...

1:32:33 as we speak easily 100 mil- order of magnitude 100 million,

1:32:37 maybe even a billion CPUs as we speak.

1:32:41 So every instruction matters.

1:32:43 There's not...

1:32:45 The impact, at least in terms of CPU, is massive for everything that we do.

1:32:51 So first you come because it's an interesting subject,

1:32:53 then you stay because it's excellent,

1:32:55 and in the end you're very proud of it because it's in the hands of everyone.

1:33:01 Like so many people like, "Oh,

1:33:04 I'm working for whatever consulting company and I'm doing

1:33:07 some portal to download invoices for your PG&E." Wow, great.

1:33:14 Like, so many jobs are like that.

1:33:16 You're not going to, to tell that to your grandma.

1:33:19 But if you go to see your grandma and say,

1:33:21 "I do this so that you can play video on your laptop," they understand.

1:33:24 And this is very important, right?

1:33:26 Because you're working on VLC, FFmpeg, H.264.

1:33:29 It's in the hands of hundreds of millions of people and you have an impact.

1:33:33 And so you can be proud of yourself.

1:33:35 And so I think that in addition to doing a great resume,

1:33:39 all those things are why people contribute.

1:33:42 Yeah, those are side effects.

1:33:43 My favorite quote on this topic is John Collison.

1:33:45 He said, "The world is a museum of passion projects." You know,

1:33:48 everything out there is a passion project.

1:33:51 And open source multimedia and open source in general,

1:33:53 you can just do that so much faster.

1:33:56 There's such a faster network effect, you know?

1:33:59 I can open a cafe and that can be my passion project,

1:34:01 but I have to get building codes,

1:34:02 I have to build a building, I have to find a location,

1:34:05 I have to do all the, you know, all sorts of things.

1:34:08 Well, in the software world, that passion project can be, can move quickly,

1:34:12 it can be amplified by the network effect,

1:34:16 and that amplification can be more than the sum of the parts.

1:34:19 You know, you can be,

1:34:21 you can find people interested in extremely obscure things and have

1:34:27 a network effect and make something that is truly amazing.

1:34:31 And on that topic of passion projects Tim Sweeney actually

1:34:35 said in a reply to a tweet that was complimenting JB.

1:34:40 He said, quote, "Many things in the world only

1:34:44 happen because an awesome person decides to do it.

1:34:47 This is the case with VLC." And that speaks to something interesting to me,

1:34:51 that it does seem that a small number of people,

1:34:53 sometimes one person, can create something incredible in the software world.

1:34:59 Like you said this over and over and over.

1:35:02 I think JavaScript is an incredible thing created by, Initially a single person.

1:35:07 Some of the programming languages like Python and C and Java,

1:35:11 like just one person has this vision,

1:35:13 has this design, and brings it sometimes over a weekend is the initial spark.

1:35:19 Yes, Linus built Git in two weeks.

1:35:22 Wow.

1:35:23 It changed the world, Git.

1:35:24 I mean, it really changed the world.

1:35:26 Linus' passion project.

1:35:26 "Hey, I'm uploading this tarball to an FTP,

1:35:28 like deal with it."- But for me, it's not just in software, right?

1:35:32 And I believe in, in individuals that are going to change the world, right?

1:35:37 And it's with a good, as you said, vision, right?

1:35:40 I want to do that.

1:35:41 It is useful, it will be useful.

1:35:43 And whether it's going to like build train or cars or rockets or something like,

1:35:48 I believe people who believe in themself and have

1:35:52 a vision can have a huge impact for humanity.

1:35:56 Let's actually zoom out before we zoom back in.

1:35:58 We'll just keep going up and down the stack.

1:36:01 So you know, we've been talking back and forth VLC and FFmpeg.

1:36:06 Kieran, you said that FFmpeg and VideoLAN,

1:36:09 VLC coexist, and there's no central point of importance.

1:36:14 It's a kind of what you call the binary star system.

1:36:18 Uh, they succeed because of each other.

1:36:20 Can you explain the difference, how they interact?

1:36:23 What is the-...

1:36:24 are they competitors?

1:36:26 I don't think they're competitors.

1:36:27 I think the simple answer is, the short answer before I go into detail is

1:36:32 VLC is to FFmpeg as Android is to Linux.

1:36:36 So they depend on each other, but they coexist because of each other.

1:36:39 So they are a binary star system is the analogy I used.

1:36:43 By the way, I feel horrible that I just recently learned that Alpha Centauri,

1:36:47 the closest star system to us, is a triple star system.

1:36:50 And when you start doing the physics, it's a nightmare, right?

1:36:53 But, but, but like-- Hence the three body problem.

1:36:56 But anyway.

1:36:57 So a lot of FFmpeg pipelines involve the x264 project,

1:37:01 which is a VideoLAN project.

1:37:03 I would put a finger in the air and say

1:37:06 80-plus percent of those pipelines are dependent on a VideoLAN project.

1:37:11 VLC, obviously, as we've discussed, a VideoLAN project,

1:37:14 uses FFmpeg, gives it reach, exposure to weird files,

1:37:19 Historically used some donation money to fund FFmpeg development,

1:37:23 and we'll talk a bit maybe about some of the reverse engineering later.

1:37:27 So it's a binary star system.

1:37:28 They work and feed off each other.

1:37:30 Many of the developers are shared.

1:37:31 There's no central location.

1:37:33 It's a virtuous cycle working together.

1:37:36 And we should mention that x264 is the encoder for H.264 video standard.

1:37:42 So H.264 is the standard.

1:37:45 X264-- Is the open source implementation of the standard-...

1:37:49 that's used by basically everybody-...

1:37:51 for everything.

1:37:52 It's, that is the main driver of this.

1:37:54 When you think of an MP4 file that has H.264

1:37:57 codec in it-- If it came from a software environment,

1:38:01 like a data center or somewhere, the chances are it was created with x264.

1:38:06 And that's under the flag of VideoLAN.

1:38:09 That's a VideoLAN project.

1:38:10 So in the VideoLAN graphic, it sits in the VideoLAN world.

1:38:14 And VideoLAN has a, says a bunch of stuff in it.

1:38:16 Go to the VideoLAN website, there's a bunch of icons.

1:38:21 Like if you look, there is so many libraries, right?

1:38:25 libdvdcss-...

1:38:27 libdvdnav, libdvdpsi, libvlc of course, vlc-unity, libblu- Blu-ray.

1:38:36 Uh, yeah, there's many more.

1:38:39 And there is so many more, right?

1:38:40 Lately, lately the dav1d project that we might

1:38:43 talk about is the last project from VideoLAN.

1:38:46 It's everywhere, right?

1:38:47 And we do, we have a libspatialaudio lately that we announced.

1:38:50 We have a-- checkasm.

1:38:52 checkasm-- We'll talk about that later-...

1:38:54 which is like an insane project-...

1:38:56 but amazing.

1:38:57 So and x264 is one of those VideoLAN projects.

1:39:00 And my opinion, for example, is that x264 was,

1:39:05 is the most amazing encoder ever designed,

1:39:09 and this helped the adoption of FFmpeg.

1:39:11 A lot of people and large companies went

1:39:13 through FFmpeg because they wanted to use x264,

1:39:17 and x264 increased the popularity of FFmpeg.

1:39:20 But also VLC had its popularity because it's played

1:39:25 so many files that were done by FFmpeg, right?

1:39:27 So it's, it's many projects that are intertwined and work together.

1:39:33 Yeah.

1:39:33 Unfortunately, there's a, there's a thing on X

1:39:35 where VLC is mentioned and there's people,

1:39:37 "A quick reminder that it's FFmpeg inside doing

1:39:39 the actual work." And that, and that's like I said,

1:39:42 it's not, that's not the case.

1:39:44 We work together.

1:39:46 And to give you an idea, right?

1:39:48 When I compiled VLC for Windows,

1:39:49 I compiled around 16 million lines of code, right?

1:39:53 One million of those are inside the VLC repository,

1:39:57 and FFmpeg in total is probably two, around two, right?

1:40:01 But so it means that so many dependencies are outside.

1:40:04 And if you also look at FFmpeg per se,

1:40:07 FFmpeg also is integrating third-party libraries like x264,

1:40:10 but LibOpus and so many others, right?

1:40:13 So we all depend on each other.

1:40:16 Uh, yeah, that's why I was hoping to do this episode

1:40:18 as we are doing that just kind of joins FFmpeg and VLC-...

1:40:23 because it's really, it's, it's really two of the same,

1:40:26 like you said, binary star system and we're all just orbiting it.

1:40:29 Can we give a shout-out to some of the people along the way?

1:40:33 We didn't really quite talk about the history of FFmpeg,

1:40:38 so- maybe can you tell me about Fabrice?

1:40:40 Can you tell me about Michael Niedermayer?

1:40:43 Can you tell me about some of the key figures here?

1:40:46 Let's just talk about the eras of FFmpeg,

1:40:48 because there's key eras and key people that made this possible.

1:40:52 Uh, Fabrice Bellard, as you mentioned, creating the concept,

1:40:56 and then probably in the 2000 era, I would call the era,

1:41:00 eras tour of FFmpeg is the 2000 era was Michael Niedermayer.

1:41:04 So key things he got done was exhaustive support for DivX and Xvid at the time,

1:41:10 and all sorts of weird variants of what's known as MPEG-4 Part 2.

1:41:15 So this predates the MPEG-4 Part 10 that we're used to.

1:41:19 So this was 2000 era video codecs where there were,

1:41:22 oh, flavor after flavor of weird decoders.

1:41:27 At the time in the 2000s,

1:41:28 you needed a new player to play every different type of file format.

1:41:31 So there was Windows Media Player to play Windows Media formats.

1:41:34 There was RealPlayer to play RealMedia formats.

1:41:36 And those were the other,

1:41:38 the other key thing in FFmpeg at the time were native decoders for those.

1:41:41 I actually do remember being a teenager, I must have been,

1:41:46 figuring out there was this one player that could play,

1:41:49 could decode these files without having separate bloated players.

1:41:53 Because at the time when you downloaded RealPlayer,

1:41:54 there was a ton of other stuff in there, a ton of ads, a ton of other things,

1:41:58 and just having a simple library that was fast led to that.

1:42:02 And then I think 2008 was a, 2008 onwards was a big change because that's

1:42:10 when H.264 got its maturity and I think

1:42:14 something hopefully we'll talk about a bit more.

1:42:16 This was the beginning of high definition video.

1:42:19 So H.264 was the key decoder of that.

1:42:22 So I'd call that the late 2000s and 2010s,

1:42:24 and that's when the big reverse engineers

1:42:26 came along and really did astonishing work.

1:42:30 The beginning was a single player that could play Xvid, DivX, Windows Media,

1:42:36 and RealPlayer was already a massive achievement in itself without codec packs,

1:42:39 without weird stuff you had to download that had weird ads and weird spyware.

1:42:44 VLC 1.0 was out on those times, 2000, 2009, 2010.

1:42:49 And this is like where it exploded.

1:42:53 Yeah, without codec packs, it just works-...

1:42:55 across all these different-- It, de facto,

1:42:58 it's just like all the codec packs are FFmpeg inside VLC,

1:43:01 plus we have other modules for all the type of codecs.

1:43:03 But back at the time that wasn't, is there were weird, in the 2000s,

1:43:06 there were weird codec packs with DLLs coming from this place,

1:43:09 DLLs coming that-- With a lot of spyware-...

1:43:12 with spyware, with you know what.

1:43:13 It wasn't reliable, you didn't know,

1:43:15 and having a single player that was open source or single

1:43:18 playback module/player that could do this that was open source.

1:43:21 But I think the thing to emphasize is

1:43:24 this task in the 2000s that Michael did was Sisyphean.

1:43:27 It was really, the number of edge cases

1:43:30 are poor beyond comprehension in terms of you

1:43:33 could have a Chinese CCTV system that did one weird variant of MPEG-4 Part 2,

1:43:38 what's known as MPEG-4 ASP, and that was a weird variant,

1:43:41 and you had to fix that without breaking everybody else-...

1:43:44 times a million.

1:43:45 So that's, so you said that's where

1:43:47 a lot of the reverse engineering was happening.

1:43:49 It started in the 2000s with the Windows Media stuff because that was-...

1:43:53 proprietary.

1:43:53 It started with the RealMedia, so with Benjamin Larsson.

1:43:56 Kostya Shishkov.

1:43:57 Kostya Shishkov, that era.

1:43:58 Those were the key, that was the key groundwork.

1:44:00 And then in the 2010s was kind of the Paul Mahol,

1:44:05 Kostya era building, doing some of the most difficult codecs.

1:44:09 JB maybe can talk about GoToMeeting 4 and GoToMeeting 5, and-- What?

1:44:14 What's the GoToMeeting?

1:44:15 So, like, let's talk about this amazing Ukrainian guy called Kostya,

1:44:20 who was at that time living in Germany, and who was in love with Sweden, right?

1:44:28 He— And the guy was the most...

1:44:32 He's like, like a lot of the people in the community are very clever.

1:44:37 He's one of those who are, like, borderline geniuses, right?

1:44:41 He was able to reverse engineer extremely complex codecs,

1:44:45 And he does that, and we do a bit of engineers with Kieran,

1:44:49 but clearly not at this level.

1:44:50 No, no, yeah.

1:44:51 Um, he reverse engineered binary blobs, which are 20 megabytes?

1:44:56 Yeah, so just for reference, one megabyte binary blob to reverse engineer is

1:45:01 probably order of magnitude a month of work,

1:45:03 and this guy is doing 20, 30 megabyte blobs.

1:45:05 Maybe we'll talk about that in a minute,

1:45:07 about the subtleties of how you do that.

1:45:09 But this guy is doing it for very difficult and very obscure codecs.

1:45:13 And did that for fun, right?

1:45:16 And so GoToMeeting was a big problem with VLC because that was like

1:45:21 the number one feature request for a long time, so I put a bounty.

1:45:25 And the guy at some point said, "Okay, JB,

1:45:28 I'm going to do it." And in a matter of two months,

1:45:31 and then he explained how he did it.

1:45:33 He was just like, "Oh, I looked at the code,

1:45:35 like this looked like a DCTs that I used to see on WMV and so on." He

1:45:40 did that, and the funniest part is that the code he's written is a ton of jokes.

1:45:47 And there is, there is a ton of JB, right, my name,

1:45:50 and, and Kempf and Kempf and Kostya jokes inside the code.

1:45:54 The code is beautiful, right?

1:45:57 So one of the things I wanna comment is I've

1:46:00 gotten a chance to speak to some of the developers,

1:46:03 some of the assembly language level People,

1:46:05 and they all always make everything sound like it's kinda easy.

1:46:10 There's a kind of humility because,

1:46:13 maybe just the level of what's required to do

1:46:17 this stuff is so high that everything else seems easy,

1:46:20 I guess is the lesson to take away from that.

1:46:23 So in the community, like some of the most impressive

1:46:26 people are the ones doing reverse engineering-...

1:46:28 and the other ones doing the assembly folds, right?

1:46:31 Um, and both of those type of people are amazing.

1:46:36 x264, for example, became amazing because, of a guy called Loren Merritt—...

1:46:42 who is, was from University of Washington, I think.

1:46:44 At the time, yeah.

1:46:45 And who was, like, who made everything great and fast doing a ton of assembly.

1:46:51 Um yeah.

1:46:52 So this is like the the golden era, I guess, where so many things got done.

1:46:57 So, yeah, if you look at Kostya, for example,

1:46:59 he looked at the world as a binary specification.

1:47:01 He didn't need documentation or anything.

1:47:02 It's, "I have a binary and I can figure all

1:47:05 of this out." And he regularly used the phrase binary specification.

1:47:09 "Ah, you know, it's not a problem." And he would go away,

1:47:11 and he would come back, and he would do interesting stuff.

1:47:15 Can you actually speak to the details or add color

1:47:18 and texture to what it takes to reverse engineer a blob?

1:47:22 Yeah.

1:47:22 So let's look at GoToMeeting, for example, is a good one because,

1:47:26 um, I record a meeting on GoToMeeting, for example.

1:47:30 How do I play it back without needing this GoToMeeting player?

1:47:34 There may not even be a player.

1:47:35 I may, I may need to send a recording

1:47:37 of a meeting to someone that doesn't have a player or whatever.

1:47:40 So first of all, there's a ton of other stuff there.

1:47:43 There's an actual video conferencing client.

1:47:44 You need to go and find, it may be easy,

1:47:46 it may not be easy to find the actual module doing the decompression.

1:47:50 You need a way to actually dump the YUV data from the module.

1:47:55 So often it involves opening in a disassembler,

1:47:58 trying to guess where the hooks are to incorporate that module

1:48:03 and run that module natively to decode a sample file.

1:48:07 So figure out where this module is doing the decoding process

1:48:10 and find a way to hook in and output the raw YUV data,

1:48:14 'cause you will need that-...

1:48:17 as a point of comparison for when you actually do the reverse engineering,

1:48:19 'cause you'll need to be bit exact or in some cases close to bit exact.

1:48:24 And then you open up your disassembler,

1:48:27 use a lot of intuition to go and figure out,

1:48:30 you know, where the DCT is, where's entropy coding.

1:48:32 There, there is a kind of, not a rule book,

1:48:36 but there's always a pattern of some sort.

1:48:38 For example, GoToMeeting, you know it will be a s- a lot of screen codec tools.

1:48:43 There's also different variants, so often I think there's, what,

1:48:45 GoToMeeting 4, 5-- Well, 2 or 3, 4, I think.

1:48:48 2, 3, 4.

1:48:49 So as you mentioned here, going to Perplexity,

1:48:51 GoToMeeting uses its own proprietary codec

1:48:53 for older s- recorded sessions historically stored

1:48:56 in WMV files that require a special decoder to play properly on Windows.

1:49:01 Without this decoder installed,

1:49:03 Windows Media Player and some editors cannot decode the video track,

1:49:08 so you may only hear audio or see a black screen.

1:49:11 Boy, do I remember that.

1:49:13 But this is reverse engineering that.

1:49:16 This is key, right?

1:49:18 Because the GoToMeeting is something that not many people know anymore, right?

1:49:21 Well, you know about Zoom and, and Teams and so on.

1:49:23 But like, now let's fast-forward 10 years, 15 years,

1:49:26 and like this is a GoToMeeting.exe for Windows 32 bits, right?

1:49:30 Which is like, oh yeah, but I'm on Android,

1:49:33 I'm on an iPad, I'm somewhere else, right?

1:49:35 How are you going to do that?

1:49:37 I'm going to be on RISC-V, on Arm.

1:49:39 Those are blocked, but there are tons of files we need support for the future.

1:49:44 And this is why those type of work are— exceptionally useful for humanity.

1:49:51 I just have to say, though, that reverse engineering process is mind-blowing.

1:49:56 It's crazy.

1:49:56 It's like, it's a kinda like, you know,

1:50:00 I've been reading a lot and interview archeologists.

1:50:02 I mean, you just have so little signal.

1:50:05 Yes, yes, you know over time you get so much experience,

1:50:09 you understand the structure of the original code,

1:50:11 so you can kinda start inferring basics.

1:50:14 But you're like-And we like archaeologists with a little brush

1:50:19 trying to reconstruct the entire human civilization- Kieran is too humble,

1:50:23 but Kieran has done some reverse engineering also.

1:50:24 Of CineForm, yeah, at the time, um-- CineForm, nice-...

1:50:28 yeah, at the time before actually led to the open sourcing of that work.

1:50:31 Um, so in parallel to doing the binary side, you obviously have samples.

1:50:35 In many cases, you don't have many samples so you

1:50:38 have to figure out what all the different flavors are,

1:50:40 and you may have a s- So CineForm, for example,

1:50:43 is actually a collection of different approaches and toolkits

1:50:46 within that codec 'cause often it grows naturally.

1:50:48 And the hard part is finding a sample that gets you kind

1:50:52 of somewhere to start without having to implement 10 different other things.

1:50:57 So start there.

1:50:59 I think thankfully at the time I found a sample

1:51:01 by pure chance that had a lot of flat blocks.

1:51:03 It was animation, so that really helped a lot

1:51:05 because it wasn't using particularly complex coding tools, et cetera,

1:51:08 and you could kind of get somewhere and then,

1:51:11 and then build up and build up until you figure, "Hey, here's a few bits here.

1:51:14 I missed this.

1:51:15 I missed this, this if branch that it does," and go,

1:51:17 "Oh." So when we say samples, you mean sample videos-...

1:51:20 and then, and then you're tracking,

1:51:22 trying to infer, like, what is this codec doing-...

1:51:25 by observing the sample and then looking at what,

1:51:28 at the lo- at, at the machine lo--

1:51:31 The machine code saying-- At the machine code-...

1:51:32 "Ah, I have byte, this byte is six.

1:51:34 Take this branch." And in a different sample, oh, it's-- That's nuts, man.

1:51:37 And-- That is nuts-...

1:51:39 so you see, this is nuts.

1:51:40 Then you go to things like GoToMeeting.

1:51:42 Yeah, yeah.

1:51:42 It's like-- Mine was easy, right?

1:51:44 imagine-- Yeah, right-...

1:51:46 two order of magnitude of more complexity.

1:51:49 A guy alone somewhere in Germany doing that.

1:51:53 And for a long time, you work,

1:51:55 you're in a black box because a decoder, for a long time,

1:51:58 because there is so many steps from the entropy decoding,

1:52:01 the intra prediction, the motion prediction, the IDCT, and so on.

1:52:04 For a long time, you don't see anything, right?

1:52:07 So you're debugging purely in memory.

1:52:10 Debugging guesswork.

1:52:11 And you may have the buffer

1:52:12 that the coefficients are stored in completely wrong,

1:52:14 and so you may be going down a complete rabbit hole thinking it's this and then,

1:52:18 oh damn, that's not, that's, that's something else,

1:52:20 and-- And you're doing that on binaries that are tens of megabytes,

1:52:25 millions of instructions, right?

1:52:27 So you're stepping through the debugger, like one by one,

1:52:30 you know, instruction by instruction going, "Hey, this instruction changes this.

1:52:33 This does this."- Pausing the program on the CPU level.

1:52:36 Like it's-- Pausing it, yeah, on the CPU level,

1:52:38 watching what's going on, trying to figure out-- Sometimes you need to, like,

1:52:40 be in a VM, so yeah, that you can pause the VM.

1:52:43 Yeah, pause the VM, dump the memory,

1:52:44 'cause there could, some of the codecs could have encryption.

1:52:46 There could be like a DRM on there.

1:52:48 So you need to dump the memory from a virtual machine.

1:52:52 Like when I joined École Centrale Paris in 2003,

1:52:55 Jon Lech Johansen basically broke the DVD specification and created DeCSS,

1:53:00 showed us how he was breaking a DRM, which was MP4 FairPlay from Apple.

1:53:06 What he did on his laptop, and I was young, I was 21,

1:53:10 was just like mind-blowing because he was basically debugging

1:53:13 Windows inside a type of VM with ex- Like, wow.

1:53:17 It's incredible.

1:53:18 It's mind-blowing and inspiring.

1:53:20 Does it get, like from your experience

1:53:22 and from what you've seen in the community, does it get discouraging?

1:53:25 Does it get-- People help you.

1:53:27 People send you samples.

1:53:28 People are keen.

1:53:29 Sometimes you don't have access to an encoder,

1:53:31 so this is even more difficult because you just,

1:53:34 you just ask and you have to ask for samples.

1:53:37 I remember VideoLAN used to tweet for samples at one stage.

1:53:40 "Hey, I need this obscure sample," and-- For a long time I was,

1:53:43 "Oh, I need this codec,

1:53:44 and I need this codec."- And if you were really lucky, you would find like...

1:53:47 If you were unlucky,

1:53:48 you'd get like one or- you'd get nothing or you'd get one or two,

1:53:51 and then they would...

1:53:52 Sometimes you'd find a goldmine.

1:53:53 It's like, "Yeah, my company has 100,000 of these files

1:53:55 'cause we're dependent on it for some reas-" Um,

1:53:57 and so those are the, those are kind of the best if, if...

1:54:00 Because then they can test bit exactness across the huge range of coding tools.

1:54:06 Can you explain bit exactness?

1:54:08 Bit exactness, so most but not all video codecs,

1:54:12 certainly from about the 2000s onwards, have a bit exact definition,

1:54:17 so every implementation must produce exactly the same bits, bit for bit,

1:54:23 in exactly the same data that comes out of a decoder.

1:54:26 For like a large number of samples?

1:54:29 For a given sample.

1:54:30 So Lex's implementation, JB's implementation,

1:54:31 and my implementation of H.264 must match bit exactly.

1:54:35 That wasn't the case in the '90s of MPEG-2,

1:54:38 probably fair to say one of the biggest mistakes the video industry made,

1:54:42 and I think people who were in the room in '92,

1:54:44 I don't think, most or both of us were in diapers, I suspect-...

1:54:47 but have acknowledged I would give a shout-out to Yuri Reznik.

1:54:53 He's acknowledged that was one of the big mistakes of the era.

1:54:56 And you're saying the encoders needed to be

1:54:57 able to run tests and then the bit exactness.

1:55:00 the bit exactness.

1:55:01 I mean, that's a nice thing to guarantee.

1:55:04 Like there's a parallel sort of development

1:55:06 here on the way the web browser works,

1:55:09 which is a, you know, takes HTML and displays it,

1:55:12 and there's no bit exactness there across the different engines.

1:55:16 I would point out actually FFmpeg is unique in the sense that it's,

1:55:20 it has been a winner-takes-all scenario.

1:55:21 You have...

1:55:22 Browsers is a good analogy because it has to parse a lot

1:55:25 of different content and render it in a particular way, like a decoder.

1:55:29 But there still are multiple browser engines.

1:55:30 There's Firefox's one, there's Chrome's one,

1:55:32 there's a few Japanese ones that are pretty decent.

1:55:35 That's not been the case in multimedia in general across a wide range of codecs.

1:55:40 FFmpeg has kind of won it all, I suppose,

1:55:43 in a sense because of, because of the fact

1:55:45 that you can get every new codec added

1:55:47 is actually worth more than the value of that codec

1:55:50 itself because it makes the whole thing better.

1:55:52 Man, this is really cool.

1:55:54 Going to Perplexity.

1:55:55 Yuriy Reznik is a multimedia and signal processing researcher,

1:55:57 got his PhD in computer science from Kyiv University

1:56:01 with over 150 papers and more than 80 granted US patents,

1:56:05 contributor to major multimedia standards including H.264, MPEG-4,

1:56:11 AVC-H.265 MPEG-4 ALS, G.718, and-- G.71 is telco stuff.

1:56:18 Telco.

1:56:19 Oh.

1:56:19 And so he was more connected to companies.

1:56:21 RealAudio, RealVideo, right?

1:56:22 That was-- Oh, yeah-...

1:56:24 very important at that time-...

1:56:25 Zencoder, Brightcove, Contex.

1:56:26 This, man, I need to hang out with Yuriy.

1:56:29 He's legit.

1:56:30 And he's like one of the most nice person-- Slack guy, yeah-...

1:56:34 ever, right?

1:56:34 Like for example, for my for my startup

1:56:37 that I'm doing right now called Kyber, right?

1:56:40 I met Yuriy because I met him every year

1:56:42 at the Mile High Video Conference, which is in Denver.

1:56:45 And he gave me like so much good ideas and good things.

1:56:50 He's like really amazing person.

1:56:52 He tells us how, how, you know- How great it is to be, you know, even know us.

1:56:56 And then we just like, you know,

1:56:58 you look at that and it's, I think it's the other way around,

1:57:00 Yuriy- That reminds me of a thing that you

1:57:04 mentioned to me about FATE testing and, like,

1:57:06 the insanely rigorous process that's used

1:57:09 to test everything that's incorporated into into FFmpeg.

1:57:12 Can you take me through the testing process?

1:57:15 Yeah.

1:57:15 So FFmpeg has a system called FATE, FFmpeg Automated Testing Environment.

1:57:19 Because FFmpeg runs on so many different OSs

1:57:21 and can be compiled with so many different compilers,

1:57:25 there's been a crazy number of configurations.

1:57:27 So you can see the absurd combination of compiler variants,

1:57:34 operating system variants, instruction sets.

1:57:37 You can see at the top macOS has tons

1:57:39 of different variants because it has iOS, it has tvOS.

1:57:43 Well, I'm looking at a page fate.ffmpeg.org.

1:57:47 81 minutes ago, 76 minutes ago, looking at the different architectures,

1:57:51 the operating systems, the different compilers, Apple Clang version...

1:57:57 Combinations are crazy.

1:57:58 the combination is insane.

1:57:59 RISC-- So these are all run by volunteers, so these are all volunteer systems.

1:58:02 The ones at the top, for example, the Macs I host in my office,

1:58:06 for example, Host all sorts of different stuff.

1:58:09 Other people host other things.

1:58:11 So it's really there to make sure...

1:58:14 because FFmpeg does quite complex C code,

1:58:16 for example, you do have miscompilations.

1:58:18 So the compiler will sometimes compile C code incorrectly.

1:58:22 For example, this happens once in a while.

1:58:25 Oh, there's like, there's a log of all the compilations.

1:58:29 Yeah, log of all the compilations, all the tests.

1:58:30 I think one of the other ones will show all the tests passing.

1:58:33 If you click, you can see all the tests-...

1:58:36 back.

1:58:36 All tests successful.

1:58:37 In logs test, yeah.

1:58:39 So you see all those tests are passing of all the different codecs,

1:58:44 all the different filter transformations,

1:58:46 all the- The level of scale is quite crazy.

1:58:50 Oh, that's nuts.

1:58:50 On all the combinations.

1:58:51 It's not just a matrix at this point.

1:58:53 It's like a pivot table of different combinations.

1:58:56 That's nuts.

1:58:57 And it's a key part of what we do because you

1:58:59 may be able to test something locally, you make a change,

1:59:03 but actually that breaks GCC version 11 on Mac

1:59:06 or something like that, and you're able to then fix that.

1:59:10 We also have miscompilation, so the C code,

1:59:12 sometimes the compiler can have a bug in it where it creates the wrong output,

1:59:16 and that can have quite a big effect sometimes

1:59:18 on a video because of the way frames have dependencies.

1:59:21 Even a small change in the output can cascade to actually quite big glitches.

1:59:28 You see PowerPC, you see RISC, you see ARM.

1:59:31 There was PowerPC, there was RISC,

1:59:32 there was weird stuff in the past like DEC Alpha.

1:59:34 There was—- You see Visual Studio, different versions of Clang or GCC.

1:59:37 Visual Studio, Intel compiler, Apple Clang, you name it.

1:59:41 What are some of the pain points?

1:59:43 Like maybe do you have emotional triggers

1:59:46 maybe nightmares about a particular operating system, a particular container,

1:59:51 codec combination of-- I mean, for me,

1:59:54 it's really easy because I have a day job.

1:59:56 My company builds...

1:59:57 The company I started builds equipment for broadcasting

2:00:01 sports matches between TV stadiums and studios, for example.

2:00:05 We have to work with 10-bit video, and 10-bit video has a set of challenges

2:00:10 that you can't process 10-bit data natively on a CPU.

2:00:14 So that means you have to stick it in 16 bits.

2:00:18 So that means you have six wasted bits.

2:00:21 So there's different packing formats to actually pack the data more

2:00:24 efficiently because when you send that over a network, you lose...

2:00:27 'Cause you need to save that 40%.

2:00:30 For example, on PCI Express, you may only have bus bandwidth to do that.

2:00:33 And so I think internally we have about...

2:00:35 Some are industry ones and some are internal to our own hardware that we build.

2:00:40 We have a, I think a 5 by 5 or 6

2:00:43 by 6 matrix of every single format to every single other format conversion.

2:00:48 In fact, one, one of them I sent you,

2:00:50 and they're all written in handwritten assembly,

2:00:52 and they're all written, and they all support different CPU generations.

2:00:57 So this is really traumatic,

2:00:58 handling all these different combinations times a million.

2:01:02 By the way, the company you're talking about is Open Broadcast Systems.

2:01:04 Yeah, so no, no relation to the free OBS streaming service.

2:01:08 But JB and I have started companies

2:01:11 broadly speaking around the FFmpeg VLC ethos, so that's really low-level work.

2:01:16 So in most companies, this wouldn't be written in assembly.

2:01:19 It would be accepted that C is fast.

2:01:23 As you can see from that, C is not fast.

2:01:26 So here it says 62 times faster than C.

2:01:31 Yeah.

2:01:31 So it's taking those, the ethos of doing low-level programming,

2:01:36 real-time programming, and using that for commercial applications,

2:01:41 and JB and I have started companies around that, in many

2:01:44 cases hiring developers from the open source community to use that ethos.

2:01:49 And so that's a great example of some of the things we're doing.

2:01:53 In most companies, it would be, say, "Oh,

2:01:56 I'll write this in C and it's fast and we're

2:01:57 done," but actually you can get a lot better.

2:02:00 For me, like, some of the headaches we have

2:02:03 is around some OS that are difficult to support, right?

2:02:07 Because if you look at VLC and thanks to FATE and FFmpeg, we run on...

2:02:15 The last version of VLC runs on Windows XP

2:02:19 and still run there and runs on Windows 11.

2:02:23 We work on macOS 10.7 to the latest macOS, whatever it is, right, 26.

2:02:30 We work on iOS since iOS 9, well, we are actually iOS 26, right?

2:02:36 We support many types of Linuxes, BSD, Solaris.

2:02:41 The last version still runs on OS/2, right?

2:02:44 Like there is maybe 10 users of OS/2 in the world,

2:02:48 and one of them is maintaining VLC.

2:02:50 Then you realize that this very small team around VLC and using FFmpeg codecs

2:02:55 and all the other ones support more OSs than Microsoft or Google or Apple,

2:03:02 and they have infinite amount of power and resources.

2:03:06 But for example, the worst is iOS.

2:03:11 For in order to build on iOS 9, we need to do some very clever mixing of several

2:03:17 versions of the Xcode IDE and SDK from Apple, from several versions,

2:03:23 and do a type of Frankenstein version

2:03:25 of that so that we can still support iOS 9,

2:03:28 which is not supported at all by the compiler of Apple

2:03:32 in order to still run on Arm32 on iOS 9.

2:03:37 And you've seen on FATE that it was still supporting iOS 9, right?

2:03:41 So, My headaches are mostly related to the support of so many OSs.

2:03:48 And it's important because, like,

2:03:49 we receive so many people saying, "Hey, thank you.

2:03:52 I still have my iPad 2 to watch movies," and it still works on iOS 9, right?

2:03:58 And it's also an impact of, like, not forcing people to buy new hardware when

2:04:02 it works fine if you optimize it correctly.

2:04:05 Which brings us to what we were saying about assembly.

2:04:08 It's also fighting, like, the fact that you need to buy

2:04:12 something new nonstop while you could optimize more, which is a lost art.

2:04:19 You gotta tell me about this lost art or this, uh-...

2:04:24 the carriers of the flame of assembly.

2:04:27 What what is, what is assembly?

2:04:30 Why is it beautiful?

2:04:31 Why is it challenging?

2:04:32 How does it work?

2:04:34 So when you write assembly code,

2:04:36 you write this using the instructions the actual processor is using directly.

2:04:39 So most of the time you would write in a language,

2:04:43 let's take C as a good example.

2:04:44 The compiler would use that to create assembly language

2:04:48 and machine code instructions for you based off your C code.

2:04:53 And there's a specific flavor of assembly

2:04:55 that we use in FFmpeg that's called SIMD,

2:04:57 SIMD, single instruction, multiple data.

2:05:00 So this means, for example,

2:05:03 say I want to add five to a number in scalar assembly,

2:05:07 so this is what's known as you work on an individual element.

2:05:11 So I wanna have a number of— I have the number ten and I want to add five.

2:05:14 I use the add instruction, and I add five to ten, and I get 15.

2:05:19 With SIMD, with SIMD, I can have a whole vector of 16 different numbers.

2:05:23 They could all be different.

2:05:25 If I want to add five to that, I can run one instruction,

2:05:29 and that one instruction sums all 16 elements.

2:05:33 And that, as you can imagine, lends itself very well to video.

2:05:38 Video is, you know, pixel grid,

2:05:40 so I can perform operations on multiple pixels at the same time.

2:05:44 The key thing that we do differently in FFmpeg is we

2:05:47 don't use any abstractions or any major abstractions on top of that.

2:05:51 So there's a part of the world that uses what's known as intrinsics.

2:05:55 So these are C functions that behave very similarly

2:05:58 but not quite the same to writing assembly by hand.

2:06:02 So the registers that data is stored in on the CPU,

2:06:07 the compiler allocates those for you.

2:06:10 And so the key thing to understand was when we write SIMD is we have a 10x,

2:06:14 and not percentage, 10x to 50x speed improvement.

2:06:18 That, that function is 62x, um—- That's nuts.

2:06:23 on the FFmpeg account, as you know,

2:06:24 posts and tweets a lot about that to try and say, "Hey,

2:06:26 we are doing this stuff."- You are a person who sees the beauty in assembly,

2:06:31 but it's also extremely useful for these kinds of application to actually-...

2:06:36 significantly outperform even C, which is crazy.

2:06:40 It is necessary.

2:06:41 Right?

2:06:42 Because, like, one of the projects that we

2:06:44 need to talk about is called dav1d, right?

2:06:48 So dav1d is a decoder for the format that was done

2:06:52 by Alliance for Open Media which is an, a video decoder called AV1.

2:06:58 So if, for people who don't know, we've been talking about H.264.

2:07:02 AV1 is another hugely popular standard and codec

2:07:07 that is increasingly taking over the internet.

2:07:13 And when this format was launched many people said,

2:07:16 especially even from the Alliance for Open Media, right, which is Google,

2:07:19 Netflix, Amazon, uh, Mozilla, say, "Well, this format is so complex,

2:07:23 it must be done in hardware to do decoding," right?

2:07:27 And well, I arrived with a few other people mostly Ronald Henrik,

2:07:33 and Martin, and we said,

2:07:34 "We need to have an extremely good software decoder because it's going

2:07:39 to take time to have hardware." And so we wrote this project,

2:07:45 which is beyond insane.

2:07:47 We are talking about 30,000 line of C,

2:07:51 but 240,000 lines of handwritten Assembly, right?

2:07:57 Handwritten Assembly, 240,000 lines.

2:08:02 That's incredible.

2:08:02 That mean— I mean, some of the stuff we're

2:08:04 talking about is probably the biggest Assembly code bases.

2:08:10 To give you an idea, and Kieran can correct me,

2:08:12 but I think the FFmpeg has 100,000 lines of Assembly for all the codecs.

2:08:17 For all codecs.

2:08:17 Mm-hmm.

2:08:18 And just this one has 240,000.

2:08:21 It's a VideoLAN project, of course.

2:08:23 And it is optimized at the maximum because the motto

2:08:29 when we're starting the project is every cycle matters, right?

2:08:33 Every cycle matters because David is used

2:08:37 in VLC and in some software AV1 playback stacks.

2:08:40 We are talking about probably 3 billion devices

2:08:44 which are going to decode video nonstop because,

2:08:47 for example, 30% of the video from Netflix are now in AV1,

2:08:51 50% of YouTube, right?

2:08:53 So, and you often don't have a hardware

2:08:55 decoder because not many devices have a hardware decoder.

2:08:58 And with dav1d, we realized that with one

2:09:01 or two cores you were able to decode 720p correctly.

2:09:05 So it is—literally—- Yeah, that's dav1d-...

2:09:09 incredible, right?

2:09:09 That's dav1d.

2:09:09 Look at that Lex.

2:09:10 Uh, yeah, so this is another spicy tweet from you.

2:09:14 This is what peak video codec should look like,

2:09:19 79.9% assembly—- That's almost-...

2:09:22 19.6% C and 0.5% other.

2:09:26 And what's incredible is with those tweets, which is factual, people get crazy.

2:09:31 They are unhappy, right?

2:09:33 They say-- For a year,

2:09:34 for the last two years they go crazy, "No, intrinsics is fine.

2:09:36 The compiler is..." Oh, they go,

2:09:37 "I have never-"- "You can optimize your compiler,

2:09:39 auto-vectorization, it's your fault,

2:09:41 you don't understand." And we've tried that forever, right?

2:09:45 For two years, and two years later,

2:09:47 showing hundreds of examples of handwritten assembly.

2:09:49 "No, no, no, you're doing it wrong.

2:09:51 The compiler can do this."- So we

2:09:53 should actually just articulate a little clearer.

2:09:55 So the intuition there from the software engineering folks,

2:09:57 when you have code like...

2:09:58 Okay, let's just take an example, C++.

2:10:01 There's a compiler that's doing a lot of the optimization.

2:10:05 Yes.

2:10:06 And the presumption is if you have a good enough compiler,

2:10:09 if you continue to improve the compiler, you're going to generate code-...

2:10:14 that can perform like optimal performance.

2:10:16 You cannot possibly beat it.

2:10:18 Yeah.

2:10:18 Yes.

2:10:18 And you're consistently challenging that thought

2:10:20 that if you do-- By orders of magnitude-...

2:10:22 by orders of magnitude-...

2:10:24 handcrafted assembly can outperform C.

2:10:27 The two things that they tell us is,

2:10:29 yeah, but modern compilers have auto-vectorization, right?

2:10:32 Because SIMD that we're doing is vectorization.

2:10:34 And like it's not even close, right?

2:10:38 It's not even close, right?

2:10:39 It's not like 5%, 10% slower.

2:10:41 It's multiple times slower.

2:10:43 So can we...

2:10:44 I don't know if you can say something philosophically,

2:10:45 because there's a lot of great software engineers,

2:10:46 great engineers, great machine learning lot of great software engineers,

2:10:49 great engineers, great machine learning people.

2:10:51 Karpathy will listen to this and say,

2:10:53 "What's the intuition he's supposed to get from this?

2:10:56 What are we supposed to..."- Karpathy learnt

2:10:57 assembly because of the tweets by the way.

2:10:59 I just...

2:10:59 He start- He went, he's like, "Oh, I think this is a movement."- He's like,

2:11:01 "Let me figure out what's happening here."- No, no, he,

2:11:03 and you know the way he documents his work and so.

2:11:06 Philosophically, what's important to realize is that we passed

2:11:09 the time where hardware was going so much faster, right?

2:11:13 We are at the end of Moore's law.

2:11:15 We have limitation for for AI, for memory.

2:11:18 You need to go down in the stack and optimize

2:11:21 more to get more power from what you have,

2:11:24 because our request for power, CPU power,

2:11:27 GPU power are exploding while the hardware is not exploding in speed, right?

2:11:35 So you- what people do is that they add more cores, right?

2:11:38 But that's basically like at some point you can add 250 cores, right?

2:11:42 So what we do is to take every inch of the machine.

2:11:47 Not just that, not just that.

2:11:48 We abuse the machine.

2:11:49 We go and use, we use the machine

2:11:51 in ways that the, that the creator didn't expect.

2:11:54 Sometimes we use an instruction that's completely unrelated to what we do.

2:11:57 We use a cryptography instruction in video processing to do nothing related.

2:12:02 And one of other things that we do, for example in dav1d, which is a bit crazy,

2:12:06 is that we don't use the function calling convention from the operating system.

2:12:13 We should explain that.

2:12:14 That is extremely-...

2:12:15 complex.

2:12:16 But basically, usually when you do move

2:12:18 from one function in, in code to another, there is a way to save the registers,

2:12:22 the state of the CPU to enter another function.

2:12:26 And this is like standard.

2:12:28 It's a bit complex.

2:12:29 I would simplify this a bit.

2:12:30 So, so dav1d does things to abuse the calling convention.

2:12:33 You could define the calling convention as I've written

2:12:37 a function and I want to call another function.

2:12:39 How is the data shared between the functions?

2:12:42 Because there's a convention, what's known as a calling convention,

2:12:45 and what dav1d does for optimal reasons

2:12:48 is create its own calling convention sometimes.

2:12:51 So if I wanna call Lex Fridman's library, we got,

2:12:54 we've got to agree on a convention so that I

2:12:56 can share data with you in the assembly language space.

2:12:59 And one of the challenges in assembly is every operating s- well,

2:13:02 not every operating system, but there are,

2:13:04 well, at least four that I can think of on x86,

2:13:07 Linux 32-bit, Windows 32-bit, Windows 64, Linux 64.

2:13:10 They all have their own calling conventions.

2:13:13 And so one of the amazing things Loren Merritt did,

2:13:15 who we talked about before, was create a very lightweight abstraction layer,

2:13:19 so you could write your assembly code once

2:13:21 and it handled all the calling convention stuff for you,

2:13:25 which was always a problem because you had to manage four different variants.

2:13:28 But dav1d takes this even further,

2:13:31 for speed reasons it does its own calling convention,

2:13:34 within itself to bypass the kind of rules,

2:13:38 the rules of, the rules of sort of functions and say,

2:13:40 "Okay, actually I'm gonna call a function this way because I know it's

2:13:44 within my library."- Does it have

2:13:46 to be special to every single operating system?

2:13:48 Well, if it's custom, no.

2:13:49 But the, the, the challenge is in general, yes,

2:13:52 and in terms of, in terms of each instruction set.

2:13:55 So the thing to also emphasize is we do this on every instruction set.

2:13:59 So every instruction set has its own handwritten assembly,

2:14:01 which is even more crazy.

2:14:03 And that, that, that matrix has got bigger in recent years because of RISC-V,

2:14:08 because of ARM64, because of the new SVE.

2:14:12 There's SME.

2:14:13 x86 has AVX-512, AVX.

2:14:16 So we do runtime processor detection.

2:14:19 We see what the machine FFmpeg is running on or dav1d's running on is

2:14:23 capable of, because you could be on a laptop from 2008 where this isn't there.

2:14:28 Runtime detection, we set function pointers accordingly.

2:14:31 And then from then on, off you go.

2:14:34 Or you could be on a machine with RISC-V.

2:14:36 Yes.

2:14:37 And in all that, we don't even respect the calling

2:14:39 convention of the operating system in order to be faster,

2:14:42 because we know that we are going to be called from within our binary,

2:14:46 so we can share data without saving all the registers in the common way,

2:14:50 because that can lead to loading and saving registers

2:14:53 on the L1 and L2 CPU and gets us faster.

2:14:58 So that's why I said that understanding CPU architecture,

2:15:03 computer architecture is key.