New video codecs explained: AV2, H.265, H.266 | Lex Fridman Podcast

0:02 Let's talk about the future of um multimedia.

0:07 I have a ffmpeg, VLC, but uh some of the codecs we didn't really mention AV2.

0:12 So, can we just lay out what is AV2?

0:16 What is the hope for it?

0:18 What is H.265, H.266?

0:20 So, AV1 is this codec that is done by the Alliance for Open Media, right?

0:25 Where there is uh Google, Netflix, Amazon, Apple, uh VideoLAN,

0:30 where we try to make a royalty-free very good codec, right?

0:34 And now it's being deployed.

0:35 But actually the codec was finished in 2018.

0:39 But a codec takes years to be used in wide scenarios, right?

0:46 So, uh AV2 is a next generation of this codec.

0:49 It's 30% better.

0:51 Right?

0:51 So, if you keep the same uh quality,

0:54 you get 30% bandwidth reduction compared to AV1.

0:58 What's the connection with uh the David and AV2?

1:01 We are going to do a David 2, right?

1:03 That I called David because "de" is two in French.

1:07 Ah, well done.

1:08 And and you have to know that David is

1:10 an actual uh what we call recursive acronym, right?

1:13 Because it means D David is an AV1 decoder, right?

1:18 Oh, nice.

1:19 Nice.

1:19 I didn't even think of that.

1:21 And uh people should know that David's spelled with a one.

1:24 Yes.

1:25 It's um And so David 2 is going to be spelled with a two, please.

1:28 It's going to be D A V 2 D.

1:31 Sorry, I don't know how you pronounce that.

1:33 Uh And again, we did a demo at the CES of VLC running the first demo of AV2.

1:39 So, can you clarify to me the specification

1:42 of AV2 and then the encoding and the decoding?

1:47 Sure.

1:48 So, the specification is like the document which

1:52 explains how the codec is supposed to work, right?

1:54 And that's really AV2.

1:56 That is AV2.

1:57 Like H.264.

1:59 Yeah, right?

2:00 Then you have an encoder.

2:02 The current encoder is called AVM and there will probably be other encoders,

2:06 probably one called SVT-AV2.

2:09 And those are the encoders.

2:10 The same way X264 is an encoder to H.264.

2:14 The same way that X265 is an encoder for the H.265 codec.

2:19 And the decoders for AV1 is David.

2:25 The decoder for AV2 is David 2.

2:28 The decoder for H.264 is FFH.264 inside FFmpeg.

2:36 The decoder for HEVC is FFHEVC inside FFmpeg.

2:41 Um and there is a next generation codec from the MPEG world after H.264, H.265.

2:48 There is one that is called H.266.

2:51 Also known known as VVC.

2:54 So HEVC is H.265.

2:57 VVC is H.266.

3:01 Why is H.266 super sexy?

3:04 It's so much better.

3:05 So the the question often we have is why are there two names?

3:08 Because most of the time it is a conjunct work from the ISO world and the ITU,

3:15 which is the International Telecommunication Union.

3:19 So these two regulatory bodies

3:21 is a private entity and one is the United Nations.

3:23 Which one is the private?

3:24 ISO is private.

3:26 In theory, H.264 is MPEG-4 Part 10.

3:31 H.264/AVC.

3:36 Mhm.

3:37 And this is the full name.

3:38 Nice.

3:39 So it's a concatenation of the ISO name and the ITU name.

3:42 Yeah.

3:43 Even though they work together.

3:44 This is this is politics, historical, you know.

3:48 And for HEVC, it's MPEG-H H.265 HEVC.

3:53 Got it.

3:54 And there is H.266, which is also named VVC.

3:57 Is there a high-level thing to say about the improvement?

4:00 30% each generation is a best summary.

4:04 This is true both for the AV codex and the H.26456.

4:09 So the professional who are listening to us are going

4:12 to kill us because they know it's 35%, 25%, 50%,

4:16 but globally you need to know that HEVC 30% better than than H.264,

4:22 H.266 30% better than H.265 because there is so many case and so many scenarios.

4:27 For example, there are cases especially for screen recording where

4:32 the gains are humongous because you arrive you have the right tool

4:37 that is done for that and so for specific video a new

4:40 generation is going to give you 70% gain or 80% gain.

4:44 Um but there used to be a ton more codex,

4:48 but now the two main codex for transmission are the H.264,

4:52 H.265, H.266 and the other is AV1, AV2.

4:55 And I guess that the major difference would be the cost of encoding.

4:59 Yes and the royalty of the patents.

5:03 And this is the reasons why you see the AV version

5:05 of codex is because um they try to be as royalty free,

5:10 which means no cost for the patents as much

5:13 as possible because what you need to know

5:16 and we've not talked about that so far is

5:17 that multimedia is what we call a patent minefield.

5:21 There is two places where it's where you have the most patents,

5:25 it's everything related to 3G, 4G,

5:27 5G RF and multimedia um because it's very mathematical

5:32 and you can get great gains and so on.

5:35 Um so Google and Meta and Netflix wanted something where it was royalty free.

5:41 There are people who say that they have patents outside,

5:43 but they are fringe patents, right?

5:45 So it's mostly true that it's patent-free.

5:48 Well, we should extend patent patent checking was done

5:51 as part of the standardization process in AV1 AV2,

5:55 whereas patents are not even discussed in the MPEG world.

6:01 Patents are off-topic completely.

6:03 Can you educate me on the patents side?

6:05 So, usually so MPEG does a format, right?

6:08 And then there is everyone comes around and say, well,

6:11 I have all those patents on the format and they

6:13 do usually a union called was called MPEG LA,

6:18 MPEG Licensing Association,

6:20 and you put all the all the patents in and then

6:24 you ask everyone who's using this format to pay for it.

6:26 Wait, can you elaborate?

6:27 What does it mean to have a patent of a codec?

6:29 Why is there many patents?

6:31 Uh imagine I'm doing something where I'm going

6:34 to instead of doing blocks which are square, I'm going to do rectangles, right?

6:39 Also, every idea Yes.

6:41 somebody patents it.

6:42 Yes.

6:44 Oh, man.

6:46 Yes.

6:46 Oh, man.

6:48 People and their How many lawyers there are?

6:51 I mean, it pays for a lot of lawyers, right?

6:52 Like The biggest issue is not the following, right?

6:54 Because at time of H.264, the patents were, let's call it like sane.

7:00 But there was there was so much money in that that for HEVC

7:05 a lot there were a ton of things that were pushed inside the specification,

7:10 which are not useful in 99.9% of the time,

7:14 but so just one could add a patent on it.

7:17 And so it became that for HEVC licensing,

7:20 there was MPEG LA plus another patent pool called HEVC Advance

7:25 plus I think was Nokia was outside of the patent pool.

7:29 of them are outside and some other ones

7:31 And so it was impossible to license, right?

7:33 And I think that several months ago HP decided that they were going to remove

7:38 support from HEVC in their Windows laptops because

7:42 the the cost was increasing of those patterns.

7:46 And it arrived where a point where and there was uncapped bad.

7:51 And so for YouTube or Netflix, we could talk about hundreds of millions

7:56 of dollars of licensing for patterns per year.

8:00 And they said, "You know what?

8:01 I don't want to pay [laughter] millions per year, you know,

8:03 I could create my own codec." And this is what they did.

8:06 And so that's why we have the Open Media Alliance Alliance for Open

8:10 Media where we are part of that is that created AV1 and creates AV2.

8:15 We create also audio codecs.

8:17 But yes, so the main difference would be that.

8:21 And because you need to work around the patterns

8:23 or go do something that are not patented.

8:27 A lot of things are different, right?

8:28 The basing things that were done in a back

8:31 to 30 years ago are of course out of patterns,

8:34 but so for example, there is things like golden frame,

8:38 S frame, or or different type of all patented ideas.

8:43 Yeah, no, it's I I can't believe it's not butter.

8:45 I can't believe it's not a B frame.

8:47 What does that mean?

8:47 I mean, it's kind of what it is in some ways.

8:49 It's like a Oh, so it's a different variant of a B frame.

8:52 Yeah, that's to try and sidestep uh things like that.

8:55 And so you need to have double creativity, right?

8:58 Creativity in terms of how being more efficient,

9:01 but creativity of being sure that you don't infringe existing patterns.

9:05 And so for example, VVC is has all the patterns of VVC plus new ones, right?

9:11 So while AV2 tries to be as royalty-free as possible.

9:15 To what degree does FFmpeg and VLC have to think about this kind of stuff?

9:19 We don't.

9:20 And one of the reasons why VLC was

9:22 in France is that France rejects software patents.

9:26 So most of those patterns are illegal in France.

9:30 Because I'm once made the compu- the the calculus that if

9:35 I had to pay all the licensing fee for VLC,

9:38 I needed to pay more than 200 euros per user, or the same in dollars.

9:43 Um, but most of those patterns are invalid in Europe

9:47 because those are called It's basically mathematical patterns or idea patterns,

9:52 and they are not valid in Europe.