How live streaming works: The challenges of low latency video streaming explained | Lex Fridman

0:02 I would love to sort of uh zoom in and and and talk a little bit

0:07 more about the distinction between kind of downloading

0:11 a file and watching it offline versus streaming.

0:14 So the the complexities, the challenges of streaming.

0:17 Is there something we could say about what it takes to uh stream files?

0:23 We've been talking about codecs and I think a lot of that implies

0:28 encoding and decoding uh without the having to communicate over the network.

0:34 Sure.

0:36 Sure.

0:35 Uh so can you can you elaborate like

0:37 what's required to do over the network stuff?

0:39 Yeah, but it is less complex than it

0:42 seems compared to everything that we've talked about.

0:44 Um especially because the most complex thing

0:47 is not about streaming in terms of um

0:51 uh streaming services but it was was

0:54 what was done to actually broadcast through satellites.

0:57 Um because in in most of the modern uh

1:01 broadcasting services you can pose and you can go on.

1:05 But when you're sending live streaming

1:07 whether it's broadcast or live for streaming

1:09 services which are live this is much more difficult because you need to encode

1:14 in real time you when you go on a satellite you have

1:17 a specific size of the link right you cannot have a burst of bandwidths

1:22 even for a second right because you don't have the space

1:25 for that in your your total file however

1:28 there is different types of challenges which are

1:31 interesting challenges but I think they are less complex than the one we've

1:34 seen with um late '9s and early

1:37 2000s about broadcasting and streaming through satellite

1:40 they're different they're control systems challenges whereas whereas

1:43 some are more mathematical I think there's a difference

1:46 in the streaming world what you have is

1:47 called what we call adaptive streaming because the difficulty

1:51 and it's not really a video problem it's mostly a CDN problem is that you might

1:55 have too many people watching the same thing

1:56 at the same time and it's a congestion of the network right so your player um

2:02 has difficulty downloading things fast enough to play them.

2:06 So what happens is that locally the player

2:09 is going to read a lower resolution of it.

2:14 Um but there are some very clever algorithms to do

2:19 that but most of it is quite basics to be honest.

2:23 Even the buffering side is pretty basic.

2:25 Yeah.

2:25 Yeah, you you you start to download a a segment,

2:28 what we call a segment, and then you you time, right?

2:30 And if you if it takes more than 50% of the time to download a segment,

2:35 you go down to, right?

2:36 And the difficulty is more about when do you go up in bandwidth in quality,

2:41 but this is not very complex to do.

2:44 When you encode, you're going to encode seven resolutions, right?

2:47 And and you're going to give the bit rate.

2:50 Um, the difficulty is to have your encoder gives the same bit rate,

2:53 but it's not as strict as it used to be.

2:57 So, uh, probably YouTube has to figure out

2:59 how the human psychology side of that, like

3:03 how pissed off do you get when it's like very low bit rate and uh,

3:11 how long should it wait before it increases

3:13 the bit rate even though the connection is better?

3:16 because maybe the changes in the bit

3:19 rate is what like affects you psychologically.

3:22 I I think actually the interesting one is the audio

3:24 that you can kind of notice when they move from um full fat AAC to the um there

3:31 are compressed versions of AAC that use spectral band replication.

3:33 You can kind of see it goes a bit tiny and that up and down is very jarring.

3:37 The video side is a lot smoother and there's less notice.

3:41 It's really the audio.

3:42 You can you can definitely you can definitely feel it from when

3:44 it's moved you from a different audio profile to one or the other.

3:47 I don't know.

3:48 We're surprisingly tolerant at skipping audio glitches.

3:51 I I'm surprised people I know who are

3:53 not video engineers how tolerant they are how

3:56 tolerant they are to watching sports at 30

3:59 fps for example whereas it should really be 60.

4:02 The world is a lot more tolerant to that.

4:04 But audio people are very there it's an immediate feedback mechanism of oh

4:08 if you hear a glitch you realize it directly.

4:10 Yeah, I get to fully realize that.

4:12 I suppose one of the things I'm afraid of when I

4:14 listen to audio more and more that I get to notice every

4:17 single tiny detail and that you can oversess when uh people people

4:22 in general are able to kind of kind of blur their consumption.

4:27 They they can they can look past certain imperfections.

4:31 But then when you combine like um an event that is for example a sport

4:37 event that is probably going through satellite

4:39 or somewhere else and goes to a central place

4:42 for encoding and then you need to encode

4:45 this all the resolution you in real time

4:47 you don't have time for QA you need to push that to CDNs you need to add

4:52 probably DRM uh protection you need to have

4:55 that over a ton of different um devices

4:58 then yes it is complex X um but and also like you're in the web browser

5:06 or in very much different devices that you use for television where you had like

5:10 a a defined setup box or cable box

5:13 that that you know where you control end to end.

5:16 Um so it's a challenge but it's less I think the networking part uh while

5:22 you agree to have 10 20 seconds of latency I don't think this is very difficult.

5:28 Speaking of networking and latency.

5:31 So your new effort as we mentioned is

5:33 Kyber which is uh aimed at ultra low latency

5:39 as you say every millisecond counts and uh you're

5:43 applying that to remote control machines like robots, drones, computers.

5:47 Can you tell me about it?

5:48 Sure.

5:48 Um if you start from where we used to be, right?

5:52 You used to use ffmpeg to encode files, right?

5:55 And then we used FFmpeg and VC to encode in streaming services, right?

6:01 And then you need to go lower and lower.

6:04 And the question was where up to where we can can we go?

6:08 Um, and this question is very important because there are many use cases

6:12 where you need to be fast and it's when you have feedback interaction, right?

6:18 We're not just listening to something, you're actually controlling it, right?

6:21 because and that's the biggest difference that compared to what we've done

6:25 so far is that I I need video to have a feedback

6:29 on something that is happening live whether it's a drone flying

6:32 whether it's um controlling a humanoid

6:36 robots from distance whether it's controlling

6:38 a rover whether it's playing a video game in the cloud gaming

6:42 because this is um what I did on a previous job right

6:46 I was CTO of a cloud gaming startup um and this is

6:50 an very interesting topic because you push to the limits the network.

6:55 You need to be to care not about the quality

6:59 like we've done on video and we've talked about with x264.

7:03 You care about latency because a milliseconds

7:07 is meaningful when you're controlling a car, right?

7:11 For well you we've you've seen you've used Whimos, right?

7:14 when whimos don't work and that happens even if 1%

7:18 of the time there is someone that is basically remote controlling

7:21 that um and this is exactly the stuff that we're

7:25 building it's a really an SDK um platform uh to do

7:30 end to end control of machines so this comes up

7:35 quite a lot in a lot of different context in robotics

7:37 so obviously teley operation teley op is becoming more and more

7:40 important and including for training uh robots uh via machine learning.

7:48 Yes.

7:49 And what we do is a bit different from any

7:51 everyone else is that we take only one socket,

7:54 one connection which is a quick uh protocol based on UDP.

7:58 Um which is interesting because it's done for low latency.

8:02 It doesn't have two of the what we call

8:03 the TCP end ofline problem and HTTP end offline problem.

8:07 It's safer by default but on the same

8:09 wire we send multiple streams like multiple track.

8:12 We send audio, we send in video, but we also send the comments, right?

8:16 Uh mouse, keyboard, gamepad and so on.

8:19 And we do that while maintaining coherence, right?

8:22 Synchronization.

8:24 Because what people don't realize is that all the clocks actually drift.

8:29 And when you're controlling a robot, a robot is going to have like two cameras,

8:33 five cameras, 10 cameras, a ton of captors, GPS, and so on.

8:37 Um, and if you want to train correctly your robotic AI model,

8:40 you need to have all those that are in sync and currents.

8:44 And what we've done, and it's all the stuff that we learn on VC

8:48 in broadcast in real time and impact ts that kons know well,

8:52 is that we account for clock drifting.

8:55 And so when I record a kyber stream, a robot,

8:59 I am sure that it's going to be predictive in the way you played back.

9:04 And so when you're going to do recording and training of your AI model,

9:08 you need to be sure that every time you retrain based on the data,

9:12 the data is going to stay coherent and clocks actually drift.

9:17 Like the existing solution works with one camera.

9:20 Once you're going to a five or seven, it's more complex.

9:23 So you want to make sure that the visual

9:27 snapshot perfectly matches the time it actually happened.

9:31 Exactly.

9:31 And also if you're going to control right I do something on robot I

9:35 need to be sure that it is actually happening at that precise time right

9:39 and so we have on the the server which would be a robot

9:42 a time of like rettime stamping mechanism

9:45 accounting for clock drift for that right so

9:47 that's one of the use case um of kyber to to control robots um

9:52 I also see like remote drones remote

9:55 uh whether it's defense or non-defense remote

9:58 cars remote submarines there There's many

10:01 places in industry or remote surgery where

10:06 the expert cannot go everywhere the machine

10:09 is because either dangerous or it's too costly.

10:11 Right?

10:11 So you you allow people to have machines next to you, right?

10:17 The goal of Kyber is to make distance disappear um

10:20 because it's either projection of skills or projection of power, right?

10:24 So imagine we we all like you've seen the meta reban and everyone else, right?

10:30 you need to stream there, right?

10:31 Because you're not going to run anything over there, right?

10:33 So, you need GPU power, whether it's on a cloud, on a phone, to stream that.

10:36 And so, all of these use cases needs to be not about extremely low latency,

10:41 but real time latency for video.

10:44 And so, that means you need we're toing with the encoders

10:48 so that the encoders encoder frame in for milliseconds.

10:52 and and Kiran with his company also goes under

10:55 those type of license of of latency because you

10:59 need to optimize at max the local latency right

11:05 because it's the decoder the encoder um and so

11:09 on um because this time is going to be

11:12 added to your networking time um so and it's not

11:16 just about low latency it's also about like reliability

11:19 we do clever things like uh forward error correction, right?

11:23 So forward error correction is you over transmit a bit of data, right?

11:28 A few percent um and while over transmit

11:31 you're allowed to lose some packets because all

11:34 of that is very difficult over a internet network

11:39 uh where you're going to do things very far away.

11:41 Um and if you check that all packets are delivered, you add a ton of latency.

11:47 If you don't want latency,

11:48 what we do is that we over transmit some data that you can retrans

11:53 reconstruct on the client side when there is um things that are broken, right?

11:57 So um and we um a few a few days weeks ago we were doing the demo

12:02 around Las Vegas for the CES about we had a a rover that is fully 3D printed.

12:08 It's very simple.

12:09 It's a car, right?

12:10 It's a small car with a um a telescopic

12:13 arm and it was actually controlled from France, right?

12:16 And the the video uh was uh with a webcam in a very small server, right?

12:21 A small a small PCB was basically running and send

12:25 that to someone that is on the other side of the planet.

12:28 Uh and so there is so many use cases.

12:31 You can also think about having AI who

12:33 are going to control many drones and so on.

12:36 And the technically we need to be amazing in video.

12:39 We need to be amazing at networking.

12:41 We need to care about any milliseconds in networking, in encoding time,

12:45 in decoding time and also you need to integrate very low level.

12:49 So sync everything together well.

12:51 But how like what kind of latency can

12:53 you get to like what when you say milliseconds, what what's the goal?

12:57 So my goal is 4 milliseconds glass to glass latency.

13:01 Um what's glass to glass mean?

13:02 So it's easy, right?

13:03 You have a computer which is running a program, right?

13:06 Probably a video game.

13:07 And this one is actually running, right?

13:09 It could be it's an example of a robot, right?

13:13 And you have the replicate that is done to the network and and you want

13:19 if you take a a 1,00 Hz camera,

13:22 you can take a picture and you want that to be at 4 milliseconds.

13:25 4 millconds means 240 Hz, right?

13:29 Yes.

13:29 Not um so far we we achieve um 7 milliseconds

13:34 from a Windows to Windows or window to Mac.

13:37 Um and if you look in the timing most

13:41 there is around 3.5 milliseconds inside the Nvidia uh hardware

13:46 encoder and around 2 milliseconds on the Intel decoder

13:50 right so like the encoder plus the decoder is already

13:53 6 milliseconds right so in order to go down

13:56 we need either to have some other type of codecs

14:00 um or some better encoder that are faster uh

14:04 but 4 milliseconds is would be the growl That's pretty nuts.

14:08 I love it though.

14:09 I I don't think anyone's ever achieved that, right?

14:12 That's fast.

14:12 You can achieve that with custom hardware, with SDI, with professional hardware.

14:18 But I want that to work over the internet.

14:22 I want the to work with any robots where you're going

14:25 to have a small Jetson Nano in it or or N150, right?

14:29 I want that because there is going to be millions of robots

14:33 or drones are just rolling robots or flying robots or or swimming robots, right?

14:38 It's just you a machine that you control and in order

14:42 either you need to teleoperate them or when everything will be fully autonomous,

14:48 you need to te observe them, right?

14:50 You need to check what's happening.

14:52 Yeah.

14:52 And in my view in the future like all

14:55 those remote cars will be teleobserv observed by an AI

15:00 model which is just going to say well

15:02 everything is gone good and when it's not good

15:04 say hey there is a problem and then you

15:06 have an operator right and this is going to be

15:08 about safety right when you have your humanoid taking

15:10 care of your grandma or my grandma I want

15:13 to be sure that everything goes well and I'm

15:15 not in those type of horrible scenarios where the robot

15:18 is dangerous or when I'm driving I I want like the car to to stop when it should

15:24 stop and if needed someone takes care of that right

15:27 and so there is so many cases scenarios about real time and so the goal of Kyber

15:32 is to make real time control of machine distance disappear

15:38 it's it's incredible and some of the same technology some of the same

15:41 ideas we've been talking about is is connected to what you're doing

15:45 and for for me it's amazingly challenging

15:47 right because I would say that on video

15:49 I'm doing okay but networking I have so much more to learn, right?

15:53 It's uh um about like congestion protocols, bit rate adaptation in real time.

15:59 Um but it's it's quite funny and and so I created this project

16:03 and and we we have fundraised in the US of course but it's open source, right?

16:08 This is important, right?

16:09 Like we've not said that, right?

16:10 But everything on Kyber is open source.

16:13 So how do you make money?

16:14 It's a dual license commercial and AGPL, right?

16:17 If you remember what you said about about uh licenses,

16:21 uh basically if you want to use Skyber in your product,

16:24 you must have your full product open source.

16:27 If you want to use this amazing technology but not open source,

16:31 you pay the commercial license, right?

16:33 So the small people or the the the hobbyist and the the very

16:37 small guys who want to do that, they can use the technology,

16:39 they build something that is open source and cool.

16:41 That's awesome.

16:42 And if you're a large company, you're going to have the support,

16:45 all the IP, the right modification and so on.

16:48 So, um yeah, it's really cool.

16:51 And and and also I'm building robots and I love that, right?

16:54 Like like the the rover we have is 3D printed.

16:58 Um we are finishing a demo where it's an actual wing, right?

17:01 Like a type of drone wings that is also fully 3D printed.

17:05 Um we are trying to do a a [clears throat] sailboat that is 3D printed.

17:10 Uh, and and and we'll work on some humanoids.

17:13 Of course, they're not going to be very good robots, right?

17:15 It's not our job, but we're here for everyone to make robots.

17:19 Cool.

17:20 Ah, you're talking to the right guy.

17:21 I love robots.

17:22 There's a bunch of them upstairs.

17:24 Uh, and tell is going to be really, really important,

17:27 especially as the number of robots goes across the world.

17:29 So, 100%