Digital Signal Processing With Audio Data - Computerphile

0:00 Well, I want to talk to you about DSP.

0:02 Mainly because when I talk about audio engineering on my channel,

0:05 most of my subscribers really run a mile whenever I touch any code.

0:11 They kind of go, I like audio engineering, but I don't want to know about code.

0:14 So, I've come to a place where I hope a few lines of C might be appreciated.

0:19 Do you think that'll be the case?

0:21 I'm hoping so, too.

0:22 I'm hoping so, too.

0:25 Whether you're dealing with sound,

0:28 which is the examples I'm going to give you today,

0:30 or whether you're dealing with any kind of time series data,

0:33 it could be financial data, it could be temperature in another, you know,

0:37 in a certain place and it was fluctuating,

0:40 you end up with like low frequency and high frequency information within that.

0:43 And sometimes it's important to do things to that data.

0:47 And um the thing that I think most people are familiar

0:50 with is having a waveform and it actually producing something you can hear.

0:54 Your ears are very sensitive.

0:55 So all the examples I'm going to give

0:56 you here surprisingly are going to be musically related.

0:59 But this is applicable in so many other places.

1:03 So the first thing I want to talk about really is equalization.

1:07 Do you know what equalization is?

1:09 I remember having an equalizer on a stereo where

1:11 you turn up the bass or turn down the bass.

1:13 Yeah.

1:13 Turn up the bass treble or if you were posh back in the 80s.

1:17 I'm showing my age now,

1:18 but you'd have a graphic EQ all those little switches and you turn that one.

1:23 So, before we get into any coding or anything, I'm just going to play something,

1:27 a drum kit through, and I'm just going to turn up the treble,

1:30 turn up the bass, turn up the middle,

1:32 just so that you can intuitively hear what an EQ does

1:36 and what I'm talking about before we get into some code.

1:38 My friend Yan played these drums.

1:44 So, this is just the drum kit as it would be.

1:46 If I wanted to turn up the treble like you would in your car

1:48 or on a hi-fi [music] that that gets turned up [music] or you can turn it off,

1:59 make it sound really dull.

2:03 The same will be true of the bass.

2:05 [music] That's distorting nicely.

2:11 So [music] you get end up with more bass there or less bass.

2:16 Sounds very thin.

2:18 If you're watching on a phone,

2:19 you probably can't tell any difference between those examples,

2:21 but um doesn't matter.

2:23 Imagine it in your mind.

2:24 And then the midfrequencies.

2:31 So audio is made up of a whole frequency spectrum with really long waveforms

2:36 and really small waveforms and uh they all

2:38 come together with different frequencies and different amplitudes.

2:41 The longer frequencies are the bass frequencies,

2:44 what we consider like the bass guitar and the bass drum, stuff like that.

2:47 And then all the and all your hisses and your sibance is right up

2:51 at the top and all the stuff that we

2:52 hear really clearly sits in the middle somewhere.

2:56 And that makes the voice and the guitars and whatever

2:58 else you listen to, the crack of the snare drum.

3:01 But let's say we wanted to do some of that.

3:02 We don't have, you know, ProQ4 or equivalent EQ,

3:06 you know, you just want to be a like a professional.

3:10 ProQ4 is pretty much industry standard um industry standard

3:15 EQs that you know in in the box studio EQ.

3:18 But [clears throat] let's say you wanted to do

3:19 your own thing or let's say you had a piece of time series data that's um you

3:24 know as I say weather information or something like that

3:28 and you had it in a spreadsheet okay and you wanted to manipulate

3:31 that and you want to take out all the little high frequency stuff.

3:34 Well obviously you can do that with maths.

3:37 This is just all it is with DSP is

3:40 just numbers that represent each point on the curve.

3:43 So digital signal processing.

3:45 Digital signal processing is just maths.

3:47 Basically, you're just getting the points

3:49 on the wave and then we record those numbers.

3:51 We've covered that in a previous video.

3:53 Basically, we're taking those numbers in and we're giving a new number out

3:57 and whatever the new number is is dependent on the code that we write.

4:01 So, let's load a piece of software here called Plug-in Doctor.

4:05 So, for those of you that don't know,

4:07 I work within a thing called a DAW or a door,

4:10 a digital audio workstation that allows you to have lots of tracks of audio.

4:14 And then, while I've got all this nice stuff behind me that's

4:18 real physical pieces of equipment that you can plug in to do audio,

4:21 uh, you can also have digital equivalent to that.

4:24 You have plugins that you can put over things.

4:26 So, if you want something to have a reverb sound like

4:28 it's in a massive room or you want to distort something,

4:31 there's a plugin to do kind of anything you want.

4:33 And at the a fundamental level, all it's really doing is taking the sample

4:38 number and changing it somehow and overwriting the place

4:44 in memory where the last uh the last sample was so that it changed it somehow.

4:48 And and digital plugins are trying to mimic a lot of the time.

4:51 They're just trying to mimic what's on the outside here.

4:54 Now, there's a great plugin,

4:56 one of my favorite plugins of all time, which is called Formula VST.

5:00 VST stands for virtual studio technology.

5:03 Um, and basically what it is, if I load it up here,

5:07 so if I load an instance of formula into plug-in doctor,

5:11 which is like an oscilloscope for DSP, if I load up formula,

5:16 it gives me something that is akin to some

5:19 sort of notepad or VS Code type window.

5:23 Due to my age and eyesight, I'm going to up the text size there.

5:27 Then as you can see it just has a function called formula main

5:30 which basically sets the output to the input and then returns the output.

5:34 This is just C.

5:35 So this plugin here is doing absolutely nothing on plug-in doctor.

5:39 If I separate those two so you can see them both on the screen.

5:43 You can see I've got a minus 6 dB signal going in.

5:46 So that's just a volume effect.

5:48 That's just the volume of what's called a direct spike.

5:50 So it's it's just a single one sample in a sea of zeros.

5:55 It's the way you do impulse responses.

5:57 You send it a single spike and then whatever happens to that will

5:59 draw a waveform and and at the moment it's doing nothing whatsoever.

6:04 So if we were to put an EQ in here like that ProQ4.

6:08 So I'll I'll load that up.

6:10 So you can see as I change things on the EQ,

6:13 it's also showing me in plug-in doctor what it's doing.

6:16 So it's a really nice way when we're coding something to be able

6:19 to see what's going on with our code rather than just hearing it.

6:23 I mean sometimes it's good to use your ears but sometimes it's good

6:27 to check that the maths is actually doing what it's meant to be doing.

6:30 So at the moment formula is doing nothing but let's say we wanted to take

6:36 the the high frequency data out which I think for not necessarily for audio

6:41 but for a lot of other applications particularly

6:43 if you were dealing with financial data or weather

6:45 data or something like that the minuteby minute

6:48 or hourby hour fluctuations might not be important.

6:51 Actually, what you want is a smoother curve.

6:55 Okay.

6:55 Well, how would you do that?

6:56 Have a guess how you do that.

6:57 It's averaging, isn't it?

6:58 Just averaging.

6:59 Yeah.

7:00 So, let's let's start.

7:01 Let's knock up a simple averaging formula.

7:04 So, outside of that function, I'm just going to do a float of last.

7:08 So, we keep the last.

7:10 I'm going to set that to zero.

7:12 And then [clears throat] we've got the output equaling the input there.

7:17 And then I'm going to basically say there output equals output plus last.

7:24 So we're just going to put them together.

7:25 And if I put all of those in brackets and then

7:28 we just divide that by two because we're adding two things together.

7:32 We divide it by two.

7:33 That's going to be the average.

7:34 And then I want to record that for next time.

7:37 So we're going to say last equals output.

7:41 And if I now hit play on that.

7:43 Ah, we've lost some treble.

7:46 Oh, so the highest frequency is on the right.

7:48 So that's the highest frequency is and that's coming down

7:51 as we saw before with the drums.

7:53 So I don't think anyone's going to be impressed with that.

7:55 A couple of lines of code and we've created a moving average.

7:58 But if you were doing that to weather data,

8:01 you'd be taking out all of those little

8:04 day-to-day things and smoothing out and getting more

8:06 of a a long-term sort of projection and taking

8:09 out tiny minor fluctuations and outliers and stuff like that.

8:13 Um, but that's exactly what you're doing with audio

8:15 when you want to turn the treble down.

8:17 Let's say the symbols on the drums.

8:18 Okay, let's say I don't want to do that.

8:21 Let's say I actually want to keep all the treble

8:23 and I want to get rid of the bass

8:25 because what I'm interested in is this real high

8:27 frequency trading and I don't care about the long-term trends.

8:30 I'd care about minuteto- minute stuff.

8:32 Okay.

8:33 Well, if we've taken all the treble off that, what if

8:36 we take that low stuff that we've kept that low frequency stuff,

8:40 what if we take that from the original single?

8:42 So we do input minus output.

8:45 Lo and behold, oh, we're left with some treble.

8:48 Not perfect.

8:48 There's a whole set of formulas called by quad formulas

8:51 which which do this in a much more refined way,

8:54 which we're going to get to in a minute.

8:55 But this is the kind of the basic principles.

8:57 Now you can see how with small building blocks,

9:00 like everything in programming and everything in life,

9:03 with small building blocks, you can build up to effectively do what I

9:06 was doing to those drums where you're boosting frequencies,

9:08 you're cutting, you have different high and low

9:10 points and all that kind of stuff.

9:12 So let's let's see.

9:13 Obviously, this is pretty useless at the moment.

9:15 If I get rid of um that and we go back

9:16 to the uh the lowass because we're allowing the low frequencies to pass through.

9:22 Let's say we want to be able to move that up and down.

9:25 Well, Formula, the Formula Vst plugin has these kind

9:28 of knobs that are named luckily knob one, knob two, knob three, blah blah blah.

9:35 So, what would be your intuition if I wanted to move up and down that?

9:41 It's the size of the average surely.

9:43 Is it the size of the average?

9:44 No.

9:45 So, it could that's that's one way of looking at it.

9:47 So, so if you record more points,

9:49 you might be able to average over a further amount of time.

9:51 That's um that's not what I was getting at actually.

9:55 I was thinking of the ratio between the two.

9:58 So for example, at the moment we're mixing

10:00 output and last together in the same ratio.

10:03 Let's say I want to use knob one as my ratio.

10:06 So if it's 0.5, which is the default setting,

10:09 then we're going to get half of output and half of last.

10:12 So let's just do knob_1 multiplied by output plus 1

10:23 minus knob_1 because we want to get the opposite times last.

10:29 Because we've got that knob there,

10:30 if they're both 0.5, it's keeping the balance.

10:32 We don't have to divide by two anymore

10:34 because it's always going to be half of one,

10:36 half the other, or a bit more of one, and a bit less of the other.

10:38 So all we've done there is basically created a uh

10:41 a ratio between the the current signal and the last sample

10:47 which is controlled by that knob which is controlled by that knob.

10:49 So at the moment that won't change anything.

10:51 Oh, what have I done there?

10:52 Output one error note to match this.

10:54 Have I got um ah I've put a bracket in there.

10:57 That's [gasps] one too many brackets.

10:59 Right.

11:00 So nothing's changed there because at the moment it's 50/50.

11:02 But if we look at knob one now and as I move

11:06 it you can see that as I get down to the bottom yes 0.01 it will die at zero.

11:12 That's now changing the frequency point where the low pass is happening.

11:16 So if I copy that code we can actually

11:19 listen to what this sounds like and adjust knob one.

11:21 So if I turn it up there it won't be doing anything much.

11:24 And uh this is the drums.

11:30 [music] So yeah, it's cutting up.

11:35 So all the treles coming off it.

11:36 As we turn it down, you can actually see

11:38 that if I bring up here the channel settings,

11:41 you can actually see the the frequencies there and how much less top end.

11:47 Now there is.

11:48 You can see there's not a lot of top end there.

11:50 And if we turn it up to there, all those treble frequencies come back.

11:54 All we've done there is just done some averaging

11:57 and we've changed the ratio on how the averaging works.

12:00 Okay, this is not complicated maths yet.

12:04 I wouldn't quite call it a functional EQ,

12:06 but it's not that many more steps to get to a functional EQ.

12:11 So, if I bring Plugging Doctor back up and we'll do a Blue Peter.

12:15 For the American viewers, you won't get that, but we'll do a Blue Peter.

12:18 This is one that I prepared earlier.

12:21 Um, so I'm just going to do a one pole, what's called a one pole lowass filter.

12:26 This is actually the code for a much better one pole lowass filter.

12:31 So I've got the sample rate in there.

12:33 I know it says float SR and I'm

12:34 sure you were thinking that was your initials, Sean.

12:37 Oh, it's all about me.

12:37 But it's all about you, isn't it?

12:39 But no, that's 48,000.

12:40 So I'm giving it that and I'm giving it my minimum frequency,

12:43 my maximum frequency.

12:44 We're then doing this calculation here where

12:46 it's the frequency cutff the actual frequency we

12:49 want is basically it's a it's a calculation

12:52 involving powers to the value of where

12:55 knob one is and then G here is basically the constant that is the equivalent

13:01 of the knob one that says the the ratio you've got the one minus

13:04 and the you got the ratio between the two and basically that's doing some

13:07 maths that's involving pi if you look

13:10 at it it's basically calculating the frequency cutff

13:13 and meaning that you're not getting loads of play in the top of the knob

13:16 and then right between 0.01 and 0.02 it's doing everything in the low end there.

13:20 It's kind of scaling it actually smoothing it out in Yeah.

13:24 So it it kind of fits the the sounds lo sounds logarithmic it is.

13:28 Yes.

13:28 Exactly.

13:29 And it's it's accounting for that.

13:30 So now if we go to to knob one here

13:33 and move it up and down there's there's a lot more granularity.

13:37 You see we're down to like 0.2 there and we're already fairly low down.

13:41 And then at higher up, you know,

13:42 it it it's it's much more much more usable like that.

13:47 So effectively, it's all of that's doing is just scaling that knob.

13:51 So, but it's still doing the same principle.

13:54 It's still taking the last and the current sample,

13:58 mixing them together somehow in a proportion that makes

14:01 sense to where you want your frequency cut off.

14:03 So, doing the posh version is actually not that much different.

14:06 All EQ is doing there is is just a moving average.

14:10 And then from there, again, one we prepared earlier.

14:13 So I can go to a low and high pass.

14:16 So if I load this, I've now got the low

14:18 pass and the highpass filter in the same code.

14:21 So I've literally just copied and pasted the code, made two of them,

14:24 and effectively done a low and a highp pass on on two different knobs.

14:28 So then on this, for example, I can do that.

14:32 So we're getting a bit of low and a bit of high.

14:35 So we're getting that that notched.

14:37 We're keeping all the low frequency data.

14:39 We keep all the high frequency data.

14:40 And then of course I can mix some of that back in.

14:44 I can mix using the dry wet that's actually built in there.

14:47 Dry wet just for people who are not dry.

14:48 A dry wet mix is basically the dry signal is

14:50 your original signal and the wet signal is what what you've

14:54 what the effect whatever the effects are doing.

14:56 And if you mix between the two again you're you're doing a a ratio between

15:02 the the the original clean signal and the and the processed signal.

15:05 So if I do that, you can see we can make that graph change there.

15:09 So this is a real world implementation.

15:12 You can see so if we do highp pass plus low pass minus the input,

15:16 we then get just the mid-frequencies.

15:18 We've isolated all the all the mid-frequencies and got rid of the low,

15:21 got rid of the high.

15:22 the fact that you can play with it in Formula VT and you could load up any DAW,

15:26 any workstation, put any song you want in there, uh,

15:30 or a drum track or a sample you download, a vocal, whatever you want,

15:33 and load that plugin over it and just start coding

15:35 in C and see what see what things do to the sound.

15:39 Obviously, we're here in a studio with all the equipment

15:42 and all this software and all the rest of it,

15:43 but can people do this, you know, for free or cheap?

15:47 Yeah, pretty much for free.

15:48 So, there's there's plenty of uh Formula is

15:51 free and open source as a as a plug-in.

15:54 Um I haven't used Audacity in so many years.

15:58 I know that's free and open source, but I'm sure someone could load a song

16:01 in there and put Formula VST over it in there.

16:04 There's a DAW called Reaper that um gives you a nag

16:08 screen if you don't pay for it after 60 days, but it doesn't stop you using it.

16:12 And you can load Formula VST into that and you could play about with audio.

16:17 There's so many ways of doing it.

16:18 Anything where you can load an audio file

16:21 into it and and put a plugin over it will

16:24 will do the job and and you can download

16:26 Formula VST and just just see what happens really.

16:29 Just start playing with things.

16:38 I'll load up the picture that you sent me.

16:42 So, here's the picture that you sent me.

16:44 And the first thing I wanted to do was

16:45 get rid of the Adobe Audition stuff around it.

16:49 So I put it into Photoshop and um I kind of got

16:52 the what the color of the green of the grid behind and basically