Fixing VFX’s Longest Problem using CorridorKey w/ Niko, Corridor Crew | NAB 2026 Puget Systems Booth

0:00 This is really cool to see so many people turning out here.

0:03 Uh look, I got a replica of the couch here and everything.

0:06 Oh yeah,

0:07 I've seen it.

0:09 Look at this.

0:09 We got a a galaxy in splat of the studio behind us here.

0:12 Uh which is pretty neat, too.

0:14 Um good to see some familiar faces here.

0:16 It's uh super cool.

0:17 Uh wow, what a turnout.

0:18 Well, I'm going to teach you guys about quarter key today.

0:21 Uh, specifically I'm gonna be talking about uh I'm gonna

0:25 be talking about how it was made, how it works,

0:28 how I hope it's going to evolve and change and why it was made,

0:32 all that kind of stuff.

0:32 Just going to tell you a story here.

0:33 So, um, yeah, green screens.

0:37 Uh, we use a lot of green screens,

0:39 and I don't like doing green screens at this point.

0:44 There's always a little bit of something that goes wrong.

0:47 One part looks perfect, it makes other part look bad.

0:48 and do you fix another part that looks bad?

0:50 It makes the part look look perfect look bad and it's a headache

0:54 and I just I feel like it should be easier by now.

0:57 So that was, you know, a thing I thought to myself years ago.

1:02 Um started getting into AI, machine learning,

1:04 seeing the different tools and I was like you know at some point

1:08 you know somebody's going to make something that kind of does the job

1:10 of green screens but you know with machine learning and kind of fix

1:14 this and a year passes and you know it doesn't get fixed.

1:18 familiar passes and you get things like uh you know segment anything,

1:22 magic mask, rotor brush, great alpha tools, great masking tools,

1:26 but still nothing that quite does what green screening does, which is to say,

1:30 you know, you have your foreground subject separated

1:32 not just from the background with the transparency channel,

1:35 but also with those green colors unmixed from those semi-transparent pixels,

1:38 which is a key to making all this stuff work.

1:40 you know, motion blur, blurry edges, uh you know, things like hair,

1:45 all that stuff relies on this ability

1:46 to kind of separate the green from the foreground.

1:48 Now, of course, you can do that with professional tools.

1:51 You know, you can use delta key or IBK like key light.

1:54 You can get there with professional tools,

1:56 but the whole idea is that that becomes a kind of an arduous

1:59 process of like rotoscoping effectively when you

2:01 want to make it look really good.

2:03 And you we have this production coming up.

2:05 We're doing Son of a Dungeon season 3, uh Dungeons and Dragons show.

2:08 And you know, one of the the key concepts

2:11 of this this series is that we play the game,

2:14 we play a Dungeons and Dragons game,

2:15 but then we eventually go back after it's edited and we

2:17 refilm scenes cinematically in the world in the game board that we've,

2:21 you know, played the game on.

2:23 And that, of course, requires having a lot of stuff on a green screen.

2:26 And I didn't want to sit there and spend, you know,

2:29 another couple months struggling with green screen keying

2:32 in the year 2026 when I can talk to my phone.

2:35 I can have my car drive itself,

2:38 but why am I still pulling a green screen key in this day and age?

2:41 Um, and so I had been, you know, waiting for something to come along.

2:44 And once again, fantastic automatting tools are coming out,

2:47 but none of them quite took that extra

2:48 step for that whole foreground unmixing bit.

2:52 And eventually, I was like, I'm just going to give this a shot.

2:54 you know, I'm not I'm not a machine learning guy.

2:57 I, you know, I don't have any sort of degree

2:59 in like advanced neural network training or anything like that.

3:03 Um, but I'm just going to give it a shot.

3:05 I'm going to try and I'm going to sit down with, you know,

3:07 some large language models and have them hold my hand.

3:10 I'm going to work through it.

3:11 I'm going to ask questions to as many people

3:13 as I can and just go for it, you know.

3:16 Um, and so that's how I spent my Christmas of 2025

3:20 was uh sitting there and talking to Gemini a lot.

3:22 That was my entire plane ride back to Los Angeles from Minnesota

3:25 was talking with Gemini on my laptop and kind of just working it

3:28 out and giving a lot of research papers and kind of trying

3:30 to hone in on an idea of how I was going to do this.

3:33 And so it kind of came down to a core idea here,

3:36 which is I can generate synthetic data pretty easily these days.

3:40 Um, pretty much we all can, right?

3:41 It's you fire up Blender or Houdini or, you know, whatever you want,

3:46 make some sort of procedural system and start

3:48 rendering out objects on a green screen.

3:49 And one of the nice things that you can do

3:50 with that is you can get a perfect alpha channel,

3:52 you can get a perfect foreground separated from the background,

3:54 you get a perfect background, kick it all out to EXRs, floating point 16,

3:58 gives you a whole bunch of options here to then

4:00 do augmentations and stuff like that down the line.

4:02 Um, so I took that idea, that concept, shoveled that workload off onto the rest

4:07 of the crew and had them knock out renders,

4:10 hundreds of renders over the course of about two to three weeks.

4:12 Um, and so I had this data set now of, you know,

4:15 basically 200,000 frames, uh, roughly 500 plus shots.

4:21 And, you know, the thought was, okay, I have all this put together.

4:24 I have the data set, you know,

4:26 garbage in, garbage out, good stuff in, good stuff out.

4:29 You know, let's sit down and start training something.

4:31 Um, so I fired up uh Microsoft's imageet or yeah, imageet.

4:37 Um, and did a fine tune on that and it kind of kind of worked.

4:41 kind of look like garbage.

4:42 Definitely not good enough to be a professional tool,

4:44 but it was kind of working and it made me feel really encouraged.

4:47 So, I kept working on it from there.

4:49 And at this point, I started keeping the results

4:52 secret from the rest of the gang at Corridor.

4:54 So, I could get some good reactions for for the video we were making about this.

4:58 And of course, we have this interesting thing going on at Corridor

5:02 Corridor where we can uh we can take a passion project

5:05 or an idea we have and we can turn it

5:07 into a YouTube video to justify us spending the time working on it.

5:11 And of course, I was like, well,

5:12 we have this, you know, I'm working on this video here.

5:15 We're going to make a video about green screen tools,

5:17 and you know, maybe it'll get a couple hundred thousand views,

5:19 and, you know, it'll let me justify the the two months I

5:23 spent working on this tool to make Son of a Dungeon easier.

5:26 Um, and the the finale of this video was me doing the big reveals for the guys.

5:32 And I spent, you know,

5:36 I spent basically the two days before rendering these shots out.

5:39 I barely even got to look at them.

5:40 I had no idea if they were even going to work because we went

5:42 downstairs and we filmed the hardest shots to deal with on a green screen.

5:47 Jordan Allen drinking from a transparent blue cup and a transparent glass

5:50 cup with water in them with transparent clown sheets on his head.

5:56 just like the hardest stuff.

5:58 Um, and I processed it and showed it to them.

6:00 And one of my one of the gigs I wanted to do here,

6:03 so those shots that you're looking at right there,

6:05 those shots right there, those are actually keyed out shots using corridor key.

6:09 And I kept that secret from the guys.

6:11 I could do a nice little reveal here because I

6:14 was very surprised at the the results I was getting.

6:17 It was very encouraging.

6:19 Um, and as you can see,

6:21 they started to have the same reaction once they saw what was happening here.

6:25 Um, and there you have it.

6:27 And so if we keep going here, you'll see some of the other shots, uh, as well.

6:32 Jordan's losing it.

6:33 Ren's losing it.

6:35 Uh, because whatever we seeing, whatever we seeing here is that like,

6:37 hey, maybe maybe I don't have to spend all that time anymore.

6:42 Maybe there's some help, some hope here, you know, to, uh,

6:44 spend less time doing the technical setup and more

6:47 time doing the compositing and the, you know,

6:50 the framing and the the artful stuff, the stuff you kind of want to be doing.

6:56 Now, you know, motion blur, motion blur is a hassle.

6:59 Like, we always shoot high shutter speed on the green screen.

7:01 We always shoot deep focus.

7:02 We try to avoid blurry edges as much as possible.

7:05 But, you know, we're quickly discovering here

7:07 with some of these tests that that's,

7:09 you know, maybe something we don't have to worry about quite as much anymore.

7:12 Um, you know, that's uh it's pretty reassuring

7:15 when you can get a shot like that.

7:17 So, I was personally really surprised with the fact that this worked.

7:22 Um cuz to me this wasn't just about green screening.

7:26 This was also about the idea of inventing your own tools as a filmmaker.

7:31 Um we're hitting a new era here where you can code.

7:36 Anybody can code.

7:37 Anybody can you know put together a script.

7:39 And you know if we're all working with digital

7:41 media here like at some point I'm sure you've

7:43 dealt with the fact like oh I've got a thousand

7:45 files I need to do some routine thing too.

7:47 where it's like I'm as an artist, I'm going to go in and just deal with the same

7:51 wrote setup that I have to do every time.

7:55 And you've probably thought to yourself, oh,

7:56 it'd be great if somebody made this plugin or somebody made this tool.

8:00 And, you know, I've had that, you know, dream myself many, many times.

8:03 This is kind of me not just trying to solve green screening, but like,

8:07 can I make tools now as an artist who doesn't who's not a computer programmer?

8:12 Can I stop having to rely on software companies

8:14 to have to make the paintbrush that I want to use?

8:16 just going to sit down there and make the paintbrush myself.

8:19 And the experience was really really positive and really

8:23 reassuring and gives me a lot of I guess excitement

8:26 for things that can happen in the future here

8:29 um around any kind of tool that you might want.

8:32 So I thought it might be interesting to talk a little bit about the technical

8:36 details of what's going on inside of core or key and how it works.

8:40 Um so we're going to get a little technical

8:42 dense here but you know let's let's go for it.

8:45 So what I ended up doing to get those results is I

8:48 ended up training on a model that Meta trained initially called Hyra.

8:53 And Hyra is the the backhome model that works with segment anything.

8:58 Um I don't know if you guys ever use segment anything.

8:59 It's kind of your classic put a few dots on the subject you want

9:02 to keep and then the AI model will

9:05 figure out okay what's back on this foreground.

9:06 It'll give you a rough mask.

9:09 Um, at its core, what you're basically dealing with is a system

9:13 that can look at an image and identify what the pixel is.

9:16 So, you know, you heard the term chroma key.

9:17 You can always think of this like a concept

9:19 key where it's going to go and identify, is this a pixel of the background?

9:23 Is it a pixel of an actor?

9:24 Is a pixel of a prop?

9:25 What is it?

9:26 And little by little, the malt gets trained to be like, okay, well,

9:29 if this is identified as a background pixel,

9:31 you're going to spit out an alpha channel pixel that's black.

9:34 It's identified as a foreground pixel,

9:36 spit out an alpha channel pixel that's white.

9:38 Um, if it's a fully white pixel, spit out the same original color of that pixel

9:43 from the input image for the output image.

9:45 And if it's a semi-transparent pixel, do that unmixing here.

9:48 And that's where, you know, I relied on the data set to basically

9:51 train this model how to do physically accurate unmixing.

9:55 So, I had this hyra model.

9:57 I had the synthetic data that I made,

9:59 but you can't just say, here's my can't just say, is this working?

10:05 Oh, this is much better.

10:06 Now, I have something to do with my hands.

10:09 Okay, so the way this needed to work is I

10:13 need to teach the computer how to study these images,

10:17 how to tell if something is working and something is not working.

10:19 And that was actually the most challenging part of this entire process.

10:22 Um because you end up with this this system

10:25 of losses of of weights of saying like hey if

10:29 this error is big minimize this error but if

10:31 this error is small you can ignore it etc etc.

10:34 So, here's an idea of what it took to basically treat

10:37 teach the computer how to um how to put all this together.

10:40 So, Christian, if you wouldn't mind opening up

10:42 one of the uh the debug images here.

10:44 Maybe not the very first one, but like something from halfway through.

10:47 And these are pretty gigantic images here.

10:49 Um and let's go ahead and zoom in to the top left corner here.

10:54 All right.

10:55 So, the first thing I had to deal with is the fact that, you know,

10:57 I had a lot of frames.

10:58 I had 200,000 frames of data, but that's not enough.

11:01 you need effectively infinite variations.

11:05 So I did everything from uh separating the background from the foreground,

11:09 doing color adjustments to the background, hue,

11:11 brightness, contrast, gamma, all that kind of stuff.

11:14 Same thing with the foreground separately from each other.

11:16 Um we gave them uh you know adjustments like aspect ratio,

11:21 squash them, squished them, things like that.

11:23 Some you know even some mirroring and that was just the input image.

11:28 So an input image comes into the system but now it needs to do some comparisons.

11:32 So basically what you give this system is you give it two things.

11:35 You give it an alpha hint and you give it a uh just your input image.

11:40 So your input image would be what's whatever's on the green screen.

11:42 Your alpha hint is just a black and white mask

11:44 saying hey this is what I want you to keep.

11:47 This is what I don't want you to keep.

11:48 So for example if you have tracking

11:49 markers on your green screen that aren't green.

11:52 You still don't want to keep those even though they're not green.

11:54 So you can't just say hey keep things that aren't green.

11:56 That's not intelligent enough.

11:57 So the alpha hint was kind of like my way of saying,

11:59 "Hey, if it's not lit up in the alpha channel, right?

12:03 Even if it's not green, ignore it.

12:05 And if it is lit up in the alpha channel, even if it is green,

12:07 keep it." And so it's kind of like that that guide because, you know,

12:10 you need a guide as an artist to be able to say like, "Hey,

12:12 this is what I want you to kind of hold

12:13 on to." So after that, the program does goes and makes a prediction.

12:18 It predicts the foreground unmixed from that green

12:20 screen and it predicts the alpha channel.

12:23 Now, I'm sure you guys are

12:24 familiar with premultiplied transparency and straight color.

12:28 Uh, so this is a system that's trained to give you that straight color output,

12:33 not a premultiplied output.

12:34 The idea there is that you're basically teaching the computer

12:37 to fully guess the color of that foreground pixel, even if it's only 1% opaque,

12:44 because at some point you're going to need to take that thing,

12:46 you're going to need to have the background color be removed completely,

12:49 tweak the alpha channels, do all all this kind of stuff to it.

12:52 And so that that straight color pass is truly

12:54 the pass that you're going to want for compositing,

12:56 but that's something that's physically impossible to capture with your camera,

13:00 but you can use a machine learning model to get it.

13:01 So all right, so the program spits out the predicted foreground,

13:06 the straight color foreground, and it spits out the alpha channel.

13:08 So now if we go to the top left once again, that's fine right there.

13:12 Um, so you got these two pictures of the guy on the left here,

13:16 we have the ground truth.

13:17 So inside the training program, it does some compositing.

13:20 It takes the input foreground, the ground truth, the rendered shot,

13:24 and it puts it on a random color background.

13:26 And it does the same thing for the predicted output.

13:28 Puts on the same random color background.

13:30 Then it compares the two.

13:32 Lucky for us, color is a pretty easy thing to define on a computer.

13:36 Red, green, blue, values between zero and one.

13:39 Very straightforward.

13:40 Works great for a neural network.

13:42 Values between zero and one is exactly how a neural network likes to work.

13:45 So you can do a simple loss comparison where you

13:48 just subtract one from the other and get the difference.

13:50 How big is that difference?

13:51 Well, how far apart are the pixels in terms of their RGB values.

13:54 So that measurement gets done between these two images.

13:58 But what if your hue, your color is off just a little bit on one of those edges?

14:03 Your RGV values might not be very far apart,

14:06 even though your hue is noticeably broken.

14:08 So not only do I do an RGB comparison between these two,

14:11 but I do what's called a Y, you know, a YUV color space comparison.

14:14 If you're familiar with YUV color space,

14:16 you basically have your your luma channel, your brightness as Y.

14:20 And then U and V are your chroma.

14:22 So you take U and V and you can just do a subtraction between

14:26 the ground truth UV and the predicted UV to get your true chroma difference.

14:31 And you can take that and multiply it to amplify

14:34 any errors there to make them even more extreme.

14:36 So okay, so I got my comp on a random background.

14:39 The background color changes here for every

14:40 single P image that goes through the system.

14:42 That way it's unbiased.

14:43 It will compound any color.

14:45 It'll learn how to fix any color that's going on there.

14:47 It's unbiased.

14:48 But not only is it doing that, if we go a little further here to the side,

14:52 the same image gets comped onto

14:54 a neutral gray background with no color whatsoever.

14:56 So that way it can't cheat and just make the edge pixels purple and be like,

15:00 I fixed it because then the purple would show up here on the side.

15:03 So same exact comparisons happen.

15:05 RGB, YUV, and of course the UV part gets multiplied.

15:09 So any errors there get ampl amplified.

15:12 So now if we go down, we see the alpha channel.

15:15 Now of course YUV doesn't work for an alpha channel.

15:18 There's no chroma in it.

15:19 Um RGB doesn't even quite work for an alpha channel.

15:22 It's just one channel, right?

15:24 So the other thing with an alpha channel is that the inside is all white.

15:28 And once you've predicted white, your job is done in terms of the computer

15:32 and you're not going to get a lot of improvement.

15:33 Even if an edge pixel is messed up,

15:36 you're getting a bunch of signal from the rest of this image saying,

15:38 "Hey, things are fine.

15:39 It's supposed to be white.

15:40 It's white.

15:41 Don't worry about it.

15:41 that little pixel on the edge.

15:42 Don't worry about it.

15:43 But we want those pixels on the edge to be perfect.

15:46 So I made something called a spatial waiting map.

15:49 Boom.

15:50 Right there.

15:52 So basically what I did is I went to the ground

15:54 truth alpha channel and said any pixel that is semi-transparent,

15:58 treat that as an edge.

15:59 And in fact dilate that edge a little bit.

16:02 You know, give me a little highlighted

16:04 outline using those semi-transparent pixels as your guide.

16:07 Anything inside of that is considered foreground.

16:10 Anything outside of that is considered background.

16:12 And of course, then you have the edge itself.

16:14 So now I can take that and make a multiplier

16:17 based on where the pixel is in the in the image.

16:20 So if the if the pixel is in the edge in that little highlighted white area,

16:26 you can multiply it that difference.

16:28 If you're doing like an RGB minus RGB difference,

16:30 you can multiply that by five, by 10, by 20.

16:32 Same with the YUV.

16:33 So, not only can you emphasize just the chroma by doing

16:37 like a YUV comparison and multiplying the difference between the YUV,

16:40 you can also do spatial weighting and multiply

16:43 the difference by where it is in the image.

16:45 And so, I use that spatial weighting to really really

16:48 really focus the neural network onto the edges of the image

16:51 to really make sure that any like differences between

16:53 the ground truth and the predicted result were getting amplified 10fold,

16:58 15fold, 20fold if they were on the edge.

17:01 So that that signal was just as strong as the signal

17:03 on the center of the image or the outside there.

17:06 Um if you go a little further here, we can see some other visualizations.

17:09 You have a difference mat between like a predicted output and the ground truth.

17:12 In fact, let's go all the way to the top right here.

17:15 Right there, that guy.

17:17 Um so this is the ground truth foreground output right there.

17:21 The the render.

17:22 And this is the predicted ground truth right here.

17:24 Or sorry, not the predicted ground truth.

17:26 It's an oxymoron.

17:27 This is the predicted foreground right there.

17:29 Now, you might notice the background looks all screwy in that.

17:32 The thing is it doesn't matter what's going on in this background

17:36 here because that never makes it into a final image.

17:38 That's going to get multiplied against a zero

17:41 for the alpha channel and that's going to give you nothing.

17:44 So if you try to have a neural network predict

17:46 what these pixel values should be in the background here,

17:49 you're effectively you're effectively entering

17:51 a divide by zero type of situation

17:53 here where it's going to go crazy trying to predict random noise

17:56 that doesn't actually affect the output and you're going to break your model

18:00 which I learned after a few weeks of trying to do this.

18:04 Um so there's a whole you know once again back to spatial weighting here.

18:07 I can say, hey, take that background,

18:09 those background pixels and ignore them completely or, you know,

18:12 add a little bit of a weight to them if needed to to try

18:15 to push it down towards black and that keeps it from breaking the model.

18:18 So, it was this combination of all these different

18:22 compositing pipelines within the training process that let

18:26 this program actually give me the result I

18:28 needed because otherwise I ended up with situations where,

18:32 you know, it would it would give me I mean,

18:33 the computer effectively learns to cheat, right?

18:35 If you don't wait it correctly, it might be like, "Oh, well,

18:38 you're trying to, you know, pull the color out of the background.

18:40 Well, let's just make all the edge pixels magenta.

18:43 That fixes it.

18:43 Gets rid of the green, right?

18:44 Problem solved." Like, well, that's not what I'm trying to do.

18:46 Or might be like, "Well, let's make all the edges uh black.

18:49 Let's just put a black pixel around there." Because you

18:51 have a big black background and with that black background,

18:53 if I make the edge pixels black, I'm rewarded for it.

18:56 I did my job.

18:58 You're like, "Well, no, that's not what I'm looking for either." So it

19:00 was this combination of compositing on a random background color,

19:04 causing on gray, doing RGB difference measurements,

19:07 doing YUV difference measurements, breaking out the chroma part of YUV,

19:12 multiplying that, breaking out the spatial weights,

19:15 multiplying based on where it was in the whole process there,

19:18 changing it as it went through the learning pipeline.

19:21 You might weigh something heavier at the beginning

19:23 till it starts to kind of figure

19:24 it out and it can ease off that weight and weigh other things differently.

19:28 And it was almost like making a soup or a stew,

19:31 but it took like a month to make.

19:34 Um, so after kind of working this over and over

19:39 and over and doing process after process after process,

19:42 honing in little by little on what was working, I ended up getting to a a spot

19:46 where I was getting results that were pretty good.

19:48 But the challenge is the Hierra backbone,

19:52 the the model that I was fine-tuning, you know,

19:54 the concept key, if you will, it's just a 512 x 512 image at the end of the day.

19:59 Uh, anything more than that just doesn't even fit on really any hardware.

20:04 Um, this is all being trained on an RTX 6000 Pro.

20:08 I was lucky enough to have one

20:09 of those machines put together by a Puget and Nvidia.

20:12 Um, and that let me, you know,

20:14 really crank this, put out some high quality stuff.

20:16 And I wasn't worried about optimizing it yet.

20:18 I wasn't worried about it running on anyone else's computer.

20:20 I was just worried about it running on my computer.

20:22 Um, so I got it working.

20:25 But that, like I said, that hyra backbone back backbone is only 512 x 512.

20:29 And if we want this to be professional,

20:31 you got to be able to see individual hairs, right?

20:33 It's got to be HD.

20:34 It's got to be 4K ideally, right?

20:36 Um, I want to come at this with no compromises.

20:38 I want this to work.

20:39 I want to be able to use this in my pipeline, not just be a tech demo.

20:42 So, what I end up what I ended up settling

20:46 with was I wanted a 2K by 2K image at least.

20:48 So, I at least had the vertical resolution of a 4K image.

20:51 And I like to think of it as an anamorphic 4K, right?

20:54 Just taking the sides and squeezing them in, you got your 2K x 2K image.

20:59 So, in order to get it up to that resolution,

21:01 there's a second step in this process,

21:03 which is something called a convolutional neural network,

21:05 which I actually don't know all that much about.

21:07 I kind of just said, "Hey, Gemini, take my hand." and uh then lead me um

21:13 which didn't work for weeks and weeks and weeks.

21:16 So I had to learn and do some research and read up on it online and eventually

21:20 I got something which I'm still not quite

21:21 sure exactly that you know the best principles

21:25 behind it to use but it's working and effectively what that does is it looks

21:28 at your your original incoming image the full res

21:31 image it looks at your predicted image which is

21:34 512 x 512 and goes okay well in the full res image I take this one

21:38 single pixel the 512 x 512 and look at the full res image I see little details

21:42 in there so let me put those details back in it's kind of like a extra smart

21:46 upres that gets to look at the source

21:48 footage to cheat and give you the upres version.

21:51 And once I had that working,

21:52 once I had the convolutional neural network plus the hyra back on fine-tuned,

21:57 I ended up with corridor key and it worked fantastically and it

22:02 blew my mind and it blew all of it blew's mind.

22:07 Um, and I was really proud of it and it was super cool.

22:10 But I don't want to stop there, right?

22:12 Like that's about as far as I could take things myself.

22:15 Uh, I also at a certain point need to make sure I don't just work on tools,

22:19 but I also take the time to then use them

22:21 for the things I wanted to use them for, which if

22:23 I were to just keep working on this and working

22:25 on this, I wouldn't ever stop and actually work on videos.

22:28 So, this was kind of where I entered

22:31 the second phase of working on corridor key.

22:34 And that's when I went to release open source,

22:38 which has been a fascinating experience.

22:40 And I should be, you know, upfront here, it's technically not true open source.

22:44 not yet.

22:44 Uh, for a variety of reasons, which I'll explain.

22:47 Um, but I'm very much figuring it out

22:49 as I go and that's been kind of my attitude.

22:52 I've been trying to be open about that and everybody's actually been

22:54 really supportive about that and okay with me like very publicly learning,

22:59 which should be a little scary sometimes, you know?

23:00 It's like, here you go.

23:01 And it's like, oh, it doesn't work.

23:03 Screw you, you know.

23:05 Um, so what I did is after I made

23:08 this, I released the model and the inference code online.

23:13 And I didn't release the training data yet.

23:15 I haven't released the training program that I made yet.

23:17 Um, but I basically put it out there.

23:19 I know we have an audience.

23:20 We have this fantastic story around it that Austin,

23:24 one of the editors at Corridor,

23:25 basically helped put together, which was the YouTube video that you guys saw.

23:28 And we put it up on our Corridor Discord, Corridor Creates.

23:33 And, you know, I've never really delved

23:35 into the world of software licenses and things like that.

23:38 Um, but I wanted to try to find a way to be able to release this for free,

23:42 let people use it however they wanted for free.

23:45 But I didn't want people to suddenly take it,

23:47 package it into a plugin, and start selling it.

23:49 Um, I wanted to kind of keep it for free

23:52 and keep it communal because the moment people start, you know, grabbing it,

23:55 and I'm I'm going to put this in a plugin and sell it here,

23:58 and I'm going to put this in a plugin,

23:59 sell it here, I'm going to put this in a plugin,

24:00 sell it here, it it fractures this whole concept

24:03 of people trying to come together and make it better.

24:06 Um, so by kind of limiting it to be like, hey,

24:10 you can all use it for free, but it's not commercialized.

24:12 Um, and you can commercialize it,

24:15 that kind of focus people to to work on it together.

24:19 Um, and kind of by leading by example, throwing it out there,

24:21 putting the whole story around it, explaining how it worked.

24:24 I got a lot of really passionate people that jumped into our community,

24:28 uh, into our Discord in particular, and really started working on it with me.

24:31 And the very first thing that happened was like when I made this it

24:34 it required you know 24 gigabyte GPU uh so you know 4090 3090

24:39 RTX 6000 like that kind of level stuff within a day people had

24:43 it running at on at six gigs um which is crazy uh you know

24:49 uh one guy who's a you know graduate student in machine learning came

24:52 he's like oh you've initialized a whole dimension that you don't need so I'll

24:55 just snip that and I'll even submit a patch to the official repo there

24:59 so they can fix them for you and and another person jumps and like,

25:01 "Oh, and your, you know, your CNN is way too,

25:04 you know, occupying way too much memory.

25:06 I can trim that down for you." And somebody else comes in like,

25:08 "Oh, you only take that from FP32 to FP16 and, you know,

25:11 have the have half the amount of RAM that you need." So,

25:13 within a day, like people optimize it.

25:15 It could run on, you know, not anything,

25:17 but it could run on, you know, normal GPUs.

25:20 The other thing is I didn't make any sort of graphic interface for this program.

25:24 It was all, you know, command line stuff.

25:26 That whole idea is that you drag in a folder full of clips

25:28 and you wait a couple hours and you get all your clips finished.

25:31 And once again, works great for us in our workflow here.

25:34 But, you know, for the vast majority of people, they want an interface.

25:37 And I totally get it.

25:38 It makes sense.

25:38 I like interfaces, too.

25:40 Um, so within two days, another user by the name of Ed came in and made

25:45 a graphic interface and just knocked it out of the park.

25:47 And because there's this whole like community

25:49 around it that's experimenting and trying things, you know, left and right.

25:52 He was getting feedback.

25:53 People are trying different things.

25:54 People were submitting ideas.

25:55 people are submitting poll requests and it was just like a matter of yeah truly

26:00 of a couple days until we had like

26:02 a full-blown graphic interface with tons of optimizations parallelization.

26:07 Um and it went from taking maybe four to 5 seconds

26:12 to render a frame to rendering effectively a frame every second.

26:16 Um which is pretty crazy to be able to knock out,

26:18 you know, one FPS on something like this.

26:20 Uh someone else on the discord just took it upon they made their they

26:25 they made it their life's mission to make this as efficient as possible.

26:29 Like truly they built a benchmarking system

26:31 that in in milliseconds defined every single process that this thing

26:36 went through and they got it running in real

26:39 time um on a dual 4090 system which is crazy.

26:44 First off crazy we have a dual 4090 system

26:46 but but the fact they got it running in real time is bonkers.

26:50 um you know not quite practical yet.

26:51 You know, you're not going to fire it up

26:52 in OBS and have it keying out your green screens yet,

26:55 but the experimentation is happening and it's happening in the open

26:58 and people are sharing knowledge around it and that's been really cool.

27:02 Um where we are at now is I haven't really had

27:08 the ability to see corridor key in use very much yet.

27:11 You know, I dropped it basically a month ago.

27:13 We've started using it a little bit.

27:15 We haven't actually shot all the green screen stuff for Son of a Dungeon.

27:17 So, I don't have like a deep personal experience with it yet.

27:21 And kind of the same thing's happening online.

27:22 People are using it little by little and I'm

27:24 getting to see what's working and what's not working.

27:26 And it's definitely scary releasing something into the world.

27:30 And like truly all I tested it on was the shots that you guys

27:33 saw there in the video with with the guys drinking water and, you know,

27:36 dancing around with clown wigs.

27:38 Um, I hadn't tested on anything else.

27:41 I had no idea this was going to work for people.

27:44 Um, but it did.

27:46 and it was working for people except it likes to flicker as you

27:50 guys probably know if your footage is noisy or you have chroma

27:53 compression or subsampling you're going to get a little flicker and that would

27:57 happen with regular keying tools as well unless you den noiseise it

28:00 and that was one of the things like okay I need to fix

28:03 that I need to figure out a way to fix that and once

28:07 again because of this this beauty of the community and this kind

28:10 of communal effort like true machine learning

28:13 professionals have joined in and they're like,

28:15 okay, you just need to add these couple things.

28:17 It will look at the frames before.

28:18 It'll make them temporally consistent.

28:21 And there's experiments going on right now

28:22 that are just completely removing the flicker altogether.

28:25 Um, and my hope is that, you know, in the next couple months to a year here, um,

28:33 we can take this to the point where it

28:36 becomes a tool that can run on most systems.

28:39 I don't think it's going to run on any computer,

28:40 but hopefully it can run on most computers.

28:42 uh it's fast and can handle truly whatever you throw at it.

28:46 Like that's that's what I want to achieve with this.

28:49 And you know, at this point,

28:51 the best thing I can do is just encourage other people and give

28:55 them that energy and give them that support to keep going with it.

28:58 Um because the my my knowledge of machine learning has been fully tapped out,

29:04 but my domain knowledge in the world of visual effects has not been tapped out

29:07 and my ability to tell a story and to like

29:09 rally people around this idea of, you know,

29:11 making your own paint brushes, so to speak.

29:13 You know, I I got plenty of that in me still, too.

29:16 So, it's been a really interesting journey so far.

29:18 Um, it's been really cool experiencing all this and it makes me really

29:24 think about what other tools are lying out there waiting for an artist

29:29 or somebody else to make because nobody else has made them yet

29:32 or a software company hasn't found the value proposition in making it.

29:37 Um, the ability to just kind of lay out your design,

29:41 pass it off to Claude or Gemini or whatever,

29:44 have it come back and give you a plugin.

29:46 It's a really brand new territory here

29:49 and something that I'm really excited about.

29:50 Like when I think about AI as a tool, like it's not specifically like, oh,

29:54 here's an LLM that does some crazy things or here's

29:56 an image diffusion model that does some crazy things.

29:58 It's, hey, you now as an individual have the tools and the power to sit down

30:04 and compose a neural network or AI for whatever it is that you need to do.

30:08 And yeah, it's going to take some time.

30:09 There's going to be trial and error and all that kind of stuff,

30:11 but compute is available for everybody.

30:14 And there's some crazy things you can do with it.

30:17 Um, and you know, I don't know what people are going to do with it.

30:19 I don't know if in, you know, 2 years from now we'll have crazy sound

30:22 tools that can remove the reverb from crappy audio,

30:25 you know, or tools that will automatically color match every shot, etc., etc.

30:29 But I'm really excited to kind of see what people do with this.

30:32 And of course, you know, it's not just artists,

30:33 but the actual professional software designers are

30:36 accelerating as well and doing interesting things.

30:39 So yeah, it's been uh it's been a journey and illuminating and really exciting

30:46 and I'm hoping to kind of just keep it going as best as I can.

30:51 So yeah, I think we should take a moment here and uh you know,

30:54 I want to say thank you to Nvidia Studio for helping with this as well.

30:57 Um but I want to ask you guys some not ask you guys some questions.

30:59 I want you guys to ask me some questions.

31:01 Uh give me an idea of some stuff you'd like to know.

31:03 What's next?

31:04 I just told you quarter key 2.0 with no flickering.

31:06 Oh, and quarter key for blue screens.

31:10 Well, what I'm really trying to figure out here

31:12 is where does the need for a green screen end?

31:17 And I don't mean in the sense like, oh, green screens are bad,

31:19 because I actually think green screens are a really handy tool.

31:22 You know, it's great as a filmmaker to be

31:24 able to have something that you can say, hey, consider this the background.

31:28 Like, you need some way to control things, right?

31:31 You need some way to direct things.

31:32 If you just say, "Hey,

31:34 just guess whatever the foreground is and remove it." Well,

31:36 there's going to be times when you're filming weird stuff,

31:39 abstract stuff on a green screen.

31:40 Like, imagine somebody walking into frame.

31:43 Sure, when they're in frame, it's like, "Oh, yeah,

31:45 clearly there's a human being there and AI can pick it up.

31:47 What about those like couple of frames when it's a blurry blob

31:50 on the edge of the frame as a person walks in the frame?

31:54 How are you going to dictate, hey, that's a person,

31:56 not the edge of the green screen, not some shadow that I want to remove.

32:00 I want to keep that." And to me,

32:01 the easiest way to do something like that is to be

32:04 like just to fall back in your traditional green screening concepts,

32:07 which is like, hey, if it's green, it's the background.

32:09 If it's not green, it's the foreground.

32:11 So to me, something like a green screen is still a really useful tool,

32:15 but you have issues like bounce light, you know, color contamination.

32:19 So is there a world where in the future the ideal screen is like

32:23 a pastel green screen or a gray screen or a white screen or something like that?

32:28 you know, something where it's like it's

32:30 a background color that the computer can at least

32:32 lean back on if it can't figure out

32:34 what it's looking at, but something that doesn't,

32:36 you know, uh, contaminate your image colors anymore or your foreground colors.

32:40 Um, so there's this whole question of like

32:43 if you have this kind of keying intelligence, this matting intelligence,

32:47 how does it affect the physical world

32:48 of what you're filming and what you're doing?

32:50 And that's kind of the experiment that we're getting into right now.

32:53 But before I get into any of that stuff,

32:54 the very first thing we need to do is finish Son of a Dungeon.

32:57 Uh because then I can actually see how it works and I can learn and I

33:00 can tweak things and the code's right there and I can adjust it on the fly.

33:05 So yeah.

33:05 All right.

33:06 Thanks Ram.

33:06 That was a good good prompt.

33:08 Uh does anybody have any questions?

33:10 Yes.

33:11 So two questions.

33:12 Is there going to be an official four-door key open effects plug it?

33:16 So you can put it into like pitch revolve something like that.

33:19 The second one for the other part with green screens is transparent objects.

33:25 So, I know it doesn't do object detections,

33:27 but could we see potentially a like refraction map or a distortion

33:32 map for like blasts of water or something like that?

33:37 Okay, good question.

33:37 Uh, all right.

33:38 So, two questions here.

33:39 First one was, is there going to be an official cord key plugin,

33:42 like an official OFX plugin?

33:44 Maybe.

33:44 I know that's not a very good answer to your question.

33:47 Um, but there is an OFX plugin already.

33:50 uh a guy named Al in uh Brazil has been working with me making

33:54 a runtime C++ tensor RT something uh OFX plugin and it works pretty dang good.

34:03 Um there's actually a bunch of After Effects plugins that are out as well.

34:06 Um I don't know where Nuke is at currently.

34:09 Uh I don't know where other tools like Baselite are at currently.

34:12 My hope is that they're in progress and that people

34:15 are working on them and I'd fully encourage it.

34:17 uh you know, if there's anything I can do to help encourage that, that's,

34:20 you know, kind of what I'm trying to do here.

34:21 Um, but the OFX plugin is is great.

34:26 Uh, it works really well.

34:27 You can download it right now for free.

34:29 Uh, it will always be free to download.

34:31 Um, the uh the second part, what was the second question?

34:36 I got you invested in the first.

34:37 Oh, yeah.

34:37 Transparency.

34:38 Okay.

34:39 So, I've saw this question a few times on the Discord as well.

34:41 What do you deal how do you deal with refraction?

34:43 Right?

34:44 You see it.

34:44 You know, somebody has glasses on and in real life, of course,

34:46 you can see the background distorted through their glasses

34:48 when they turn their head to the side.

34:49 But on a green screen, it just becomes full trans fully transparent, right?

34:52 It's not how glasses work.

34:54 It's not how refraction works.

34:56 Well, at the same time that this whole keying

35:00 stuff is happening and map prediction stuff is happening,

35:03 there are also models out there that are predicting normals,

35:06 uh, depth, um, albo maps, specular maps, things like that.

35:11 And a lot of what we're doing here is meant for this pipeline.

35:14 We're going to be using beeble um for the son of a dungeon stuff because

35:18 that's like a fully built uh almost

35:21 like material prediction pipeline for your footage where

35:24 you give it your footage keyed ready to go and it goes okay I'm going

35:28 to predict all the normals how the surface is you know where surface is facing.

35:31 I'm going to predict uh you know what's

35:33 the colors there without any shadows or lighting on them.

35:35 I'm going to predict how shiny things are.

35:37 And with that, then you can throw it

35:38 right into a program like Unreal Engine and you

35:40 can relight your real footage in Unreal Engine

35:43 or Blender or you whatever any program, right?

35:46 Any 3D rendering program.

35:48 And my thought when it comes to transparency,

35:50 as long as you predict the transparency

35:51 correctly and you subtract that background color so

35:54 you don't have like a green tint in your glasses in order to get refraction,

35:58 then you have to lean on your normal prediction.

36:01 Um, you know, what's the curvature of that surface?

36:04 Uh I did see a really fascinating idea somebody had which was

36:07 to make another synthetic data set with um UVs uh basically UVs mapped

36:14 out as another output so you can train the program to predict

36:17 what the UVs are for the things that you're looking at um which

36:20 is a crazy concept but once again as this stuff gets out there

36:25 and my plan is to release the the training data and the training

36:29 program which once again it's it's a vibecoded you hobby fest like

36:35 it's probably not that great but at least it can start right.

36:38 Um once that's out there somebody else be like oh well

36:40 let me just render out these passes instead go you know

36:43 wait a few days and end up with a model that's

36:45 trained to predict a different kind of output but running running out

36:47 the same structure and of course if somebody else comes out

36:49 with a different backbone model to try or a way to train

36:52 a model from scratch ideally because this is open in the community

36:56 people can kind of jump in and do this kind of stuff.

36:58 So I would encourage you to give it a shot.

37:01 Honestly, you know, it's uh maybe like my plan is to release the uh

37:05 the training data and the training program

37:06 once there's a good plugin for every program.

37:10 So I figured, you know, for it's one thing to have the model,

37:12 it's one thing to have the inference ability,

37:14 it's another thing to have the tool.

37:16 And that's a big thing I realized like cool, I finished it, it works great.

37:18 And I release it and people like, oh,

37:20 this is like a command line interface that requires 24 gigabytes of VRAM.

37:22 This is useless.

37:23 I was like, yeah, they're right.

37:25 I need to be able to actually just use it in the program that I use.

37:27 everybody needs to be able to use it in the program that they use.

37:30 So once you can fire up After Effects, Dinci Resolve, Fusion, Nuke,

37:34 etc., it's, you know, download for free or it's there or whatever.

37:37 And you can just use it.

37:39 Once it's that easy, then I'm like, okay, put the training data out there,

37:42 put the training program out there, let people go to town.

37:45 You know, at least it's been centralized, it's been demonstrated.

37:48 you know, there's a a a focal point

37:50 for when people train new models or new ideas.

37:53 There's an architecture on how it can then get utilized in software,

37:56 which is a whole second step from the first

37:58 step of just train it in the first place.

38:01 Um, so that's where that's at.

38:02 All right, I'll take another question.

38:03 Yes, you're testy.

38:06 Did you consider using apps or flag?

38:16 Cool.

38:16 Thank you.

38:16 Um, so his question was, "What do you think about doing like a dark

38:19 gray or black background?" Something I thought about.

38:22 Um, it comes down to being able to queue in the computer

38:25 as to what you want to keep as foreground and background.

38:27 Uh, let's say you're filming a miniature exploding and a bunch

38:30 of tiny little black particles go flying off into the air.

38:34 How do you know if those aren't tracking markers?

38:35 How do you know that those aren't background?

38:36 Right?

38:37 So, I think there is a certain point as a person filming, as a filmmaker,

38:42 you still need to step up and help the computer a little bit.

38:44 And that's where I think at the very least a light gray might be the way to go.

38:47 So you still get your bounce light but it's not contaminated with the color.

38:50 Or you go with a hue, you know, a touch of magenta or tan or green

38:55 or something like that so that worst comes to worse,

38:59 you can still just fall back into a key.

39:00 That's one of the things I like about filming

39:02 on a green screen is I can still use color keying, right?

39:04 Like color key tools still exist and they're still not they're not bad.

39:07 Like they're pretty great.

39:08 It's just that this is one click uh close to one click and that's the idea.

39:12 you know, this gets you there quickly and painlessly as much as possible.

39:16 That's the goal.

39:18 So, I don't quite know what the end result will be.

39:20 And frankly, my hope is that once again, people experiment, right?

39:24 It's just a matter of kind of setting up your computer

39:26 to kick out some renders for a while and then

39:28 setting up the training program to chunk on those renders

39:30 for a while and then you'll have a different model.

39:32 You could train a gray model, you could train a an orange background model,

39:35 you can train a blue model, etc.

39:37 So, yeah.

39:39 Uh, you mentioned Gemini a couple times.

39:41 Is there a reason Gemini was your go-to?

39:43 Good question.

39:44 All right, so I did mention Gemini a couple times,

39:45 and I'm sure everybody's like,

39:46 "Well, why aren't you using Claude?" Um, it's because Gemini is cheap.

39:52 Uh, for 20 bucks a month,

39:53 I can basically get unlimited Gemini, and I don't get unlimited Claude.

39:57 Uh, the other thing is that the challenge here wasn't so much the programming.

40:02 Um, it was the design.

40:04 And frankly, you need to be able to write the entire program

40:08 in plain English to the exact details of how the compositing formulas work.

40:13 Once you can do that, you can throw it almost any LLM.

40:16 Doesn't need to be clawed and it'll get the job done.

40:19 And that's basically what I needed to do here.

40:21 The other thing I learned is that, you know, LLMs,

40:23 they're great at taking structured language and turning

40:26 it into, you know, script code, etc.

40:28 They're great reasoning logic machines.

40:31 They don't understand images whatsoever.

40:33 So when I'm comping an image,

40:34 it doesn't know what premultiplied is or straight or linear or sRGB sRGB

40:40 versus 2.2 do like gamma gamut like all these things are you know even

40:45 though it thinks it knows it doesn't know them and constantly I was

40:48 dealing with issues where because I had to do basic code a compositing pipeline

40:52 for this you know for all the testing that the you know that the loss

40:56 and the weights go through as a pro model is trained and decided

40:59 to program in like okay let's let's take in this sRGB EXR file let's

41:04 convert it to linear let's unpremultiply it

41:08 let's comp it on this background And then, you know, we go through it like, oh,

41:11 it's using the premultiplied compositing formula,

41:15 not the straight color compositing formula.

41:16 So, little by little,

41:17 you go and you find these little problems and you have to lay it all out there.

41:20 And, you know, maybe in the future the highest

41:22 end LLMs will catch that kind of stuff,

41:24 but my experience has been that you really have to bring

41:27 your domain knowledge and then the LLM like Gemini will do the coding.

41:31 And at that point, Gemini was completely capable for what I was,

41:34 you know, what I was needing.

41:36 Yeah.

41:37 All right.

41:37 Any other questions?

41:39 Nice correlative peppers.

41:54 A lot of the white background.

41:58 It is interesting to see that.

42:03 Yeah, good question.

42:04 Okay, so the question was how do you

42:07 deal with things like background objects like a light,

42:10 a stand, the edge of the green screen, edge of the green screen.

42:13 Um, so that's where something called the alpha hint comes into play.

42:17 So my thought was rather than having the program completely

42:21 guess what the foreground and the background is supposed to be,

42:23 once again you're an artist, you're a filmmaker,

42:25 you can at least give it a hint, right?

42:27 Aka an alpha hint.

42:28 You can pull a green screen key and just be like, "This is the background.

42:31 This is the foreground.

42:32 Here's a black and white image that shows that." Or you

42:34 can do a magic mask or roto brush pass or whatever.

42:37 Like it doesn't need to be good.

42:38 Just a way to be like, hey, if it's white pixels, it's a foreground object.

42:41 If it's black pixels, it's a background object.

42:43 Even if it's not green and it really comes down to control.

42:46 Like you need a way to control your shots.

42:49 Like you can't just be like, AI, take the wheel.

42:51 Help me.

42:52 Like you got to have a way to tell it what you want to keep.

42:55 Um and so that's why I rely on this whole alpha hand concept where,

42:58 you know, pull a quick key.

42:59 It doesn't matter.

43:00 Or use magic mask, but just be like, I want this.

43:03 I don't want that.

43:04 And then the model ideally is intelligent enough to divide it from there because

43:08 the the actual separation of foreground

43:10 from background isn't happening through a chroma process.

43:13 The chroma really only comes into play

43:15 when it's that unmixing on those semi-transparent edges.

43:18 There's also this concept of like not reinventing the wheel.

43:22 There are research projects out there that are so good at alpha extraction,

43:26 like amazing immaculate alpha extraction.

43:28 There's re research projects out there that do so many different things.

43:31 I didn't want to have to reinvent that.

43:32 I just wanted to really focus on the whole

43:34 unmixing side of things and just making it usable

43:37 one click or as close to one click

43:39 as possible for our workflow and then go from there.

43:42 Yeah.

43:44 Weird question.

43:45 How do you think it would handle retrofosing retroreflective compositing?

43:50 Um, it probably handle it fine, honestly.

43:54 Yeah, you know, it's uh I think it probably handle it.

43:56 I I'm very surprised at how capable this system

44:00 is at pulling the foreground from the background,

44:03 whatever the background might be.

44:05 Um, but just as long as I kind of like a flat wash of color,

44:08 it could be anything.

44:10 Um, yeah, you know, but it's also one

44:12 of those things where I don't know until people try it

44:15 and it's only going to get better if people

44:17 try it and kind of give feedback to each other.

44:19 So, yeah, with dealing with things out of focus, I keep on the background.

44:25 say the shot request to finish this talk to each other.

44:29 Uh, do you think it'll be helpful to deal

44:31 with that sort of mouse stream or is that like

44:35 Okay, so the question was how do you like when dealing

44:38 with really out of focus stuff like an over the shoulder shot, right?

44:40 Uh, over the shoulder f 2.3 or something like that, you know,

44:44 it gets really blurry.

44:46 Um, it handles it pretty well actually currently.

44:48 Um, I really made sure that the data set

44:51 that I was feeding it contained extremely blurry shots,

44:55 extremely motion blurred shots, fine hairs,

44:58 other crazy kind of things like that.

45:00 Um, like I basically made sure that there's a render that documented

45:05 every single problem I've had to deal with over my, you know,

45:08 20 plus years of compositing stuff.

45:11 Um, so that as long as I can handle it in the data,

45:14 I knew I can handle it in real life.

45:15 And it generally does.

45:17 I mean, if you go really extreme and you have

45:19 a gradient across the entire frame of opacity, yeah, you know,

45:25 you still got to like try a little

45:26 bit to like treat your green screen with respect, but it handles it pretty well.

45:32 And I think that's a sign of it working how it should.

45:34 Um, to me, the the stress test I would

45:36 do is I'd throw like really transparent smoke at it.

45:38 Um, you know, wispy smoke elements, things that are already keyed.

45:41 I'd recomposite them on green so I'd have that ground truth still.

45:44 I just see how well it could do.

45:46 And it did a pretty dang good job.

45:48 Yeah.

45:50 Yeah.

45:51 All right.

45:52 Any other questions?

45:55 All right.

45:56 Well, guess I answered them all.

45:57 Okay.

45:58 Yes.

46:01 Oh, I are any looking cors.

46:08 Yeah.

46:08 So, the real time is tricky uh with machine learning stuff.

46:11 A big thing I've learned with this is that getting a machine learning model,

46:16 a neural network to run on your computer the correct way is really challenging.

46:22 Um, you know, by default your computer likes to run things

46:25 as a single thread on the CPU and call it a day,

46:28 which is fine if you're just going to, you know,

46:30 change some folder names or do some basic math.

46:33 But if you're going to run, you know,

46:36 long matrix multiplication and try to fire it

46:40 on all thousand of your CUDA CUDA cores or something

46:42 like that, you need to be able to talk

46:44 to your hardware in a really specific way.

46:47 And that's been a bit of a challenge.

46:49 Um, I mean, there's definitely tools for it.

46:51 Like if you're set up correctly, you have the proper,

46:54 you know, most recent drivers, you have the proper pietorch, etc., etc.

46:57 Like you can make stuff sing.

46:59 The problem is that a lot of people I mean there's

47:01 a technical barrier between just installing a program and then making sure

47:04 that you have all the proper necessary drivers and dependencies and I

47:10 guess triggers and flags to really make things work correctly on your GPU.

47:14 And we're also kind of in an era where that hardware

47:17 has been kind of rapidly evolving over the past couple years.

47:19 It's like you go back to hardware older

47:21 than four or five years from, you know, ago, like 2020 and earlier,

47:24 and you start to really lose the abilities

47:26 to put the stuff through a network fast.

47:30 Um, and at this point, I can just kind of, you know,

47:33 just put it out there and hope that people

47:35 will work together to solve those things with that hardware.

47:39 Have any of you guys used quarter keys?

47:41 Has anybody experimented with it?

47:43 What was your experience like?

47:46 Really, really good.

47:47 Okay, that's good.

47:49 What kind of shot were you keying?

47:51 Uh just a basic green screen stuff and I did a I use Blender a lot

47:54 so I made a lot of like shots

47:55 with blurry backgrounds smoke and it was very impressive.

47:59 Okay, that's I'm glad to hear it worked for you.

48:00 Um what kind of system are you using?

48:03 Um I have a I have a 20 or 30 I have a 350 and a 370.

48:10 I ran on both.

48:11 Uh but I also have a 10 1080 that works on a 1080 too.

48:15 It was fine.

48:15 I mean it took a while but yeah it worked.

48:20 Yeah.

48:20 You know like the the next steps for me

48:22 besides just trying to make the plug-in better and foster

48:25 that community development is also try to just make sure

48:28 that like I'm learning a lot about software licensing this process.

48:31 like right now it's it's under like a creative common license uh

48:34 that uh protects against non-commercial but I'm

48:37 learning that creative common doesn't really apply

48:39 to code and yada yada so just little by little day by day

48:43 talking to people and they're guiding me through it but at the end

48:45 of the day as long as people are kind of coming together

48:48 into a central point and I can like maintain that energy I feel like we

48:54 can make it happen right we can make green screens easy feel like

48:58 somebody would have done this six years ago but at least it's happening now.

49:03 Yeah.

49:04 Um, let me see how much long we got left.

49:06 That's kind of everything.

49:07 We got a few more minutes.

49:08 Um, yeah.

49:10 I guess anything else?

49:11 Any other questions?

49:12 Anything that people want to know?

49:13 We got one in the back there.

49:14 Yes.

49:15 So, what is the advantage of?

49:21 Oh, good question.

49:28 Gotcha.

49:29 Okay.

49:30 Uh you guys familiar with the sodium vapor process?

49:32 The most mathematically precise way to pull a mat these days.

49:37 Uh so the question was why didn't why aren't we using sodium vapor?

49:40 Um sodium vapor is very cool but it's very specific for how it can be used.

49:45 A couple challenges.

49:46 One because we don't have the magical Disney prism that has

49:50 the uh split in the middle the splitting wave wavelengths of light.

49:54 We have to take our light split it first and then filter it.

49:57 What that means is that every exposure

49:59 is getting half the light it would otherwise.

50:02 So you start really having a blaster set with light.

50:05 The other challenge is it's a rig with two cameras.

50:07 So the challenges of shooting stereoscopic apply to shooting uh sodium vapor,

50:12 which is to say you have to have

50:13 two cameras perfectly lined up, same focal length.

50:16 If you're going to pull focus,

50:17 you have to pull focus across both cameras simultaneously.

50:20 If you're going to zoom, you have to zoom across both cameras simultaneously.

50:23 Um so there's definitely restrictions of sodium vapor.

50:27 The thing I haven't done that would be interesting

50:30 to try is to make a data set using sodium vapor.

50:32 You know, take the time, set it up, put on a tripod,

50:34 put in the background, you're not going to pull in focus, whatever.

50:36 Make life easy, right?

50:38 Take that, record real life footage, get a bunch of different stuff,

50:41 and then you have your perfect alpha channel mask,

50:44 you have your perfectly unmixed foreground, etc.

50:46 That would be a a really interesting way to use sodium vapor,

50:49 but I think using sodium vapor in real

50:52 life is really relegated to very specific setups.

50:55 you're not pulling focus, you're not pulling zooms.

50:58 Camera could move, but you're going to be dealing with a lot of hardware there.

51:01 You know, some ideas that people had mentioned back when they made the sodium

51:03 vapor video with Paul here was using sodium vapor for uh the backgrounds

51:08 of sets like when you're in like a in a skyscraper apartment set

51:11 or anything else a lot of reflection and glass and that kind of stuff.

51:14 Things that would be tough green but aren't so bad when you have

51:16 like a perfectly physically filtered system that's

51:19 giving you that kind of, you know, perfect wavelength of light.

51:22 That's a good question.

51:23 Sodium vapor is really cool and it's like

51:24 our touch point for like the ground truth.

51:26 Like if we can hit that quality, then we're good.

51:29 We're not quite there yet.

51:30 Not even with Corey.

51:31 It's still not quite as good as sodium vapor, but it's way easier to set up.

51:35 Yeah.

51:36 All right.

51:37 Any other questions?

51:40 Yeah.

51:40 I've coded before.

51:43 Uhhuh.

51:42 I've hit a lot of roadblocks.

51:45 Uhhuh.

51:45 What were things you learned along the way?

51:47 I tried doing like a nonlinear editor and just troubleshooting bugs,

51:51 all that kind of stuff.

51:52 Yeah, very very hard.

51:53 So, when it comes to vibe coding, Nico's vibe coding tips.

51:56 All right, five minutes of Nico's vibe coding tips.

51:58 Uh, I'm by no means a professional, but I quite enjoy doing it.

52:02 Um, make sure you're using an intelligent like IDE like cursor or anti-gravity.

52:09 Uh, I use anti-gravity personally.

52:11 Um, it makes your life way easier.

52:13 You're not just deal with copy and pasting

52:15 and chunks of code and all that kind of stuff.

52:17 The other thing is to really make sure you take

52:19 your time to plan and to write out your plan,

52:21 whether it's by yourself or with an LLM.

52:23 It could be separate.

52:24 It doesn't need to be in, you know,

52:26 your coding environment, but plan everything out.

52:29 Um, another really important thing to do is to look up previous research.

52:34 Um, find other papers that have been published that deal with the same problem.

52:39 Read up on them.

52:40 If there's something that you don't understand, take the time to understand it.

52:43 Whether it's, you know, Claude, what does this mean?

52:45 explain it like I'm five, you know, or whatever.

52:48 It doesn't matter.

52:48 Just try to wrap your head around it.

52:49 Um, I'm a big believer that even

52:51 the most sophisticated advanced concepts can be understood

52:55 if they're phrased the right way by almost

52:57 anybody and you have like the right foundation, the right nuance to get there.

53:01 Um, I guess I would say the other thing was just uh,

53:05 you know, building it modular.

53:07 Uh, don't try to do everything at once.

53:09 Build it piece by piece.

53:10 Don't let the the LLM go to town on your entire codebase.

53:14 It's going to screw things up.

53:15 it just it doesn't have that awareness.

53:16 You know, you really have to think of it almost like a translator.

53:19 Um you're writing the book,

53:20 the translator will get it into the appropriate, you know, language for you.

53:24 Um but if you're not writing the book,

53:25 the translator can't sit there and just invent the story points for you.

53:28 So that's I would say, you know,

53:30 the pitfalls and things to think about to look out for.

53:32 Um one really cool thing is that as your program is being made,

53:37 if the LLM does something that you don't understand, take the time to say, "Hey,

53:40 explain to me what you did." And walk through it from there as well.

53:43 So, it's really really important that you steer the ship.

53:46 And it's really easy to not steer the ship,

53:48 but that's when things will fall apart.

53:53 Yeah.

53:52 Okay.

53:53 Anybody else?

53:56 That's really Oh, yes.

53:59 I don't know too much about that.

54:01 I said pop up like Steve, but tell us a little bit about that.

54:09 Yeah.

54:09 Yeah.

54:09 Okay.

54:09 Uh so the question was to talk a little

54:11 bit about the Bible integration here and the process there.

54:14 So Bibble it's a funny name um is a AI piece of software

54:19 that will take your footage um it can do an AI roto on it.

54:23 It will predict normals it'll predict albido specular etc.

54:26 And then it also offers a very kind of fast pipeline

54:30 to put that into some 3D software package of your choice.

54:33 One of the really interesting things about

54:35 it is the compositing aspect of visual effects kind of gets blended with your 3D

54:41 layout and rendering side of your pipeline.

54:44 And it's kind of a a change up and in a really magnificent way.

54:49 Let's say you're going to be doing a shot

54:51 in Unreal Engine walking down the dungeon halls, right?

54:55 You got torches burning and you got dungeon steam,

54:58 I don't know, in a skeleton in front of you or something.

55:00 Dungeon steam.

55:02 Um, those are things that you'd have to deal with, uh,

55:05 compositing in layers, like either going to deal with like,

55:07 you know, depth compositing or you're going to deal with just,

55:09 you know, rendering all your assets.

55:10 You pock them into whatever compositing program you're

55:13 in, stick things on top of each other,

55:15 tweak the color levels, get the bloom right, etc., etc.

55:18 But now if your if your footage of your actor

55:22 lives in the 3D scene and has normals,

55:25 so it's lit by that scene and it has the proper

55:28 specular and highlights and that kind of stuff and it's motiontracked.

55:32 Well, now when you render out the scene,

55:33 you're rendering out one unified image where all those elements

55:36 are being comped in that scene as it's rendered,

55:39 you know, uh, physically based and it's, you know,

55:42 any lens effects are happening to the entire scene as one full stack.

55:45 And so it it skips over some

55:47 of this chunkiness that happens when you kind of have

55:50 to like fake things all being in one

55:52 composition versus just being in one actual composition.

55:55 So we're really excited about that because when

55:57 it comes to this this D&D show, you know, the idea is that we're going to be

56:01 literally in the game board that we filmed on.

56:04 Uh you know, the whole game board was built in 3D.

56:06 It was 3D printed and painted, but we still have the 3D files.

56:09 So we played this game,

56:10 but now we can be in the actual game board at the same time.

56:13 And uh you know if you're dealing with Dungeons and Dragons,

56:16 you're dealing with many many minutes of footage.

56:18 So you know film your actors in the green screen once,

56:21 take a two-minute shot, bring it to life in that world,

56:23 skip the compositing step, just everything's in Unreal Engine or everything's

56:26 in Blender and just hit render and you're done.

56:28 So that that's the whole idea here.

56:31 You can kind of see some examples there of, you know,

56:33 the the older versions of Son of a Dungeon.

56:36 Um yeah, so that's kind of where things are at and it's really exciting.

56:41 Uh, you know, it's cool to see the turnout here,

56:44 you know, for learning about quarter key and talking about it.

56:47 Um, there's something really fulfilling about making a tool

56:52 and then people actually finding use from that tool.

56:54 It's uh, it's a very good feeling.

56:56 Uh, hard to describe.

56:58 Um, so I'm really really happy with that and it's really

57:00 energizing and keeps me wanting to go and make more stuff.

57:04 So yeah, thank you to everybody uh, for coming out and joining us here.

57:08 It's super cool to see everybody.

57:09 It's good to be back at NAB.