Dev Community Live: GTC Vibe Hack Winners – Building Impactful Agents

0:00 video game experience could be like,

0:01 especially if you don't have a scenario where there's

0:05 somebody professionally on Twitch uh commentating for you, right?

0:09 This is something that uh really gives that that experience.

0:12 So, I'm assuming that uh one or both

0:15 of you are playing esports or or doing uh something.

0:19 I think you showed Rocket League as an example uh in your demo, right?

0:23 Um yeah it it was a Rocket League but uh then

0:28 we generated like a Sora copy of Rocket League to show

0:32 on this one and yeah and I can I can walk

0:35 you through the UI interface right now if it's still there.

0:39 I'd love to see that.

0:40 Yeah.

0:41 So the first thing you do is upload a video what uh what you

0:44 want AI to understand generate commentary for and yeah you can just I will open

0:50 this video and then you have a couple of slides um sliders first one is

0:55 for the excitement level but this doesn't

0:57 change the tone of the voice itself because

0:59 we using like fairly simple model for text to speech but it changes the style

1:03 of generated commentary for example make it

1:06 more calm or more energetic and the second

1:08 slider is just original volume um so

1:12 that the um generated generated um generated

1:17 audio would be a little bit louder than than the sound itself in the video.

1:22 After that the we just press the analyze clip and it will in the background

1:29 uh it will analyze uh it will extracts the frames from the video.

1:33 Yeah, you can have you can see the trace like on the on the right side.

1:37 It would extracts the frames from the video then

1:40 tries to understands it describes what is happening generate

1:43 short commentary and classifies which comments are useful as well

1:47 and we can see the commentary on the bottom.

1:50 So these these are the commentaries which will be generated

1:54 as speech and then synced with the video and I

1:57 don't think I can I can play the video

2:00 on my computer but I don't think it will be streaming.

2:02 So, can I ask guys to play the video I sent to you before?

2:09 At least the audio like you.

2:11 So, we we have Zach in the background who's going to help put that up.

2:15 Now, as he's working on that, I again want to stress

2:18 to people who are watching this was a 2hour vibe hack.

2:21 And these two came in with just kind of a a napkin pitch level of detail

2:26 on what they might do for the project

2:28 and came away after two hours with something like this.

2:31 Pretty cool.

2:32 Glowing ball in the neon arena.

2:36 Flaming engine adds intensity to the high-speed play.

2:41 Rocket booster car poised for action on the blue platform.

2:48 That that was short.

2:49 That was short.

2:50 Yeah.

2:51 But uh you understand the idea.

2:52 I I think.

2:53 Yep.

2:54 Well, and that that is the thing that I

2:56 see is really unique to hackathons today that are

2:58 focused on AI is that you're able to much

3:02 more quickly get something that shows what the vision is.

3:07 You're not spending hours on end in C++ or you

3:10 know uh Python like trying to build something that that works.

3:14 You can literally Vipack and get something in two hours that is functional.

3:18 That that must be pretty exciting as uh hackers

3:22 100%.

3:22 I feel like it's it just lets you express like your ideas uh more rapidly which

3:28 lets you then like figure out which idea you even want to do in the first place.

3:31 Like a lot of um time is usually spent in like building

3:35 the thing only to realize that you might have to pivot later on.

3:38 And this just lets you AI in general just like being able to like use

3:43 human language to help guide the process

3:45 of development just lets you spin up ideas faster.

3:49 So you can actually choose which idea is

3:50 actually worth like investing more time into it.

3:54 For sure.

3:54 And what we see now at hackathons are just so many

3:58 ideas where the most exciting thing is the hacker's unique personal vision.

4:04 So that allows hackathons and AI to provide a new

4:09 canvas where hackers are both brilliant engineers and artists as well

4:14 because they can get to this point of personal expression

4:17 even in a two-hour window which is kind of crazy.

4:21 Yeah, the time frames are really compressed.

4:22 Like normally for something like this it would take,

4:24 you know, potentially even weeks, you know,

4:26 and you wouldn't be able to have

4:27 a hackathon really that's lasting weeks at GTC, right?

4:31 So

4:32 yep.

4:32 Well, speaking of GTC, so we know that this was uh the last day of GTC.

4:38 It was early in the morning.

4:40 Can you walk us through kind of a a day in the life

4:43 of attending GTC and what it was like to do a hackathon there?

4:47 Because there's a lot of people listening who maybe have

4:50 thought about going to GTC and haven't pulled the trigger yet,

4:52 uh, or, you know, might find it

4:55 interesting to compare their experiences that they have.

4:57 What was it like for you guys to be at that show and then go to the hackathon?

5:02 Yeah.

5:02 Um, if you want me to go first, Sergey.

5:04 Uh, it was it was awesome.

5:06 Actually, it was my favorite conference I've ever been to.

5:08 Not just saying that because I'm on the Nvidia stream,

5:10 but I felt like the level of technical

5:12 depth was like really what I was expecting.

5:15 And yeah, dayto-day, I mean, we had like lots of talks that we'd attend.

5:19 They were, you know,

5:20 obviously like Open Claw and like Nemo Claw was like big topics

5:24 of discussion um for like people who are just learning about agents.

5:28 Um obviously Sergey and I we we build agents as part of our job.

5:33 So it we are a little bit more technical but um yeah like they

5:37 we went into like all kinds of different

5:38 talks with like baseline knowledge of what we expecting and then they dove deep

5:44 into like reinforcement learning or even like

5:46 stuff on like the video game side like on gaj and splats and whatnot.

5:50 Um, so yeah, it was just lots of like information that was coming

5:54 at us along with like just great ways of like networking with people.

5:58 Um, we got the inference engineering book from uh,

6:02 one of the engineers from base 10.

6:05 Um, which is really cool,

6:07 a signed copy of that which actually let me

6:10 like started like exploring that on my own, you know.

6:12 So, it's pretty cool like how a random idea might just pop in your head

6:17 by attending there that you might not have had um going into it.

6:21 So, [snorts] you you come in, you get some lessons and even some

6:23 inspiration walking out as to what you're able to do next, it sounds like.

6:27 Yeah.

6:30 Concise way of saying that.

6:31 Yeah, pretty.

6:32 Okay.

6:32 No, it's Yeah, that's that's perfect.

6:34 And I actually do want to dive

6:35 a little deeper into the hackathon itself because again

6:38 a lot of people uh have heard a lot

6:41 about hackathons but haven't done it uh themselves.

6:43 So what was that 2-hour hackathon like?

6:46 Uh walk us through that from the line before you

6:49 got there to what kind of uh you know breakfast treats

6:53 you chose to uh how much was iteration and how

6:55 much was actual execution in the two hours that you had.

6:58 countdown like on a big screen uh just feet away from you.

7:04 Sure.

7:04 I can give like a short short my my my feeling how it how it went.

7:09 Um so the first first of all we were not planning even to go to hackathon.

7:13 It was like the idea which pops us in our head like

7:15 a day before and when we were staying in line to enter the building

7:19 we just saw like 15 people and then we we we just thought

7:24 that maybe we should we we can get the price not even like participating.

7:27 So everyone gets a prize in this line and then like 5 minutes after when

7:31 we entered it just like it was over 100 people like lots of teams uh competing.

7:37 Yeah.

7:37 It was it was really like um impressing impress.

7:41 Yeah.

7:42 Um but uh what so the structure what what we did first is

7:48 uh of course we discussed the idea and uh we also we discussed how

7:52 to split the work clearly so we don't step on each other's toes

7:56 and uh the thing we we decided is that we will be running we

8:02 will be building the same project in parallel so will be working

8:07 on his project I will be working on mine and then at the end

8:10 of the hackathon we will just compare what's what's the best what we can

8:14 show and then uh and then show to the to the to the audience.

8:19 Um now that's a perfect way to divide and conquer figure out what works best.

8:26 Now in any kind of corporate environment that would be like really rough like

8:30 if you have two teams pitched against each other for the best uh idea.

8:33 But when you're friends and you're doing it,

8:35 that's actually a pretty novel way to get to the highest

8:39 quality uh most compelling pitch and kind of double your chances.

8:43 Exa.

8:43 Exactly.

8:44 That and we did it because we will be

8:47 we we were using AI for writing code and it's

8:50 really hard to like get a boundary you know

8:53 to just like split the functionality for like each of us.

8:58 So we thought that like AI can generate like everything in one shot.

9:01 We don't want that but still it will be like really difficult to like split

9:04 responsibilities between us and then yeah we we

9:07 we thought that like generating the whole project

9:10 and then just choosing the best one would be a better idea and that way when

9:16 we don't then thought about that we could

9:19 just have be in the separate teams as well.

9:24 That's interesting too but I think actually

9:26 there's a great benefit to working together.

9:29 First of all, in that two hours,

9:30 it's a compressed time, but you have that camaraderie.

9:33 And also, I think it really is the best

9:34 way to double your chances because at the end

9:36 of that, you have to decide which one is the the one you want to go with.

9:40 And I won't ask who is the one who uh

9:42 provided the winning one because you're both winners in this one,

9:44 you know, but u uh it's a smart way to do it.

9:48 And uh that leads me to want to know for people that are participating,

9:53 what are the tips you'd have coming away

9:55 from this for being successful at a VIP pack, something that's two hours?

10:00 Uh is it uh you know a specific way you two work together?

10:04 If do you want to extrapolate on that?

10:06 Or uh you know just uh things to bring to the table in advance.

10:09 What what do you think contributed

10:11 to your winnings that other people could learn from?

10:14 I think uh one thing too is like for example um uh as Sergey mentioned like we

10:19 both like worked on our own branches like

10:21 um but we we we were collaborating right beside

10:24 each other the whole time like we're talking about

10:25 what models we're going to use and in fact

10:28 the first 40 minutes of the actual hackathon

10:31 we actually spent time looking for the right models.

10:34 So we're on the and we're like trying to like

10:37 formulate our problem statement like more like articulate it better.

10:41 So I think for as a takeaway like

10:44 actually understanding what your like your idea in like

10:47 enough detail that you can then write like

10:49 a comprehensive spec to help guide your coding

10:52 agents and it doesn't have to be insane

10:55 levels of detail but you should understand like

10:57 the boundaries of like what your idea is um and what you're not going to do

11:01 because like for example we wanted to have

11:03 a 3D talking avatar that while we're looking

11:06 for the models to do that we're like actually this might be out of scope like

11:09 we don't think we're actually going to be able to do this re reasonably well um

11:13 within like an hour and a half um

11:15 time remaining because we're spending all that time researching.

11:19 So yeah, really being able to articulate

11:21 your problem statement during a hackathon.

11:23 Um obviously you're going to be moving very fast so having some level

11:27 of like a sturdy foundation to like then build upon is like really helpful.

11:34 Great.

11:34 And Sergey, anything to add to that?

11:36 Um I would start with the challenges we faced before I answer the question

11:41 just like yeah and uh I I I see three challenges like the idea

11:45 so you come up with a strong idea quickly or like even have

11:48 this idea before the hackathon and then

11:51 teamwork as I already mentioned like divide

11:53 the work divide the work clearly don't step on each other's toes and then

11:58 of course timeline time uh build the full pipeline end to end that's

12:02 that's the the the most important challenge and It should be working because

12:07 the judges will come to your table and you have to show the working solution.

12:12 Yeah.

12:12 So uh yeah and for that we already had an idea.

12:15 We decided really quickly how to split the work

12:18 and to our delivery just like you know it's uh scope control.

12:22 We just we just understand that we

12:25 would not be able to implement everything which

12:27 we have in the list and just implemented

12:30 the the most important parts of the project.

12:34 So, what was the judging experience like for you?

12:37 Because something that I think people only understand

12:40 once they get to the end of the whole

12:41 process is just how important it is to know how to articulate the high concept.

12:46 Because obviously, you're not going to have

12:48 something shippable in two hours, right?

12:50 You'll be able to demonstrate that you have a technical aptitude,

12:54 but a big part of winning these events is being able to communicate what

12:58 the idea would become if it was going to be a shippable product, right?

13:01 So, how did you manage that?

13:02 What was it like to pitch it?

13:06 Uh, you want me to go, Sergey?

13:09 Um, I'll start.

13:10 Yeah, sure.

13:11 I was going to say like it was pretty intense.

13:13 Um, like the judges came over to our table maybe

13:16 like the within like the first few minutes like when judging

13:19 started and seeing them like talk to other like tables

13:22 and then you feel like a little bit anxious like okay,

13:25 they're going to come to us and like how do we actually explain our idea?

13:28 Um, I think going back to like it's almost like the same kind of like foundation

13:34 you use for your coding agents to actually

13:36 build what you're actually envisioning in your head.

13:38 You have to almost like take that same like

13:41 representation but then like maybe like reduce some levels

13:44 of detail and then just say that to the judge

13:47 like this is what we're trying to build.

13:49 This is what we actually built.

13:51 Um, yeah.

13:53 Yeah.

13:53 It's uh it's hard to do that because actually in some

13:56 ways it would be hard easier to do a long description, right?

14:01 I I remember Hemingway said uh I uh

14:05 didn't have time to write you a short letter, so I wrote you a long one.

14:09 Right.

14:09 It's one of those things.

14:10 Brevity is the wid of the soul.

14:12 That's right.

14:13 That's right.

14:13 And uh you know something else I look at is when you

14:17 have more team members like you could have had as many as four.

14:21 Would that have been a benefit or would that have

14:24 just slowed you down as opposed to the team of two?

14:28 I think slowing down and Sergey,

14:30 you could touch on this, but I think the reason why we

14:32 also wanted to divide the work the way we did was because

14:36 it it's really hard to coordinate when the velocity of shipping is

14:40 so fast and there the time frame is like really uh tight.

14:45 um because then like you're spending more

14:47 time like deciding how to like divvy things up and like there's going to be

14:51 bottlenecks being formed with like a larger team.

14:54 So I think actually what we saw was I don't think there was

14:57 any like fourperson team that uh um yeah I I think that there

15:04 was pro some teams but um you know I think it is harder

15:08 to coordinate like to form like a a really good solution towards the end.

15:12 Yeah.

15:12 And I know that you weren't the only ones that were small teams that won.

15:16 So that was interesting to see as well.

15:18 And I think the takeaway is well

15:19 certainly you can be successful with four people.

15:21 If you're going to these events and you just have one friend that you're

15:24 a co-orker with and can go to a hackathon with, that's going to be enough.

15:28 Don't feel like you'll not be able

15:30 to compete because you don't have a full-size team.

15:35 100%.

15:35 I think with AI it just shows you how much like you as an individual

15:38 as well as like a smaller team like you and your friend can like get.

15:43 I I think I think two people team is is the best because you're not alone.

15:49 You can like share your thoughts.

15:50 You can discuss this but uh at the same time you're not that you're

15:55 not spending too much into uh in in in the discussions

15:58 you know and trying to convince something that like

16:01 this approach would would work best in this situation.

16:04 So, I was the guy who went up on stage after

16:08 the two hours had uh filtered down and said, "Pencil's down.

16:13 We can't do any more hacking and give us some time to look

16:18 at the projects with the judges." And then when we announced the winners,

16:24 tell me what that experience was like because you

16:26 had no idea whether you're going to win or lose.

16:27 You're sitting there with all these other people.

16:29 When you heard your name, what was that like?

16:32 What was it like to feel kind of the before moment

16:35 when you had no concept of what the conclusion would be?

16:38 Tell me all about that.

16:41 I can I can start I can I can summarize

16:43 it in two sentences actually because I was not expecting

16:46 us to win and uh when we were announced it

16:49 just like it it was it was something special for me.

16:52 I already closed my laptop and even even turned it off and then someone said

16:57 yeah just you can you should go quickly on the stage and then show everyone.

17:01 It was it was really hard for me

17:03 to even find the the project we were working on.

17:06 Uh yeah, I don't know how you feel how you felt, Derek.

17:11 Yeah.

17:11 Um same.

17:12 I also like was about to close my Mac and uh you know,

17:16 like just hearing our names being mentioned.

17:18 Well, our actually esports uh caster and yeah, it was just kind of surreal.

17:23 I was just like, "Oh my god." Like really awesome.

17:27 Hell yeah.

17:28 So, so you won and everyone was clapping.

17:31 What happened next?

17:34 So, there was like a little bit of uh like uh you

17:37 know like I guess setup from like the judges like on like you

17:40 know the presentation details like going on stage to present your uh idea

17:44 and like actually like you know collect the the signed 5080 from Jensen.

17:50 So yeah, I mean we went on stage and uh Sergey

17:54 um was doing the live demo and uh Sergey runs Linux.

17:57 So we had a little bit of a a hiccup

17:59 when it came to like the HDMI like streaming, but uh we got it to work.

18:04 Um so that's great.

18:06 And where where is the 5080 signed by Jensen now?

18:09 Did you guys end up uh installing it or using it as a trophy?

18:12 Like what what was the decision?

18:15 It is it it is a trophy.

18:16 It is a trophy.

18:17 We gave it to uh to the company to market gaming.

18:20 Yeah.

18:20 And it's Oh, nice.

18:21 Very good case.

18:22 Yeah.

18:22 It's in like a glass box with our our names engraved and

18:28 hopefully inspire more hackathons.

18:29 Yep.

18:31 Well, so I also want to know like, okay, you won.

18:34 You got the prize.

18:35 And who were the first people that you talked to to let them know that you won?

18:40 Like outside of YouTube,

18:41 like you must have called someone or let folks know like

18:44 what uh what was uh the first person to hear about it?

18:50 We actually had some other uh co-workers that uh

18:52 and our boss that actually came um to GTC.

18:56 So, that was one of the first things we're like, "Guys,

18:58 you're not going to believe it." Cuz they came

18:59 in the morning um to actually see us like get set up,

19:03 but they didn't get to like see us actually like win or present.

19:06 So, we just messaged them like, "Hey guys, like you're not going to believe it.

19:10 Like, we won." That's totally awesome.

19:13 And then showing them the pictures.

19:14 But then I I got to know what it

19:16 was like to actually show your boss the physical GPU.

19:20 You want to go Sergey?

19:22 Um yeah, actually it they were really surprised about

19:27 the picture as well that we won the the 5080 and

19:32 yeah um the other there are people we informed of course like our relatives,

19:37 wives, kids, they were also like really happy about that.

19:43 [snorts] It's it's really a pretty incredible achievement

19:44 and still amazing to just look at that project

19:47 and I think one of the things that really

19:49 makes it sing is that it's something that when

19:52 you think of impact as a gamer myself

19:55 that's something I would love to have because I

19:57 could imagine having some boundaries in terms of like

20:00 the level of snark that the announcer has.

20:03 would love to be able to figure out do I want one

20:06 that's like just super excited to cheer me on or someone who's

20:10 kind of negging me and uh you know making me feel like

20:14 uh I have to work harder or you know whatever it is.

20:16 It's like there's so much space that a person's imagination just

20:21 kind of fills in all the amazing things you could do

20:23 with this idea that's so good that it's only obvious in retrospect

20:27 which is what all the best ideas I think uh come down to.

20:31 And but yeah at the same time there are like lots of areas to improve.

20:36 So if you were working on this project for a bit of a little

20:40 bit longer it will be I I would work on like synchronization

20:44 a little bit and caster personal is probably better voice quality because we

20:49 were using like the the smallest uh TTS model with that we can find.

20:55 It's really popular on hugging face but still and then yeah as you said like

21:00 ever lip sync and simple body motions would

21:02 also like add uh like a better experience.

21:07 Well and I'm sure you both love what you do.

21:09 You're working in games.

21:10 You're working together on fun stuff with lots of visuals.

21:14 Uh but it is nice.

21:15 It has to be nice as an engineer to be able to step away for just

21:18 a few hours and do something completely different

21:21 to reset and say I am at a hackathon.

21:24 I'm going to try something that I've never done before and I'm going to share it

21:28 with the team of uh other engineers who

21:31 are also working hard to share their ideas.

21:35 Yeah, I think I think that's the thing I kind of like about

21:38 hackathons even though this is like the second one I've ever been to.

21:41 Um is that like yeah,

21:42 just the energy of like other engineers just building things and like

21:45 you guys are all going to show something towards the end.

21:48 Um, and just the level of collaboration

21:50 that happens with like your teammates on something

21:52 that isn't like a problem that you may work on like every single day.

21:57 Um, just lets you like think outside of the box

21:59 like okay well we're you know we have

22:01 a little bit more like creative freedom in this like

22:04 it's not purely just for a specific reason.

22:07 Um, so yeah, I think just the the creative like

22:11 opening up of your mind like while building and just

22:14 having more of a uh atmosphere around you that's everyone

22:17 else is also having the same kind of like experience.

22:20 It's pretty awesome actually.

22:22 Well, it's awesome for us, too, because what we really want to do is not just

22:29 get this type of hardware and software into people's hands,

22:32 but uh build a community.

22:34 And it starts with folks like Tariq and Sergey here who

22:38 not only were a part of the hackathon and were at GTC,

22:40 but were kind enough to take time weeks later

22:44 to sit down with us and give us some details.

22:46 So, uh, guys, I'm looking forward to seeing what you do next,

22:49 and I really appreciate you taking the time today to sit

22:52 down with us and give us the details of, uh, how the event went for you.

22:58 Thanks for having us.

22:59 Thank you.

22:59 Yeah, thank you, Eton.