Can we fix this AI agent in 60 minutes? (Live builder Q&A)

4:00 LUIS SALA: Hello, and welcome, everybody,

4:02 to the "AI Agent Clinic." This is our first episode,

4:06 and it's a worldwide premiere.

4:07 We're going to take your agents and refactor them for production.

4:10 My name is Luis.

4:12 JACOB BADISH: And I'm Jacob.

4:13 Super excited to have you here for the first episode.

4:15 LUIS SALA: All right.

4:16 We're going to be hanging out in the live chat right alongside with you.

4:19 So we're going to answer any questions you might have throughout the broadcast.

4:22 So don't be shy.

4:23 Please go ahead and drop them in.

4:25 JACOB BADISH: Yeah, and please stick around till the very end.

4:28 We'll be doing a live technical Q&A, doing a behind-the-scenes code review.

4:31 It's going to be a lot of fun.

4:33 I think for now, let's roll the tape.

4:36 I vibe coded this in the evenings and weekends.

4:39 I was blown away at how doable it was.

4:42 If I can do it, anyone can do it.

4:43 LUIS SALA: You've already identified the one

4:45 superpower you want to give your agent.

4:47 FRANK: OK, builders, your time starts now.

4:49 LUIS SALA: OK, I don't know about you, but I got to get set up.

4:52 Let's plug in.

4:54 And structured messaging is the best way to connect one system to another.

4:59 And so I'm very, very proud of you, frankly, for doing this.

5:01 JACOB BADISH: Well, you should be proud

5:03 of Gemini because— [LAUGHTER] FRANK: Time is up.

5:05 LUIS SALA: No!

5:06 FRANK: Hands off keyboards.

5:08 JACOB BADISH: I thought we were going to get this done in about 30 minutes.

5:10 LUIS SALA: I think the problem is I talk too much.

5:16 Welcome to the "AI Agent Clinic," where we

5:19 work side by side to triage your broken builds,

5:22 untangle your code, and help you get your agents production ready.

5:25 Joining me in the clinic today is Jacob Badish.

5:28 Jacob, welcome to the show.

5:30 JACOB BADISH: Thanks, Luis.

5:31 Appreciate the time.

5:32 This is awesome.

5:32 LUIS SALA: So, Jacob, if I understand correctly,

5:34 you have no technical expertise, and yet you built an agent by yourself.

5:38 Tell us about it.

5:38 JACOB BADISH: Yeah, that's right.

5:40 It's called Titanium.

5:41 And I vibe coded this in the evenings and weekends.

5:44 I was blown away at how doable it was.

5:46 If I can do it, anyone can do it.

5:48 What Titanium does is it finds certain executives at our customers and partners,

5:53 figures out pain points that they're struggling

5:56 with, challenges that their businesses are facing,

5:58 and ties it back to strengths of Google's in ways that we can help them.

6:01 LUIS SALA: So it researches the customers.

6:03 JACOB BADISH: Yep.

6:04 LUIS SALA: And then it generates what, an email?

6:06 JACOB BADISH: It does, yeah.

6:07 It provides research, messaging, and a really clean email.

6:12 And again, it's really to save the executives' time.

6:15 I know that you're struggling with A, B, C.

6:17 Google's really good at doing X, Y, Z.

6:19 Let's sit down and have a conversation.

6:21 And it uses— I've actually hard coded— this is

6:24 probably not the best way to do it.

6:25 But I've just hard coded in about 10 case studies.

6:29 So Google has all these public case studies.

6:31 I picked 10 of them.

6:33 And I was noticing that it would use

6:35 the same data points in these emails and messaging.

6:38 And it's probably not the best way to be doing that.

6:40 LUIS SALA: All right.

6:41 So it sounds like you've already identified

6:43 the one thing you want to give your agent.

6:45 Let's elaborate on this.

6:46 So if I understand correctly,

6:48 Google has over 1,000— actually 1,600 case studies.

6:51 And you want to be able to leverage those instead

6:53 of the 12 or so hard-coded case studies that you have in there?

6:57 JACOB BADISH: Yeah, exactly.

6:59 It's a good tool that works for me.

7:01 It's not really scalable.

7:02 I can't roll this out to my team quite yet.

7:04 And because of the limited case study set, it's not as robust as it could be.

7:10 And I think if there's a way to dynamically

7:12 do this and have it really understand the company

7:14 that we're trying to work with and help

7:16 tie it back to a really relevant case study, that's just a lot more valuable.

7:21 LUIS SALA: So now we know what you need to take

7:24 this agent and release it to your colleagues and get it production ready.

7:27 JACOB BADISH: Yeah.

7:27 LUIS SALA: So we've given ourselves 60 minutes

7:29 to fix this agent and then give it superpowers.

7:33 All right, Frank, what do you think?

7:34 Can we get started?

7:35 FRANK: OK, builders, your time starts now.

7:39 LUIS SALA: OK, I don't know about you, but I got to get set up.

7:42 JACOB BADISH: Yeah.

7:43 LUIS SALA: Let's plug in.

7:44 JACOB BADISH: You'll walk me through what we want to do?

7:47 LUIS SALA: That's right.

7:47 JACOB BADISH: OK.

7:48 LUIS SALA: I see that you have a function called orchestrate all.

7:52 And what it looks like it does is that it goes through

7:55 a loop where it goes through every company that you intend to target, right?

7:59 It launches a set of parallel tasks.

8:01 JACOB BADISH: Yep.

8:02 LUIS SALA: Wow.

8:03 JACOB BADISH: Yep.

8:04 LUIS SALA: That is awesome.

8:05 So you are actually performing what's called a fan out,

8:09 where you are executing these research tasks

8:13 not on a single company in sequence, so not one at a time,

8:17 but you're actually researching multiple companies at the same time.

8:20 Essentially, we can think of them as subagents,

8:23 each one responsible for processing a single account

8:26 and passing along certain information like who the customer is,

8:30 what the SDK client is, and so forth and so on.

8:33 So that's beautiful.

8:34 And as it executes, you start collecting those results,

8:38 and you're going to assemble them.

8:40 JACOB BADISH: Yeah, and this came about—

8:42 it would take 15 minutes to run originally.

8:45 And this is where I got so much better at working

8:49 with Gemini and prompting because you just go back and forth,

8:52 and how can we improve this?

8:54 And it would say, this is a really rudimentary way to do this.

8:57 We can do X, Y, Z.

8:59 And I would, again, collaborate through.

9:01 OK, let's try this idea.

9:02 That sounds good.

9:03 No, I don't want to do this.

9:04 Rewrite the code.

9:05 Redeploy the code.

9:06 Test it.

9:07 So it was all— I have exponential backoff.

9:10 There's clever ways of interacting with the Gemini APIs.

9:15 So it was able to build— again, it's a simple app,

9:18 but there's a level of robustness that the AI

9:21 has just taught me how to implement along the way.

9:24 LUIS SALA: I really love it.

9:25 And the fact that you're throwing out terms like exponential backoff,

9:28 it shows that you understand what the fundamental challenge was,

9:34 and that is that, for example, calling LLMs sometimes is unreliable.

9:38 It's an unfortunate fact of life, right?

9:41 Sometimes you might get rate limited,

9:43 or sometimes your Wi-Fi might disconnect or something like that.

9:46 So building reliability into your agent is important.

9:50 FRANK: All right, folks, 10 minutes past.

9:52 LUIS SALA: Already?

9:53 JACOB BADISH: Oh, man.

9:53 OK.

9:53 LUIS SALA: Oh, no.

9:54 OK, so we got to get started with the coding,

9:56 but let's finish that section very quickly here.

9:59 OK, so I see here this is the actual meat of this.

10:02 This is how you call the model.

10:03 And so once again, you're using the Gemini SDK to connect.

10:07 And you're passing it along— a configuration

10:09 to use grounding with Google Search.

10:11 Bravo on that.

10:13 You also set the temperature low.

10:14 This means that the model is going to generate less random answers,

10:20 which is important for something that's factually grounded, so brilliant.

10:25 Love it.

10:26 And if we scroll a little further up,

10:28 we see— aha— a function called verify intel.

10:31 It is structured very similarly to the previous one.

10:35 However, your prompt is basically kind of saying,

10:37 hey, we're trying to ensure that we're not hallucinating.

10:40 Well done.

10:40 Bravo.

10:41 This is great.

10:41 And I see it uses the same basic techniques as before.

10:44 That's it.

10:45 Oh, and here are your case studies.

10:47 I see.

10:48 All right.

10:49 So let's go ahead and think about what the next version should look like.

10:53 If we think about the version one that exists today,

10:57 you launch it, it's going to research a company, it's going to draft an email,

11:01 it's going to verify everything and then gives you the final content.

11:05 When you're researching, you're also providing just this list,

11:08 this hard-coded list, of case studies.

11:10 When you're developing an agent, it is absolutely,

11:14 perfectly a legitimate tactic to use the native SDK,

11:18 such as the Gemini SDK or Gen AI SDK that you're using today.

11:23 However, it occurs to me that in order to set yourself up

11:27 for success in the future for better performance out of your agent,

11:31 it might make sense to shift over

11:34 to a specialized SDK specifically designed for agents.

11:38 And that would be the Agent Development Kit would be my recommendation.

11:42 We move on to a V2 plan that uses ADK, but it preserves the existing workflow.

11:50 So it actually would replicate your V1 functionality.

11:54 And we're going to take baby steps.

11:55 Just like in any software development cycle,

11:58 we want to start with the basic functionality.

11:59 And then we want to add one feature at a time

12:02 until we have the full thing up and running.

12:05 So while we would start with a replica of version 1 using ADK,

12:10 which ultimately will give us something that's

12:13 more maintainable and more robust to build

12:16 on top of, then we can then start breaking out the workflow a little bit,

12:22 changing it around, so that we can add additional functionalities,

12:26 such as— my laptop just froze.

12:30 [MUSIC PLAYING] Folks— yeah.

12:38 So what we're going to do is we're going to use some prompting techniques

12:41 to create a faithful replica of your existing

12:44 agent in a way that works with ADK.

12:47 So let's go ahead and just build a prompt that does this.

12:50 JACOB BADISH: Yeah.

12:51 LUIS SALA: What's your first instinct?

12:52 How would you phrase it?

12:53 JACOB BADISH: I would just describe what we're trying to do.

12:56 So I would just start typing and say, I need to port this over into ADK,

13:01 create a plan to do this, and see what

13:04 it comes up with, and just go step by step.

13:06 LUIS SALA: Beautiful.

13:06 That's exactly right.

13:07 The idea of creating a plan is vital.

13:09 We don't want to just start coding without having at least

13:13 an idea of what the agent is going to do for us.

13:16 And we're talking about the coding agent right now,

13:18 not the Titanium agent that Jacob is building.

13:21 JACOB BADISH: Something I learned, I always now add in, then reverify your work.

13:26 Make it go back a second time because it catches things that it misses.

13:31 LUIS SALA: Now, I notice that as your agent is beginning the planning process,

13:36 it's saying, I analyzed skill.md.

13:39 The ADK team recently released a set of ADK dev

13:44 skills that can work with a wide variety of coding agents,

13:48 not the least of which is Antigravity,

13:51 but it also works with Gemini CLI, Claude Code,

13:54 and a wide variety of other agents that teach the agent,

13:57 the coding agent, how to develop ADK applications.

14:02 So yeah, so that's what it's doing.

14:03 Now, I see that it is displaying a plan.

14:06 Let's take a look and scan that plan.

14:08 So I see it's going to generate an agent.

14:12 Verify the agent.

14:13 Use a pipeline.

14:14 JACOB BADISH: So it looks like it's taking the functionality in my code and just

14:19 creating agents for each of those tasks

14:21 that I would have Gemini do, for example.

14:23 LUIS SALA: Yeah, and it's proposing wrapping

14:25 them together in what's called a sequential agent,

14:27 which a sequential agent then executes steps 1, 2, 3, 4 in sequence.

14:32 That's in the name.

14:33 JACOB BADISH: Got it.

14:34 I'm going to start.

14:35 LUIS SALA: Yeah, so go ahead, and you start coding.

14:36 In parallel, what I'm going to do is I'm going to create a crawler.

14:40 So the intent here is that we're going to crawl a particular website.

14:43 So I'm going to say, create a crawler that uses— and I've installed Playwright,

14:52 which is a headless browser automation software— uses Playwright CLI.

14:57 [MUSIC PLAYING] All right.

15:02 While that's happening,

15:03 I'm going to start getting a couple other things ready so that we can

15:08 accelerate our development because I'm sure Frank

15:10 is looking at me, giving me the eye.

15:13 JACOB BADISH: And I'm making progress here.

15:15 It's building out the new code in ADK.

15:18 And this is still Python code, which is great.

15:20 LUIS SALA: Excellent.

15:20 And at the moment, everything looks good.

15:22 I'm just going to remind it there's a .env file

15:28 with the necessary variables to connect

15:34 to GCP and Application Default Credentials.

15:43 All right.

15:44 So it's progressing.

15:45 JACOB BADISH: Great.

15:46 LUIS SALA: Little hiccups that you're finding are natural.

15:48 They're going to happen.

15:49 JACOB BADISH: OK.

15:49 LUIS SALA: Frank, dare I ask, do we still have time?

15:52 FRANK: Perfect timing, Luis.

15:53 You have 20 minutes left.

15:56 LUIS SALA: OK.

15:57 No pressure.

15:58 JACOB BADISH: Yeah.

15:59 LUIS SALA: All right.

15:59 In the meantime, I am observing that my coding agent is— actually,

16:05 it started crawling.

16:07 So looking at the terminal output,

16:10 you see here where it says "clicking show more?"

16:14 So if we look at the actual website, notice this.

16:17 This is the website that we're crawling.

16:19 There's a search box on the top.

16:21 And then there are roughly 12 case studies that it shows.

16:25 And then the Show More button— JACOB BADISH: Yeah.

16:26 LUIS SALA: Right?

16:27 So there they are.

16:28 So if I click Show More, it shows me another 12, right, and another 12.

16:32 So what's actually happening right now is that the agent

16:36 developed a tool that goes to the website.

16:39 And it's actually manually clicking— JACOB BADISH: That's cool.

16:43 LUIS SALA: --that button and figuring out the links

16:45 to every single one of these case studies.

16:48 And once we have the links, we can then download them all.

16:51 That's pretty darn cool.

16:53 And so we can start moving on to the next step,

16:55 which is that we're going to take this ingested data or this downloaded data,

17:00 and we're going to start figuring out how to write it to a vector database.

17:05 And to write it to the vector database, we need to first create it.

17:08 And so I can see already that the agent is trying to figure that part out.

17:12 So we'll let it do that on its own.

17:15 OK?

17:15 JACOB BADISH: Got it.

17:16 LUIS SALA: Yeah.

17:17 So we're doing really, really well.

17:19 [MUSIC PLAYING] Oh, slash agent?

17:25 FRANK: 10 minutes, folks.

17:26 10 minutes.

17:26 LUIS SALA: Oh, come on.

17:28 All right.

17:28 So we're inside the project.

17:30 We have improved agent.

17:32 We have a root agent declared here.

17:34 This is exactly right.

17:35 What I find interesting is that it did

17:37 a slightly different approach than I would, but that's correct.

17:41 It's just, once again, slightly different.

17:44 Key thing is that it's declaring a root agent,

17:46 and that is what we care the most about.

17:48 JACOB BADISH: Great.

17:49 LUIS SALA: All right.

17:50 In the meantime, I think we've got

17:51 to start getting ready to push your code, actually.

17:53 And give it to me.

17:54 There it is.

17:54 [COUNTDOWN BEEPS] This— FRANK: And then time is up.

18:02 LUIS SALA: No!

18:03 FRANK: Hands off keyboards.

18:05 Off keyboards.

18:07 JACOB BADISH: Down to the wire.

18:09 LUIS SALA: Indeed.

18:09 JACOB BADISH: I was a little more confident.

18:11 I thought we were going to get this done in about 30 minutes.

18:14 But there's always little things that go wrong that you have to work through.

18:17 LUIS SALA: I think the problem is I talk too much.

18:19 So all right.

18:20 Do you want to try to see if this works?

18:22 JACOB BADISH: Let's give it a shot.

18:23 LUIS SALA: Type it in, sir.

18:24 You do the honors.

18:24 JACOB BADISH: Yeah, let's see how this works.

18:28 Give it a spin.

18:29 LUIS SALA: All right.

18:30 A simple prompt, a short prompt,

18:32 just basically targeting the company and the role.

18:34 And with that information, we start seeing what's happening.

18:37 Oh, my gosh.

18:38 So it's actually running a bunch of Google searches.

18:40 JACOB BADISH: Wow.

18:41 LUIS SALA: If we click here, we can mouse over,

18:43 and it says, execute a search for current CTO of the company.

18:47 We should start seeing some results in here.

18:51 So these are the different function calls.

18:53 Look at all these different searches.

18:54 JACOB BADISH: Yeah.

18:56 LUIS SALA: So it's executing the searches,

18:57 and then it's consolidating the response.

19:00 JACOB BADISH: Yep.

19:01 LUIS SALA: The initial response.

19:02 So this is the initial research

19:04 on the company and what their priorities are, right?

19:06 And then it's executing vector search.

19:09 But what you're seeing here on the side, these are the case studies.

19:14 So as we scroll down, we start seeing, then,

19:17 the consolidated list of case studies

19:19 that have been selected as the preferred ones.

19:22 And then finally, we start generating your final punchy email.

19:29 What do you think?

19:30 JACOB BADISH: This is awesome.

19:31 I mean, taking it from a tool that I'm using

19:33 as really a pilot into now something that's much more production ready,

19:37 scalable to my team, I'm much more confident talking about ADK

19:41 and these tool sets with my customers and partners.

19:44 The accuracy and value that this can bring now,

19:47 having a vector database really up

19:49 and running in a meaningful way, it's incredible.

19:51 Thank you.

19:52 LUIS SALA: Well, thank you.

19:54 And this brings us one step closer to bringing this project to production.

19:58 Let's pause this right here.

19:59 We forgot one critical thing that would make this agent production ready.

20:04 Can you guess what that was?

20:05 It's the user interface.

20:06 This is what the new user interface would look like,

20:09 allowing us to enter the target company and their domain.

20:17 Once we get the research complete, we can copy paste it and send it off.

20:22 What's next?

20:23 Well, we're going to put some polish on the code,

20:25 make this all readable and easy to understand.

20:28 We will also author a blog post.

20:30 So you can find links in the description to the fully finished code,

20:34 the ADK project, and a blog post about everything that we cover today.

20:38 So thank you so much for joining us.

20:40 And thank you, Jacob, for joining us.

20:42 And we'll see you next time on the "AI Agent Clinic." Well done.

20:47 JACOB BADISH: Nice, that was great.

20:48 LUIS SALA: Very good.

20:48 JACOB BADISH: Good job.

20:49 LUIS SALA: Let's get back to work.

20:51 FRANK: Well, Luis did great today.

20:54 He clearly was very clear with his approach.

21:00 And he's like a teacher.

21:01 He explained the concept really clearly.

21:03 And he's very patient.

21:05 But he did spend 20 minutes before actually start coding and building anything.

21:10 And so my advice to the future host

21:13 is trying to get to the production-building phase quicker.

21:20 LUIS SALA: Indeed.

21:21 I probably talked too much.

21:24 But we're back.

21:24 It's so, so great to be back with you after

21:27 that first look at the "AI Agent Clinic." Thank you for chatting,

21:31 by the way, and asking your questions.

21:33 Jacob, I have a question for you.

21:35 Have you recovered from all that craziness?

21:38 JACOB BADISH: No, that was so much fun.

21:39 Yeah, Luis, I'm still hanging in there.

21:41 I have to say, the energy in the chat, it's been a lot of fun, really fantastic.

21:45 I've loved seeing tons of questions coming in, comments, a lot of support.

21:49 And a lot of the questions, I actually had myself.

21:51 So it's nice to see I'm not the only one.

21:54 LUIS SALA: Yeah, well, it's now time to pull back the curtain.

21:57 So we're going to move into our technical deep dive to unpack a little

22:00 bit more detail as to how we built and broke and deployed this agent.

22:05 So, Jacob, why don't you take it away?

22:08 JACOB BADISH: Yeah, perfect.

22:09 And I've already started pulling together a couple of questions from the chat.

22:13 Let's jump into the first one.

22:17 In the video, we see the agent figuring out the vector database ingestion,

22:22 but we don't get to see the actual build.

22:25 How exactly did you architect the vector store behind the scenes?

22:29 Did you use a managed service like Vector

22:31 Search or a local instance like Chroma DB?

22:35 And what was your chunking strategy for those case studies?

22:38 So there's a lot of meat here, Luis.

22:40 Maybe start with a simple explanation.

22:43 LUIS SALA: Yeah, absolutely.

22:45 So indeed, we did not have too much opportunity to dive into what was developed,

22:50 so let's talk about that.

22:52 The vector search engine that we used was Google's Vector Search,

22:57 specifically the 2.0 version that was released in the last month or two.

23:03 And this readme that you're seeing here is actually what the development

23:07 agent that I was using created as it was planning the process.

23:12 And so what we did is that we first needed

23:16 to create a crawler that could crawl a specific website,

23:22 and then that crawler then needed to— one second here.

23:27 That crawler then needed to be able

23:29 to actually retrieve the content from the website.

23:33 So this happened in two phases.

23:35 First, we go to the website that we were crawling, and that's this one here.

23:41 And we needed to extract the URLs

23:42 of all the different case studies that were there.

23:45 Once we had a list of URLs,

23:47 then we would go through a content extraction process,

23:51 where we would grab the HTML from the specific

23:54 case study and create a markdown version.

23:57 So that was a two-step process.

23:58 Once we had a markdown version,

23:59 then we stored it inside the Google Vector Search database.

24:05 I could have used, and we could have used,

24:07 a locally installed vector store such as Chroma DB or Pinecone

24:13 or any of a myriad of other similar services out there.

24:17 But we chose to use the Google vector store.

24:21 So once that happened, then the next part comes in.

24:25 We've crawled the data.

24:27 We have stored it inside the vector database.

24:29 Now we need to expose this to the agent.

24:31 And so we exposed vector search functions to the agent that allows them

24:37 for the content to be read and then processed and presented to the user.

24:42 So let's actually look at that code very quickly here.

24:46 This is the crawler, right?

24:48 And the crawler, as I mentioned, executes in two phases— first,

24:52 discover the different URLs, and that's what you're seeing here.

24:55 And so it would load a page.

24:57 It would then click on the More button

25:00 while grabbing all the URLs that were present there.

25:04 Once it found those URLs, it extracted them.

25:07 That's what you're seeing here.

25:09 And they got stored in memory and then saved to a file.

25:14 And that's what you're seeing here.

25:15 So the URLs are now stored.

25:18 Now we can then move on to phase two,

25:21 where we extract the actual content from the pages.

25:25 So we would fetch the contents of the individual URLs.

25:29 And then in here, what's cool is that we're

25:31 using Gemini itself to pull out the content,

25:36 reformat it, and prepare it for insertion into the vector store.

25:42 So this is the model call just using the regular Gemini SDK client.

25:49 And once we received our response, which we used JSON conversion that's built

25:54 into the SDK to format in a structured fashion,

25:58 then we went ahead and then we inserted that into the actual vector database.

26:04 So once that part is done, then we can move on to the actual querying.

26:07 But let's take a look at the output generated by this crawler.

26:10 So here is a list of URLs.

26:12 And we've got 1,600 or so links that were extracted by this crawler.

26:20 And then that in turn pulled out the content.

26:23 And so we have a very large JSON file containing all

26:27 these different case studies that were pulled out from those pages.

26:31 So now we've done that.

26:33 It's in the vector database.

26:35 We need to be able to query it.

26:37 So a little later, perhaps we'll show you

26:39 what the agent looks like and how it searches.

26:41 But ultimately, this is the function that works the magic.

26:45 And this particular function,

26:46 what we're going to do is we're going to take a query string,

26:49 and we're going to pass it along to two searches.

26:53 So we're going to execute a semantic search.

26:58 And then we're executing a text search.

27:01 And we're going to combine those results.

27:03 And the reason for this is that— it's

27:05 actually something we were discussing in the chat

27:07 just a moment ago— is that sometimes you might need to do a hybrid search.

27:11 And this is a great opportunity to take advantage of those capabilities.

27:16 And we'll show you a little bit later, perhaps,

27:18 how we take advantage of these two different types of search results.

27:22 So yeah, once we get those results, we will combine them and rerank them.

27:27 And then we present them to the agent

27:30 for the agent to then present that to the user.

27:33 All right.

27:34 What's the next question?

27:35 JACOB BADISH: Awesome.

27:36 Yeah, thanks, Luis.

27:37 So this one comes in from Dan here.

27:39 He says, what's the benefit of using ADK versus

27:42 the original version that I wrote, that Jacob wrote?

27:45 Can you compare the ADK version against the original?

27:49 LUIS SALA: Yeah, certainly.

27:50 So let's do a little side-by-side for that.

27:52 OK, so let's go back to the screen share.

27:54 And here, on the left-hand side here,

27:57 we're going to see the original agent that you developed.

28:01 And then on the right-hand side,

28:02 we're going to see how that is expressed in an ADK.

28:06 And so let's look at the original version first.

28:10 The original version uses the regular— oops, that's the wrong function.

28:15 This is the function.

28:15 Here we go.

28:16 The original version uses just regular Python to create the orchestration flow.

28:22 And so if we look in here, we have a function called process account.

28:26 And you call a function called generate intel,

28:29 which actually goes and researches as a company.

28:34 You have a verification function that gets called,

28:36 all wrapped inside different if-then statements

28:39 and loops and so forth and so on.

28:41 And so you've orchestrated it using these nested if statements.

28:48 And so that's fine, but we can express

28:52 that far more succinctly if we use ADK constructs.

28:57 So ADK is specifically designed to make

29:00 it easy to develop these types of workflows,

29:03 these types of step-by-step processes.

29:05 And so what you see here in the highlighted section

29:08 is a representation of your agent as expressed by ADK.

29:15 And what you can see here is that we have this concept of a sequential agent.

29:19 That sequential agent is capable of executing steps in order.

29:23 So this is the same steps here, such as research the company,

29:26 which is analogous to your generate intel function, your case study researcher,

29:31 which in this case is how we injected the new functionality,

29:35 where we actually look for real case

29:37 studies as opposed to using the hard-coded ones.

29:40 And then we draft an email.

29:41 So you can see it's succinct.

29:43 It's also extensible because you can imagine now adding additional

29:48 functionality just by adding it somewhere in this sequential list.

29:53 And in fact, that's what you see up above.

29:56 Think of this as an advanced version

29:58 of your original agent that does a little bit more.

30:01 It still executes the company research, but then it also executes a plan.

30:07 And the plan, think of it as adding a little bit

30:12 of forethought and planning to how we're going to research the case studies,

30:19 how we're going to craft a message.

30:21 And that is then going to be leveraged by the case study agent.

30:25 And we added another agent called a selector agent,

30:28 which will look at all the results that were returned by the vector search tool.

30:34 And it will select one, maybe two, case studies that are most relevant.

30:38 Why is that?

30:39 Because your use case required sending a very punchy email.

30:45 You don't want to generate a message that's three paragraphs long.

30:48 You want to respect the executives' time.

30:52 And so sending a very short, punchy email is important there.

30:55 And so that's essentially it.

30:58 Now, when we look at a little bit further above— and we're just going to look

31:01 at one function here that I think

31:04 kind of captures the bulk of this functionality,

31:08 and that's your original researcher here.

31:11 So let's do a side-by-side.

31:13 And we're going to do that there.

31:15 And we're going to go here.

31:17 So this is the ADK version on the right-hand side of your generate intel agent.

31:26 Here, it looks very similar.

31:28 The prompts are very similar.

31:29 Look at the steps.

31:31 The objectives are to scour the internet.

31:33 That is repeated here on the right-hand side as well.

31:36 It's going to do the deep research.

31:38 It's going to identify the industry and so forth and so on.

31:42 So the prompt is very, very similar, but a couple changes.

31:46 One thing we don't do on the right-hand side is

31:49 that we don't specify a JSON schema in the prompt.

31:52 In your original agent, you were providing a JSON schema.

31:56 Here, we don't do that because in ADK,

32:00 we can actually supply that schema as a value.

32:06 And so you can see the actual code part is much more succinct,

32:13 much more readable and maintainable and extensible than the original version.

32:18 Here, you can see that your agent was doing a lot

32:22 of regular expression matching and so forth and so on.

32:25 So it would be harder to maintain over the long run.

32:29 So that's why it was important to not just add the superpower

32:33 of vector search and case study searching and crawling to your agent,

32:37 but also refactoring it to use something

32:40 that is much more purpose-built for creating agents.

32:44 So hopefully, that answers the question.

32:48 What's next?

32:49 JACOB BADISH: Great.

32:51 OK, the next question here— agentic loops can burn through

32:54 tokens incredibly quickly if they get confused or hit an error.

32:58 How do you prevent the auditor agent from getting stuck in an infinite

33:02 loop if it continually struggles to verify

33:04 a specific technical claim using search?

33:09 LUIS SALA: So first,

33:10 I really cannot think of a scenario where an infinite loop would occur.

33:15 And the reason is pretty straightforward.

33:18 ADK itself, when we do something like invoke an LLM or do, in this case,

33:27 the vector search or something like

33:28 that, there are constructs and parameters already

33:31 in existence that control how many times

33:34 the agent will retry a request, for example.

33:37 And so those limits are built in, and they exist.

33:42 There's not just a retry limit but also a timeout limit

33:46 so that if you issue a request against a remote server,

33:51 we're going to wait up to a certain amount of time for a response to occur.

33:56 This is standard development practice.

33:59 This is standard software engineering.

34:01 And the ADK already encapsulates those types of capabilities.

34:06 When it comes to predicting costs— excuse me.

34:08 I'm sorry— controlling costs and avoiding runaway costs,

34:11 that is a very relevant question in this world

34:14 of LLMs where you're paying by the token.

34:17 I encourage developers to look

34:21 at their prompts and focus on prompt optimization.

34:26 And while maybe I would not recommend obsessing too much over token efficiency,

34:35 one should at least be thoughtful about it and be

34:38 able to ensure that their use of tokens is reasonably efficient.

34:43 We don't have to go all the way overboard,

34:46 but let's think about what the minimum amount of information

34:52 the LLM might need in order to provide a contextually relevant response.

34:58 So yeah, so as I mentioned, we have an ADK that gives you all these capabilities

35:03 out of the box in terms of retries and so forth.

35:06 It gives you exponential backoff.

35:08 It also includes, when it executes like a fan-out,

35:12 each of these subagents has its own time limits and retry limits.

35:17 So it's going to execute in a finite amount of time.

35:20 And therefore, we're going to avoid infinite loops and avoid runaway costs.

35:25 JACOB BADISH: Yeah.

35:25 And actually, Luis, you brought up the next question really, really effectively.

35:30 LUIS SALA: Oh, yeah?

35:31 JACOB BADISH: Yeah, there's another one here that asks,

35:33 you use the fan-out pattern with ADK

35:35 to research multiple target companies in parallel.

35:39 When dealing with asynchronous, multi-agent orchestration,

35:43 how does the router handle race conditions?

35:46 What happens if one of the worker agents hangs while the others finish?

35:49 Great question.

35:50 LUIS SALA: Yeah, in some respects,

35:51 I answered it previously because if something hangs,

35:54 it eventually times out, and the ADK will terminate execution.

36:00 However, there's still more in here that we can tease out.

36:07 So race conditions, let's talk about that.

36:11 The way that this particular implementation worked is

36:15 that we will execute multiple searches in parallel.

36:19 And then we're going to save the results

36:22 into a shared data structure, a variable.

36:25 And in this particular case— we can apply

36:30 standard development practices here to avoid race conditions.

36:33 But at the end of the day, we have an in-memory object.

36:36 It's a Python dictionary,

36:38 or actually a Python list in this case, and we're just appending.

36:42 And by using an append function, we avoid a race condition.

36:47 We're not having multiple agents or, I should say,

36:51 subagents trying to write to index number 3 at the same time.

36:56 They're just appending their results to a list.

36:59 And therefore, you can avoid that.

37:00 But yes, otherwise, feel free to develop and use

37:04 standard practices like locks and semaphores and so forth

37:08 to address any potential concurrency-related issues that you might encounter

37:12 when you're having parallel tasks executing at the same time.

37:17 JACOB BADISH: Great.

37:18 Thanks.

37:20 OK, another question here.

37:23 Setting the temperature to 0 for the fact checker

37:25 agent is a great move to enforce deterministic output.

37:29 But what happens if the search tool

37:31 returns conflicting information about a company's tech stack?

37:35 How does the agent resolve ground truth disputes?

37:40 LUIS SALA: Yeah, so the agent in this case, first, it did employ a reranker.

37:46 And the reranker helps, I'm going to say, sort through the results.

37:51 But let's focus on the last bit.

37:55 How does the agent resolve a ground truth dispute?

37:59 If your data set is, I'm going to say,

38:05 imperfect or contains contradicting information— so for example,

38:11 your data set might have a statement such as "the sky is blue" and another

38:17 one saying "the sky is purple." We have a problem with the data set.

38:22 And so what I would suggest employing here

38:26 to avoid things like conflicts and whatnot is

38:30 to instruct the agent through the prompt

38:34 to either disregard conflicting information, that is to say,

38:39 acknowledge the fact that we have two potential facts that are

38:45 in conflict with each other and might be difficult to disambiguate them,

38:49 and so therefore we might want to just ignore them altogether,

38:52 or to call it out or do something else.

38:54 My point here is handle the exceptions somehow.

38:57 You want to identify that an exception occurred,

39:01 and you can start doing that with prompting.

39:04 You can implement other kinds of safeguards as well,

39:08 but also look at your underlying data.

39:10 If your underlying data is dirty— I say dirty,

39:13 but it's inconsistent and it contains these potential factual errors,

39:20 then we might want to clean up that data to begin with.

39:24 Beyond that, let me see what else.

39:28 We can also rely on the model itself to try to disambiguate on its own.

39:36 So all large language models are effectively— as Andrej Karpathy might say,

39:42 they're a very lossy compression of the entire internet.

39:47 But they contain facts.

39:50 They've been trained with the contents of the internet.

39:53 And there are facts that are captured inside the model weights.

39:57 That means that it's entirely feasible for the model to disambiguate

40:03 on its own because it knows that the sky is blue,

40:07 not purple, except perhaps under a very beautiful sunset or something like that.

40:13 So that's the idea there.

40:15 So hopefully, that helps answer that question.

40:17 What's next?

40:18 JACOB BADISH: Awesome.

40:19 Thanks, Luis.

40:21 So OK, another one here,

40:23 debugging nondeterministic agents in production is a massive headache.

40:27 Yep.

40:27 Now that Titanium is off the local

40:29 terminal and running in a decoupled ADK architecture,

40:33 what does the observability stack look like?

40:35 Yep, how do you trace exactly why an agent made a specific decision?

40:40 LUIS SALA: Yeah.

40:40 So observability is critical.

40:43 Being able to understand what any code base is doing, let alone an agent,

40:49 is absolutely paramount to ensure that the agent is operating in a predictable,

40:55 reliable, and performant fashion.

40:58 So where do we start?

41:00 Tools like ADK give you observability out of the box.

41:04 And it starts at the developer workstation.

41:07 So when you're developing an agent using ADK,

41:09 you have essentially a built-in chat app called ADK Web that you

41:15 can use to inspect the variables and inspect the flow of the agent.

41:20 And that in and of itself is very,

41:22 very powerful and gives you that insight as to what

41:26 the agent is doing and what data it's receiving and so forth.

41:30 You can also implement step through debugging.

41:33 So this is standard development practice.

41:37 Run your agent inside a debugger and step through it.

41:41 And look at the variables.

41:43 Look at what's happening.

41:45 You can use that to identify potential timeouts

41:47 or even potential race conditions and so forth.

41:51 So that is also a very, very good practice.

41:55 Finally, virtually every agent toolkit that I'm

42:00 aware of, whether it's ADK or others,

42:03 such as CrewAI and so forth, have observability built in.

42:08 And observability primarily through OpenTelemetry support is

42:11 the most common thing that we see.

42:13 So you can enable OpenTelemetry, take the traces,

42:18 and store them in a centralized location,

42:21 whether it's Google Cloud's operations suite or a third-party service,

42:27 and you can use that information,

42:29 especially now that your agent may be deployed to production,

42:32 where you can go ahead and observe its behavior.

42:36 You can also set alerts to ensure that your agent is adhering to any

42:43 service-level objectives that you might have set

42:45 and ensure that you're providing a high-quality service.

42:49 Hopefully, that helps.

42:50 What's next?

42:51 Are there other questions?

42:52 JACOB BADISH: Yeah, we have a couple more here.

42:54 LUIS SALA: Excellent.

42:54 JACOB BADISH: A few from Philip.

42:56 Let's get to these ones.

42:57 And thanks again for the questions, team.

42:58 Really appreciate it.

43:00 The first one from Philip, how did you guys trigger the crawler?

43:04 LUIS SALA: So the crawler was a one-time task.

43:06 So there's just a separate function that dealt with the crawler.

43:10 It lives outside of the agent.

43:12 It is the kind of thing that you might want to run,

43:15 say, once every 24 hours or once a week.

43:18 So it's a standalone tool.

43:21 It does not really require any sophistication in terms

43:26 of agentic behavior other than the LLM-powered content extractor.

43:32 So that's how it was triggered.

43:35 In this case, I was using development agent.

43:38 So the code agent triggered the execution.

43:44 But it's its own standalone thing.

43:46 And when we share the code, you'll see that crawler implemented there.

43:51 JACOB BADISH: Yep.

43:52 And second question from Philip,

43:54 do you always start by default with a classifier agent?

43:58 And with that, what would be the core few points for deciding

44:02 how to isolate agents with respect to latency and with respect to task?

44:09 LUIS SALA: I'm not sure how to parse that question entirely,

44:13 but at a high level, no, you do not always start with a classification.

44:19 Or in this case, the sophisticated version of the agent included a planner.

44:26 A planner is good when you're doing research or when you're completing

44:31 a task that has a non-deterministic kind

44:34 of workflow or process associated with it.

44:37 So generically, if I zoom out a little bit, generally speaking,

44:41 I think what we want to do is just

44:44 look at the type of process we're trying to implement.

44:48 If it's a very structured process where we know it's like step 1,

44:52 step 2, and then at step 2,

44:54 we make a choice to go to step 2A versus 2B and so forth,

44:59 if it's a very discrete, well-known workflow,

45:03 then you might not need planning or optimization or anything along those lines.

45:09 But if it's a more loose kind of a process

45:13 where there's a lot of judgment that might come into play,

45:17 there may be some degree of planning or something analogous might be valuable.

45:23 So just look at the use case.

45:25 I encourage you to experiment.

45:27 Don't be afraid to go down one route

45:33 and uncover that maybe it's not the most optimal path.

45:37 That's, first of all, how we learn.

45:39 We learn by experimentation, by doing.

45:43 And we learn most when we make mistakes.

45:48 So please don't be afraid to do that.

45:50 And take those learnings and apply them to the next

45:54 iteration of whatever it is that you're developing,

45:56 whether it's an agent or some other application.

46:00 JACOB BADISH: Yep.

46:00 Thanks, Luis.

46:01 And I think this is the last one here.

46:03 This is from Osvaldo.

46:05 Can these agents be combined into an n8n workflow?

46:10 LUIS SALA: Yeah.

46:10 So n8n is actually a really cool workflow tool.

46:16 It has a visual workflow builder.

46:19 And so it can execute these processes.

46:23 It supports LLM integration.

46:25 And indeed, you can expose an agent built in a different framework.

46:32 Whether it's ADK or LangChain or something else,

46:34 you can absolutely expose it as a node in the overall workflow.

46:39 So absolutely, you can do that.

46:42 It's not very challenging at all.

46:48 All right.

46:49 Well, I think we're going to wrap.

46:52 So we're here at the end.

46:54 I want to thank you, Jacob,

46:57 for being the first guest and guinea pig on the show.

47:01 So many thanks to you and the audience for joining us at the Google Cloud

47:06 "AI Agent Clinic." We will be posting

47:09 a link to a GitHub repo in the description, so please keep an eye out for that.

47:14 We'll also be publishing a blog post with additional details

47:18 summarizing everything that we've learned

47:22 while upgrading Jacob's Titanium agent.

47:25 JACOB BADISH: Yeah.

47:26 No, and thank you, Luis.

47:27 It was so much fun.

47:28 I learned a ton.

47:29 We made some massive improvements.

47:31 To everyone on the live stream and in the chat,

47:33 thank you for participating and engaging and asking all these questions.

47:37 We also want to hear from you.

47:39 So there are so many different ways to build these agentic workflows.

47:43 Let us know in the comments how you would have fixed Titanium,

47:46 what architecture changes you would have made,

47:48 perhaps how you would have done things differently.

47:50 LUIS SALA: Yeah, indeed.

47:51 There's no single right way to do this.

47:53 Now, we have a special invitation for you.

47:56 Are you building an agent, and are you hitting a wall?

47:59 Do you want to bring it into the clinic and upgrade it with a Google expert?

48:03 Well, I want to see what you're working on.

48:05 So get in touch with us.

48:07 Tell us about your project by emailing agent-clinic@google.com.

48:12 In the meantime, thank you so much for watching.

48:15 Please keep on building, and we'll see you next time on the "AI Agent Clinic."