Building GPU-accelerated multi-agent apps with Google ADK and Gemma 4

1:08 We are looking at a 5,000 share order.

1:50 Hello, everyone and welcome to another episode of Google Cloud live.

1:54 I'm Chelsea chop, a senior product manager at Google.

1:58 And if you're interested on how AI

2:00 agents can tackle complex real world environmental challenges,

2:05 then you're in the right place today.

2:07 We're going to walk you through a brand new multi-agent

2:11 sustainability intelligence app that's designed to pull in satellite imagery,

2:16 live sensor telemetry,

2:17 and dense policy documents and instantly generate a decision

2:22 ready risk report for the developers tuning in.

2:25 We're also going to get totally hands on.

2:27 You'll see a Google ADK orchestrator managing

2:31 specialist agents combined with NVIDIA accelerated GPU inference,

2:35 Milvus policy retrieval, and also a Cloud Run deployment.

2:39 So it's all serverless.

2:41 And I couldn't do this alone.

2:42 Joining me today is Mitesh Patel from NVIDIA.

2:45 Mitesh welcome.

2:46 Do you want to introduce yourself quick.

2:48 Sure hi.

2:49 Thanks, Chelsea.

2:51 Welcome, everyone and thanks for having me.

2:53 My name is Mitesh and I lead the developer advocate team here at NVIDIA.

2:56 And the goal of our team is

2:58 to evangelize different applications that can help developers envision

3:02 how they can use all these state

3:04 of the art tools in the work that they're doing.

3:07 So I'm excited to be here and talk

3:09 about the sustainability app that we have developed.

3:12 Yes, thanks for joining.

3:13 It wouldn't be the same without you showing us how to do this.

3:16 And before we dive in, I want to know where everyone's tuning in from today.

3:20 So drop your city or your country in the chat.

3:23 And also, what are you currently using agents

3:25 for or where are you using agents in production.

3:28 We'd love to be able to hear from you.

3:30 I'm out of Park City, Utah today,

3:33 so looking forward to seeing where everybody's listening from.

3:37 And quick housekeeping note before we get started,

3:40 if you have any questions during this walkthrough or the presentation,

3:43 put them right here in the chat and we'll answer them.

3:45 We have folks behind the scenes here from Google and NVIDIA

3:48 standing by ready to answer any questions that you have.

3:52 So with that, I'll get started here.

3:59 So what are agents enabling.

4:01 They're enabling cross platform automation.

4:05 Asynchronous productivity I know that's where

4:08 I'm using agents from that standpoint.

4:11 They're enabling real world transactions.

4:15 And the focus of what agents are able to do

4:17 is that they're optimizing for outcomes instead of just output.

4:22 So an AI application is a Cloud application that contains one or more

4:27 or many in the use case that you'll see later AI agents an AI agent.

4:33 So what is an AI agent.

4:34 It's a service that autonomously reasons to be

4:37 able to solve a task using tools and data.

4:41 And all services, they must meet your compliance requirements.

4:46 Use your CI/CD pipelines and protect against abusive behavior

4:52 traffic and do so securely across all of your resources,

4:57 and they must be cost and performance efficient.

5:00 They need to meet your latency SLOs

5:02 and elastically scale for capacity for traffic.

5:05 And it all has to do that with security.

5:08 So you need those security guardrails.

5:10 So that's a lot.

5:11 Let's look at a concrete example.

5:14 How can Google Cloud help.

5:16 So in this example.

5:17 Assume that you need to build an agent to be able to respond to users

5:21 requests to generate a new interactive dashboard

5:24 and be able to surface new insights.

5:26 So what does that mean the agent needs to do when the prompt comes

5:30 in, the AI agent needs to be able to parse the text, build a plan,

5:34 and then make a series of calls to different endpoints,

5:36 such as an LLM to generate code,

5:39 MCP servers to be able to pull that data from the databases or the user.

5:44 And then finally, they need to coordinate and execute all

5:47 of these in a useful way back to the user.

5:51 And when we look at how different components involved in the workflow,

5:55 we can see how complex the executions can

5:58 be across the various durations of the executions.

6:02 So some may be running longer executions of reasoning.

6:06 Other models may be calling multiple MCP servers,

6:09 and some may have a shorter call to generate

6:12 a simple SQL queries to update a database.

6:15 Now, with all that complexity,

6:17 we need to make sure that we choose the best Cloud platform that will help one.

6:21 Help us scale for high elasticity.

6:24 Two rightsize our compute.

6:25 So that way you get the best cost per performance,

6:27 and you're not paying for what you don't need.

6:30 And you need to do that while meeting a latency slow.

6:33 Because if you have to sit and wait,

6:35 what is the point of having an agent being able to do that on the backend?

6:39 And most importantly, I would say too, is it has to happen securely.

6:43 You can't have agents going and being

6:47 broken into or breaking your security guidelines.

6:50 OK, so that was a mouthful.

6:52 Let's get into it and summarize these in six key points.

6:56 Your agentic workloads need to be treated as untrusted.

6:59 They need to be able to scale and meet bursty traffic.

7:03 And a lot of them are latency sensitive.

7:05 Some of them have long running tasks that are happening in the background,

7:08 and a lot of them have idle cycles.

7:11 Finally, some of the more complex ones are usually those are the hungry ones,

7:15 especially the memory hungry ones.

7:18 So AI agents introduce a lot of unique challenges

7:22 that can really stress infrastructure and expose the vulnerabilities.

7:27 So there are three categories of challenges that we

7:29 see that we expect you to see when deploying agents.

7:32 First is latency and throughput.

7:35 We need to ensure that applications can scale

7:37 for elasticity in a world where accelerator capacity is constrained,

7:41 while being able to optimize for that latency.

7:44 Second is compute efficiency.

7:46 We need to be smarter about the way that we use

7:49 the compute to increase efficiency and density

7:51 and reduce the idle compute resources.

7:54 And third is security and governance.

7:56 We also need to supply apply security controls so that we're

8:00 empowering agents to be able to handle those long running controls,

8:04 complex tasks all at the same time to ensure

8:07 that you have control and you have access to it,

8:10 you can debug them and you have auditability

8:13 so that you know and have that governance.

8:19 So at Google Cloud, we build the AI Hypercomputer stack

8:22 to be able to optimize for all of the AI,

8:25 ML workloads and all the ML lifecycle stages.

8:28 Now it addresses three main challenges for AI agents to as three major,

8:32 major layers at the bottom.

8:34 My favorite the purpose built hardware.

8:37 You have your open software platform and flexible consumption models.

8:42 The purpose built hardware doesn't mean anything if you're not able to use

8:46 it with your open software and be able to use exactly what you need,

8:49 when and where you need it.

8:51 But Google Cloud, we offer a wide range of NVIDIA GPU products,

8:54 from hopper to Blackwell to Vera Rubin.

8:57 And today, I'm going to give you a little high level overview

8:59 of the G4 product that we launched last year or early this year,

9:04 which is based on NVIDIA's RTX Pro 6,000 Blackwell GPU,

9:08 because I believe it's a very interesting workload for agentic workloads

9:11 as well as it's what letitia's demo is built on today.

9:16 So mattusch is actually going to use in his Demo

9:19 Cloud Run to be able to do it now.

9:20 Cloud Run is a serverless platform that takes all

9:24 these components that I just talked about in AI Hypercomputer

9:27 and brings them together so that you don't have

9:29 to use and pick and choose what exactly works for you.

9:32 Across the stack, you're able to just kick off a job with Cloud Run,

9:36 which is why it's so perfect for workloads.

9:39 So you can use Cloud Run GPU services to perform real time multimodal inference.

9:46 Now in the demo that is going to show he's going to use Gemma.

9:49 Gemma for on it.

9:51 And that has multi-modality capabilities.

9:53 That's how he's able to get sensor data,

9:57 policy recommendations as well as real time telemetry

10:00 and bring them all together into Gemma for now.

10:04 For example, say you want to run a chatbot chatbot application for your end

10:09 users if they want to generate images or use the text response prompts.

10:14 You can use Cloud Run to be able to deploy AI agents to be able to do that.

10:18 Or say, for example, you want to run an IT support agent

10:22 or customer service to autonomously answer user questions.

10:25 Cloud Run services can be used for those non AI compute intensive resources,

10:30 so you can use it to kick off media serving, generating images and AI,

10:36 or being able to use some of those agentic

10:39 workloads to kick off background of tasks.

10:42 So let's look at a couple design patterns to see how you can

10:46 use and pull in Cloud Run to be able to support your agents.

10:51 So here's a design pattern for on demand inference AI inference with GPUs.

10:56 So say a request comes into Cloud CDN and you have your request caching.

11:00 It would kick off a Cloud Run job via the API.

11:03 Say it runs in such as Gemma for catches

11:07 and takes the LLM weights from Cloud Storage that you've customized.

11:11 Takes it over a VPC with VPC egress,

11:15 and kicks off a Cloud Run GPU to be able to perform that inference,

11:19 such as text to text or text to image.

11:22 And then it sends back the response via the front end.

11:26 Now let's look at another design pattern here.

11:29 Say Cloud Run GPUs are doing LLM fine tuning and batch inferencing.

11:35 Fine tuning Gemma models is a very popular way

11:38 to specialize them for a special use case or domain.

11:41 Instead of being able to tune them from a large model from scratch,

11:44 you can take the pre-trained demo models to train

11:47 them further on a smaller custom data set.

11:50 This is a lot more efficient and cost effective as well.

11:54 Really high level.

11:55 There's some use cases that are really good for fine tuning.

11:58 If you have domain specific knowledge that you want to bring

12:01 in, the demo models have a broad understanding of the world,

12:05 but they might lack your industry specific

12:07 or your company specific specialized knowledge in a particular area.

12:13 So a financial services company could fine tune Gemma

12:15 on a data set of SEC filings or market

12:18 reports or internal company memos that helps you

12:21 get the best answers using for your agentic workloads.

12:26 There's also task specific behavior or for customer service that you want

12:30 them or say you want your agent to be able to do.

12:33 Say you have a certain set of customer

12:35 interactions or a way that you handle this.

12:37 That's what it's really good for fine tuning.

12:39 Or if you want it to adopt a specific persona or style,

12:43 you can teach the model to adapt to that tone,

12:45 personality, or your own writing style.

12:47 So, for example, say, an online game developer could use a Gemini model to act

12:52 as a non-player character NPC with a unique personality and speaking style.

12:58 So there's some fine tuning techniques.

13:01 It's typically done using parameter efficient tuning.

13:03 So PEFT.

13:05 Unique techniques like Laura.

13:06 Low rank adoption.

13:08 This method freezes the original model weights and only

13:12 trains a small number of new lightweight adapter layers.

13:16 They drastically reduce the memory and compute requirements,

13:19 and that makes the fine tuning possible on modest hardware.

13:23 And pros and cons.

13:24 So running Cloud Run with GPUs is a great platform for being able

13:28 to fine tune the smaller models

13:30 or performing asynchronous one off fine tuning tasks.

13:34 And you're able to kick off that workload and it can run in the background,

13:38 and you don't have to worry about managing everything.

13:41 So let's dive in a little bit more about G4 or NVIDIA

13:44 RTX 6,000 GPUs compared to the previous generation that we had,

13:51 the L2, L2 VRMS, which is the L4 GPUs,

13:58 it's 7x more performant, 4x more GPU memory,

14:02 and 3x most host memory compared to our G2 VM offerings,

14:06 which was the first GPU that we had

14:08 with the AMD Turin CPU supporting NVIDIA's GPU.

14:14 So GPU is optimized for peer to peer multi-gpu workloads,

14:19 so the benefits that you get

14:21 from this is that it accelerates multi-gpu workloads.

14:25 You get 2x the nickel collective performance

14:28 when you're using the full VM shape, or up to or on a four GPU VM shape.

14:34 So there's also fractional GPUs that you could use all the way up to, say,

14:38 four or eight GPUs in the full VM.

14:41 And if you're using the whole VM then you can get up to 2.2.

14:44 Nicole collective performance gains.

14:47 All you need to do to be able to get

14:49 that performance benefit is just turn it on with an environment flag.

14:52 Now where do customers use this and where have we seen it successful.

14:57 Flipkart, one of the largest e-commerce platforms in India,

15:03 uses it for an AI led catalog enrichment.

15:07 So what they do is they take images and they convert it into video.

15:10 So it's really cool that you can have a first flame,

15:13 last frame and it'll generate a video with that output.

15:16 Now in the background,

15:17 it's agents that are kicking it all off and it's running on G4 GPUs.

15:21 So what were they able to achieve with that.

15:26 They were able to use the P2P

15:28 communication capability with the two Blackwell GPUs,

15:30 that they can talk directly to each other

15:32 without involving that host CPU or system RAM.

15:35 Now, this gives you higher throughput running video generation models

15:39 compared to just using across and running it across PCIe.

15:43 Now they found that they were able to get

15:46 a 50% latency reduction and a 50% cost reduction, which is phenomenal.

15:53 So they were able to do that with GeForce GPUs, which is NVIDIA RTX Pro 6,000.

15:59 And then also they customized their media pipeline

16:01 and they ensured that the P2P topology design was on.

16:06 Now another area where we're seeing a genetics really

16:09 being used a base 10 share the set next, which is really great.

16:13 Other three areas that they really see that agents are being used.

16:18 And I'm curious again, put it in the chat where you're using agents,

16:20 but they're seeing it in real time voice agents,

16:23 encoding agents and research agents.

16:25 So say you're looking at a code base and you're asking it to build a feature.

16:29 Now we can go and research and understand that code

16:31 base and give you some developed chain of inference.

16:35 So chain of thought as to how you're able to do that.

16:38 And another thing that they said that was

16:41 their recommendation was to be able to fine tune.

16:43 That's how you get the best agent productivity.

16:46 But now let's see it all come together.

16:48 I'm going to hand it over to Mitesh

16:51 to walk you through our sustainability agent app.

16:56 Yeah, thanks.

16:58 Before I dive into this application, it's amazing how based on Flipkart,

17:05 are utilizing agents for various application

17:09 utilization and how our Blackwell GPU

17:11 or the Pro 6,000 are helping enable them to accelerate their whole pipeline.

17:17 And one of the most important thing that you talked

17:19 about in your slide deck is latency is very important,

17:22 and throughput is very important.

17:23 And cost effectiveness is also important because when

17:26 you're scaling the systems out at production level,

17:28 cost is the primary factor that drives all this adoption.

17:31 So it's amazing that we have it, that all your customers,

17:36 all the customers Flipkart based in are looking

17:38 into those KPIs or SLAs when they are designing systems.

17:44 So that's great to see.

17:45 Yeah, I'll just share a quick anecdote.

17:48 I have a friend who started a company,

17:50 and their MVP product didn't take into consideration

17:54 the agentic kind of demand from that standpoint.

17:56 And when they started using it, the demand went off the charts,

17:59 and it wasn't designed in a way that they could sustain the cost for that low,

18:04 that low latency as well as the low cost efficiency.

18:07 So they had to pause the MVP,

18:09 take it back, and then re-architect before they went live.

18:12 But I'll let you walk through the demo.

18:14 That's what everybody's doing anyways.

18:16 Yeah, exactly.

18:16 So let's dive into it.

18:18 So what we wanted to showcase here.

18:22 Let's go to the slide deck first I think I would like to walk them through the.

18:27 Yes OK.

18:28 Here we are.

18:28 So sustainability agent is something that we orchestrated

18:33 to showcase the effectiveness of various agent toolkit,

18:36 how we orchestrate this entire use case

18:39 because it has got multi modality involved.

18:42 You're looking at image data.

18:43 You're looking at telemetry data.

18:44 You're looking at text data.

18:45 So different modality of data is coming in as input.

18:49 So you need to design specific agent or agents that can absorb these modalities.

18:54 And on the modeling side, you need to have a multi-modal that is

18:57 capable to take all this information as an input,

19:00 this modality of information as input,

19:01 and then process it and figure out what is the best course, answer to that.

19:06 So with this, that's the reason why we used the climate data as a use case.

19:11 But this is just one example.

19:13 There's nothing stopping you from utilizing the same

19:17 framework or the same orchestration that I'm showcasing here.

19:21 For any application that you're building that involves multi modality

19:24 and you want to capitalize the capability that Gemma is providing.

19:29 So how did you orchestrate this agent.

19:32 So we used Google SDK toolkit.

19:34 As Chelsea mentioned we use Google Cloud Run

19:37 and then we use Gemma for model under the hood.

19:39 And then for inferencing purposes, we used VLM engine.

19:43 There is nothing stopping you from using

19:45 any other inference engines or back end.

19:47 It can be SGLang, it can be Dynamo which comes

19:50 from NVIDIA based and uses it in production for their customers.

19:54 And then for vector database we used Milvus,

19:57 which is a GPU accelerated vector database.

20:00 But again, there is nothing stopping you from using

20:02 any other vector databases that are out there, which are GPU accelerated.

20:06 And how did you orchestrate this whole thing together.

20:10 So at the top, you see the main.

20:14 The SDK toolkit that is the main orchestrator.

20:16 And then underneath there are three agents.

20:19 Each agent is responsible to take in taking different modality as an input,

20:26 and then process it and then provide an output.

20:29 And the main orchestrator will combine all this information coming

20:32 from two or three different subagents and generate a report for you.

20:36 So if you as I said before, we are taking or absorbing multimodal information.

20:41 So we are taking in satellite imagery as an input.

20:43 We're taking it we are taking in telemetry data

20:45 that is coming in from different weather station information.

20:48 So we take that as an input as well.

20:50 And then we are looking at policy documents.

20:53 And this policy documents can be documents either created by state.

20:59 Or if you tone it down further,

21:02 it can be policy documents created by different countries or different cities.

21:07 So for this use case, we looked at a few counties.

21:10 We collected the PDFs or the policy documents that we had,

21:14 and we embedded them into vector embeddings offline

21:17 or a one time process beforehand and then at runtime.

21:21 So when a user or when a decision

21:24 maker is asking all those questions to the agents,

21:26 we will go and retrieve the information

21:28 from those embedded documents and take it out.

21:33 Let's dive into the.

21:35 I would like to show you the real demo of this work.

21:38 So here is how the UI is orchestrated.

21:41 So at a higher level we are using the Gemma for $31 billion parameter model.

21:46 We are using a quantized version of a version of it, which is an gn-fp-4 format.

21:52 Again, this format came out of NVIDIA and it is a very good

21:56 format to use when because the model size is quantized quite a bit,

22:03 and at the same time the effectiveness stays the same,

22:06 so the quality of output stays the same.

22:08 So we used the gn-fp-4 version of Gemma for the embedding sides.

22:13 To embed all the policy documents, we use the embedding Gemma 300 million

22:18 parameter model so that embedded all the PDFs,

22:21 all the policy document that we had into vector embeddings,

22:25 and then for inferencing purposes for this specific use case,

22:27 we used BLM as an inference engine.

22:29 But there is nothing stopping you from using other inference engine.

22:32 As I mentioned before, and from the input standpoint,

22:35 we are using satellite imagery,

22:36 we are using telemetry data and we're using policy document.

22:40 So I will reset the platform and hope the demo

22:44 gods are with us to make sure it runs.

22:49 All right.

22:49 So we are using the Phoenix urban heat

22:52 island risk data set which is satellite imagery.

22:55 So if you look at the baseline and the current image.

22:57 So we are trying to see what is the heat risk that we have.

23:00 We are also using incense information and policy documents.

23:04 So I'll run this analysis.

23:07 So it dispatch all the information.

23:10 So the main orchestrator is dividing the task to the agents out there.

23:16 So right now we just finished analyzing the satellite imagery.

23:19 The other subject is processing the telemetry.

23:22 The policy agent is going to go and review

23:24 all the policy documents and retrieve the right information,

23:27 and then it will all the synthesis will

23:29 happen where all this information is combined together

23:31 and will get a really good report which

23:36 states what is the state of the heat risk.

23:41 So it's giving the policy document.

23:46 Let's hope OK.

23:47 That's done.

23:48 So it's generating a report.

23:49 Now it's combining all the information that we have.

23:53 And it's going to give us an output of what it thinks.

23:57 The outcome of combining all this information is.

24:05 There you are.

24:06 So it is giving you an executive summary and the recommended next

24:09 step of what are the immediate mitigation that can be taken care

24:13 of or cooling strategies that counties or states can adhere to in order

24:19 for them to bring down all the risk factors that this country is facing.

24:24 So this is just one example, but from an orchestration standpoint,

24:26 we are looking at a lot of things that we are putting together.

24:30 And the advantage of using all these toolkits.

24:33 Google Play toolkits, as Chelsea mentioned initially in the call,

24:36 it's a handful of things that you need to pipe in together.

24:40 And Google SDK toolkit is a very good orchestration agent

24:46 that helps you streamline all these plugging in different agents,

24:51 talking to different retrieval systems.

24:53 We have not integrated any not we have

24:55 not integrated any mcps-prs for this use case.

24:58 But there is nothing stopping you from integrating different mcps-prs as well.

25:02 A simple example would be we pre embedded the policy documents.

25:06 Now, if there is an MCP server available

25:08 from the state or the County or the city,

25:10 we can always tap into those MCP servers

25:13 and that can be orchestrated as well using the SDK toolkit,

25:16 or it can be orchestrated offline as well and then plug into the toolkit.

25:20 So all this flexibility comes into play at the abstraction layer,

25:23 because if you try to code it yourself, it's not impossible.

25:26 But your time to market will just be way longer.

25:31 And that is where these orchestration toolkit becomes very easy to use.

25:35 The Inference framework becomes very easy to use

25:37 because they are give or take three or four click Deploy systems in terms

25:41 of getting the inference engine up and running,

25:44 how you can orchestrate that, how you can orchestrate agents with SDK toolkit.

25:49 One last thing I'll add is the way we

25:52 designed this entire agent platform is deployed on Google run,

25:57 but we took care of different latency or modalities.

26:02 And what I mean by that is we made sure that the web

26:06 app is orchestrated differently than

26:08 the inference engine and the agent orchestrator.

26:11 And one of the primary reasons we wanted to do that, or we want to showcase

26:14 that it's possible to do this on Google Cloud Run is because let's say me,

26:20 and few other folks from our team are using the same agent,

26:24 then the concurrency comes into play,

26:26 and then we want to make sure that the system can be scaled.

26:29 So all those facilities are available that one can capitalize within Cloud Run,

26:33 as well as how you're orchestrating your entire framework using Google.

26:38 So that's the demo I wanted to showcase,

26:42 but few key messages that I would like developers

26:44 to take home from this demo is not the agent, but the orchestration capability.

26:50 Gemma for model multi-modality please take a look at it.

26:53 It is really, really good.

26:55 It's open source.

26:56 As Chelsea mentioned in one of the workflows,

26:58 you can always download it from Hugging Face.

27:00 You can post train it for your use case.

27:02 It has the ability to absorb multi-modality.

27:04 That's one.

27:05 The second thing gn-fp-4 format.

27:07 You can always quantize that model further and bring down the model

27:11 size so it can be deployed on hardware with smaller GPU footprint.

27:15 And the third thing is the SDK toolkit.

27:18 It has the ability to orchestrate multiple agent system,

27:21 which becomes very important for practical use

27:24 cases if you're building a demo for yourself.

27:26 Multi agent might not be the way to go,

27:28 but if you're building production pipelines,

27:30 multi agent will become important and there are plenty of use cases.

27:34 Chelsea already mentioned a few of them.

27:35 There are plenty of use cases out

27:37 there where multi agent orchestration is very important.

27:42 Great Thank you.

27:44 So I figured it would be good if we have a little discussion.

27:46 I was just going to kick off with a simple question,

27:49 since we both used this acronym a few times.

27:53 If you can explain it.

27:54 What is MCP and how is it and why is it so important to agents.

27:59 Yeah, so it is a protocol.

28:03 And the reason why the protocol is designed is it's kind of a rule based.

28:09 It's kind of a rule that has been

28:12 established that you can talk to different servers.

28:15 So if you are trying to let's say I'm trying

28:17 to speak a different language and you're speaking in English,

28:20 we will not know what each of us are talking about.

28:23 NCP sets that layer where you can talk to different servers as far

28:29 as they have orchestrated that their entire

28:32 information and provided access through an MCP.

28:35 So multiple context protocol, I think, or am I'm forgetting it,

28:40 but context protocol are the last two full forms of the acronyms is my take.

28:44 But the goal of MCP.

28:46 Yeah, there you go.

28:47 But the goal is to make sure that if

28:53 different providers out there want to talk to agent,

28:58 there has to be an API layer or a call that is streamlined across the board.

29:05 So all the information providers can orchestrate and make sure they are

29:12 following the MCP protocol and all the agents on the other side,

29:16 they can talk to all these MCP in the same language.

29:19 And how this will help the entire ecosystem is eventually.

29:23 You don't have to create different blocks

29:26 of code to talk to different service providers.

29:29 If everyone is sitting or using MCP as the protocol,

29:34 we just use MCP to talk to all of these services,

29:38 and we don't have to orchestrate different code base or different

29:41 rules or different APIs calls to talk to all these providers.

29:45 That is the most important thing of MCP

29:47 that we are streamlining across the ecosystem.

29:49 How you can talk to different service providers.

29:52 Well, fantastic.

29:53 So it's like the universal usb-c of I to connect everything kind of.

29:58 Yes that's the right analogy oversimplifying it there.

30:01 Yeah so you work across so many different industries and streets and areas.

30:07 Where are you seeing agentic AI or agentic really take off for workloads.

30:14 So if you think about it, agent started from the software space.

30:21 So of course, the absorption or adoption is going to be first there.

30:24 So for software engineers who are coding or creating different applications,

30:30 that is where AI has become very, very prevalent.

30:35 That application space coding agents is becoming reasonably mature,

30:39 that you are even reaching a stage where if

30:43 you are trying to design an application from scratch,

30:46 there are so many coding agents out there

30:50 who can help you plan your entire system,

30:52 create an entire workflow so that it adheres to what you

30:56 are looking for and then go and create a baseline for you.

30:59 So it becomes a very good assistant for the engineer

31:02 who is coming up with coming up with that entire application,

31:06 and then they can be the decider that I want

31:08 to orchestrate my architecture this way and not this way.

31:11 So that is one area where I am seeing

31:14 a lot of absorption in terms of coding agent.

31:17 Now, if you go scale it out,

31:19 take two degrees of freedom outside the engineering space.

31:22 Where are the other places where these agents are becoming useful.

31:25 Healthcare domain.

31:26 One of Mason's client is talking about it.

31:29 My medical doctor, when I go to they will always take my consent.

31:33 That is it OK if I record this call.

31:35 So it helps me transcribe that.

31:36 Please go for it.

31:37 I am the best advocate for that.

31:40 So that is another place where we are seeing this application.

31:45 And what I'm seeing lately is a lot of folks who are trying to create different.

31:53 Things a simple example would be you are creating a video content,

31:56 a video explainer, or you're writing a blog and you want to Polish it further.

32:01 These kinds of agents can become very handy.

32:04 A simple example would be let's say I'm writing

32:06 a blog of how to use Google Ad toolkit,

32:09 and then how to orchestrate that using VLM.

32:11 Now I'll get all the technical content.

32:13 That's easy.

32:14 But if you're writing a blog out of it and you

32:16 load few bullet points and you want to expand it further,

32:18 or you want to Polish your language,

32:20 these agents can be very useful in that space.

32:23 And the last place where I'll touch it, which is my personal favorite physical.

32:27 I am a robot.

32:28 I was going to ask you about that next.

32:31 I was going to say you haven't even touched on your background.

32:34 Yeah that's true.

32:35 Yeah my PhD in robotics.

32:39 So I was going to ask, where are you seeing it in physical AI.

32:43 Yeah so multiple places.

32:46 Sorry I'm going way too fast.

32:49 But in physical AI space,

32:52 there are multiple places where you will find applications of agents.

32:56 The very first place is with physical AI.

32:59 Data generation is very important.

33:01 How do you figure out what kind of data you already have.

33:04 What kind of data do you need.

33:05 And agents can become a very good assistant

33:08 in figuring out what kind of data you have.

33:11 And it can be a very good system of generating data as well,

33:15 because you can hook in your agent

33:16 with all those synthetic data generators out there,

33:19 both in form of World foundation model as well as in form of simulation.

33:24 You can orchestrate agents to go and create

33:26 those use cases for you in real world.

33:30 So this is more on the engineering side when you are understanding your data.

33:34 If you try to take the systems into production,

33:36 where you're trying to understand the lay of the land

33:39 or your environment in which your robots are operating,

33:41 let's take manufacturing example manufacturing facility has an example.

33:46 So in manufacturing facilities you might see some anomalies.

33:49 So if you have designed your humanoid

33:52 or your ground robot to go and monitor those environment,

33:56 you can simply throw an agent, ask simple query questions.

34:00 Can you find the places on the manufacturing facilities which should

34:04 not have chairs so the robot can maneuver in the environment.

34:07 It will figure out how to scan the entire environment.

34:10 It will do its own path planning.

34:11 But the high level task you give that the robot has

34:14 to go and scan the environment and figure out if there are anomalies,

34:17 which is chairs in this case, so the robot can maneuver,

34:20 map the whole environment, how it is going to go about in the environment.

34:24 And whenever it sees an anomaly,

34:26 it can send a red flag to the control station or the main station

34:31 where humans are there and say that I saw a chair at this location,

34:36 then the user or the human can go in and clean up the environment.

34:39 But that is another place where these kinds of applications will become handy,

34:43 and agents will be the main orchestrator or the communicator

34:47 between the robot as well as the human.

34:50 Awesome And what do you think the biggest

34:54 challenges are to deploying and using agents in production.

34:58 So there are always challenges.

35:01 These are new technologies.

35:03 So there are always challenges.

35:05 And I think some of them you walk through in the initial part of your talk.

35:10 So new challenges that will become very important,

35:13 very important to address to understand the use case.

35:18 I think a lot of times we are wasting tokens in terms of generation.

35:23 So understand your use case very well.

35:25 This will help you understand what models to use.

35:28 It will help you understand.

35:30 Once you have figured out what models are the best fit

35:33 for the use case that you're working on, it will help you.

35:36 What GPU footprint.

35:37 You should use it while you're

35:39 orchestrating the entire workflow or your pipeline,

35:43 from GPU selection to model selection to how

35:46 your work is going to orchestrate your agent,

35:49 all these decision making becomes very important

35:52 when you are evaluating your use case.

35:55 So understanding your use case very well.

35:58 What models this will eventually flow into.

36:01 What GPU footprint or what software stack you need.

36:05 And then you go into orchestration mode.

36:07 So the challenge in my opinion comes into play

36:11 is having a good understanding of the use case.

36:13 And once you are orchestrating all these things into production.

36:16 The other things, as you pointed out very well

36:18 in the initial part of your presentation, is guardrails.

36:22 We don't want agents to produce not safe for work content.

36:25 So you need to put guardrails in place.

36:27 So when you're scaling this out in production,

36:28 all those things becomes very handy.

36:30 And then elasticity as Google Cloud offers that you talked about it as well.

36:33 How you scale up, scale down the system based on the demand that is coming in.

36:37 So Google Cloud offers that as well.

36:39 So all those things becomes very important in terms

36:41 of decision making when you're orchestrating your agent.

36:50 I think you're on mute.

36:51 Chelsea, we can't hear you.

36:52 You are so right.

36:53 Another question for you.

36:57 How do you prevent in that optimization.

36:59 How do you prevent agents from becoming stuck

37:02 in an infinite thought loop and burning tokens through to make

37:06 sure that you are efficient in what your token

37:09 usage is doing and your compute from that standpoint.

37:12 Yeah, this is one of my favorite use cases.

37:17 When you're orchestrating agents, you have one agent that is creating things.

37:21 In my opinion, you should have an evaluator agent as well,

37:24 which is evaluating what your first agent is doing.

37:26 That is the way you can break the loop.

37:29 If the agent is constantly in some hallucination

37:31 mode and is going in a circular fashion,

37:33 and it is not giving the right answer that you care about,

37:36 so you are burning tokens when the generation is happening.

37:38 But if there is, if humans is evaluated happening, this is not the right answer.

37:42 We should change the prompt.

37:43 And we should retest it again.

37:45 But the evaluated agent will eventually

37:47 learn where the agent is making mistakes.

37:51 So if you have an evaluator agent orchestrated along with the generator agent.

37:55 It's like the battles of the classic.

37:59 So one is generating, the other is saying, no, you're wrong.

38:01 You tried this way and then it generates again.

38:04 You're right this way.

38:05 So it goes through those things.

38:06 And that is the way you can break a hallucination.

38:09 That is one aspect.

38:09 The other aspect is of course, you can put rules or guardrails around

38:13 if the generation is hallucinating and not good.

38:16 You always have an evaluator along the way.

38:18 So the generator is not doing the right thing.

38:21 How long do you want the generator to keep generating.

38:24 And there is a new thing that is coming into play a lot,

38:27 which I'm observing is routing mechanism.

38:30 There is some work done by one of a member

38:34 on my team of how you can do smart routing.

38:37 So let's say at the lowest level where I talked about applications,

38:41 you have different models.

38:44 Now we only talked about four o'clock in the sustainability agent.

38:47 But let's say you have a club of models Gemma.

38:50 For is there Gemma 3 is there an emotion that comes out of NVIDIA.

38:54 They are there.

38:55 We have multiple agent selection capabilities,

38:57 so you can orchestrate your agent in such a way

38:59 that you are routing your query to the model,

39:04 which you think is the best model to answer that question.

39:09 Oh, fantastic.

39:10 Yeah if you look at the agent categories right now,

39:12 all the agents are talking to Gemma.

39:14 But under that can orchestrate model selection too.

39:17 So you can see your agent orchestration can get very complex, very fast.

39:21 Oh, 100% It just spans out.

39:24 So it spans out.

39:25 It exponentially scales out very fast.

39:27 And you can be fancy at the same time.

39:29 You can be efficient as well,

39:31 of how you make your model selection and take the right

39:35 steps to mitigate the hallucination issue that you talked about.

39:40 So say you're just getting started in this area.

39:43 And I know a lot of people are probably just hesitant to be able to start having

39:49 a model or an agent in that space to be

39:51 doing the evaluation and transition from humans to agents,

39:54 doing those guardrails and checking in.

39:57 When do you think is a good line

40:00 or a baseline that you're ready to make that transition,

40:04 make that transition from humans to human in the loop

40:07 to agents that kind of monitoring and keeping those safeguards in.

40:12 So this question is very well studied in the research world as well.

40:16 Roboticist so I know you studied well there it is.

40:19 It is studied well in other domains as well.

40:21 One one of the places where I see when you can transition from human

40:27 in the loop to agency in the loop

40:29 is when agents are reaching human level accuracy.

40:34 So if the expectation is humans are going to take

40:38 this decision and the agent is making the same decision,

40:41 that is the place where you can taper down transition

40:44 from human in the loop to agents in the loop.

40:47 The best example is self-driving car Waymo.

40:51 It took them 10 years to go into production

40:54 that we at least I'm from the Bay Area.

40:56 I use it a lot that we use those services a lot.

40:59 But it took them a while to take the human out

41:01 of the equation and let the machine make all those decisions.

41:05 And I have a feeling that the place where

41:09 they went from human in the loop to the AI

41:12 in the loop is when the accuracy or the decision

41:17 making capability is reached at the same level as humans,

41:19 the same analogy can be applied.

41:22 Or when you should take the human observer out

41:24 of the equation and let agents make those calls for you.

41:28 Oh, fantastic.

41:29 I was just thinking you better pick the right human to do the benchmark.

41:33 Do you want to.

41:33 Do you want a teenage driver or do you want an F1 driver.

41:38 So that's true.

41:39 And there are so many facilities out

41:41 there where you're not relying on one human.

41:43 You are relying on collective of human my thoughts.

41:47 You're combining all the thoughts.

41:48 Statistically evaluating it that if you scale it

41:51 out from 10 human evaluators to 1,000 human evaluators,

41:55 is this the right answer.

41:56 So there are good mechanisms out there to make sure that bias is addressed.

42:01 Fantastic OK.

42:03 I'm going to take some questions from our audience

42:05 since we said we'd have to address those questions.

42:08 So we have one that came in from YouTube.

42:10 It says is Gemini enterprise.

42:13 Is Gemini enterprise an option here.

42:15 I know it comes with a runtime, but not sure about GPU access.

42:18 So I'll take that one.

42:20 There are a lot of different areas of Gemini enterprise.

42:24 There's areas where it can deploy and choose the best compute platform,

42:27 and you don't even have to worry about it.

42:29 Or there's other areas to where you can customize and say,

42:33 I know I need this underlying accelerator power to be able to kick things off,

42:38 and you want to train more in depth there.

42:40 So there's a lot of different options there where you can

42:44 even choose the runtime from that standpoint and keep the questions coming.

42:48 This is fantastic.

42:50 Here's a question that came in from LinkedIn.

42:54 The idea of orchestrating specialist agents

42:57 with ADK for sustainability intelligence is fascinating.

43:01 I do agree.

43:03 I'm curious about the specific challenges faced in ensuring

43:07 these agents consistently retrieve accurate policy data from Milvus under load.

43:13 Practical insights on that retrieval layer would be gold.

43:18 Do you have some goals that we could share with everybody here.

43:22 Let's try.

43:23 I'll share my thought.

43:25 So one way to do it is the tedious way humans,

43:29 humans going in and keeping an eye on how when the policy documents are updated,

43:34 you can embed that into the sustainability framework where you're doing vector

43:39 embedding of all the documents and then retrieval will take care of it.

43:42 That is one part which is kind of a tedious part, but is the easiest part.

43:48 And I'll tell you why.

43:49 A lot of counties have their documentation online,

43:53 but they have not orchestrated MCC so

43:57 that you can access these documents in real time.

43:59 I think that is the challenge.

44:00 So till the time all these counties, cities,

44:04 states don't upgrade their information system

44:07 and make it easily accessible through MCC,

44:10 we will have to go through the manual route.

44:13 But eventually these counties will come along in providing

44:15 access to the documentation through these kinds of protocol.

44:18 So then you are getting the most up to date information.

44:22 So I think for now, if you are trying to solve something today or tomorrow,

44:26 the first thing I check is I'll start small.

44:29 I'll start with the County that I care about or the state that I care about.

44:32 Start with the state.

44:34 If all the documentations are not available

44:37 through an API call or through an agent,

44:39 sorry, through an API call, or through an MCC app server.

44:42 You are next to break it down.

44:44 Go smaller.

44:44 Go to state or County.

44:45 Sorry go to County or go to city and see if you can.

44:49 If there is any availability of accessing those documents

44:53 through an API call that is your bottleneck.

44:56 So if that option is not available for now,

45:00 you'll have to go through the manual path where

45:02 these policy documents don't change that often they do change,

45:06 but they don't change on an everyday basis, which is good news for us.

45:10 So you can always orchestrate embed.

45:13 Let's say first of every month you

45:15 do a sweep and get the documentation embedded.

45:17 And then you can orchestrate your agent in such a way

45:21 that it will always tell you when it is retrieving the information,

45:26 which data dump it is retrieving the information from.

45:30 So it gives you some human level

45:32 understanding of whether you can use this information.

45:35 Is this reliable enough based on some instance

45:38 that you're trying to rectify or you need to go to those servers and try to see

45:42 if there is a new policy document out there.

45:44 So that is where you have to play with different levels of resolution.

45:48 Go on a monthly dump, go on a weekly dump,

45:50 go from state or go to County or go to city.

45:54 Fantastic thank I hope that was some goal that you could take away from that.

45:59 I hope so too.

46:00 That was the right goal.

46:01 Not fool's gold.

46:02 That was great.

46:03 Thank you.

46:04 And Ashutosh had a really good question from LinkedIn as well.

46:08 I think this came from one of our talking about how

46:11 the agents can spawn out a multi-agent multiplies the token bill,

46:16 and how do you approach that on security, privacy,

46:20 and regulations on a company specific data when such orchestration is used.

46:28 That's a tough one.

46:32 Tough questions.

46:33 When you are trying to add up to a lot of rules that are set around,

46:39 and then you're trying to make sure

46:40 the answers are in line with what you're expecting.

46:44 And then you also don't want to burn your token Bills, right.

46:50 I think that is where the challenge comes

46:52 into play of how efficient you can make your models,

46:55 how efficient you can make your retrieval system,

46:58 or how efficient you can make your orchestrated orchestration.

47:04 Few things that comes to my mind is the biggest token burn will come

47:08 from what you're trying to ask and how your model is going through it.

47:12 So for simple questions, don't reason.

47:16 You will burn a lot of tokens in reasoning for and you

47:20 can orchestrate that in some way or the other in your toolkit.

47:24 We have done that with smart routing.

47:25 The example that I was talking about,

47:27 one of my team member has done that and it is available on GitHub.

47:31 Try and look into it.

47:32 So if you do smart routing based on the question,

47:36 you can also figure out whether you want

47:38 to keep reasoning or on or reasoning off.

47:40 That is a place that you where you can bring in the most efficiency.

47:45 The other place where you can bring in efficiency is multi-agent orchestration.

47:49 Sorry, multi-modal orchestration,

47:51 but the issue that you will face there is the GPU consumption,

47:55 because you need to keep this model floated in memory

47:58 and available for you to address the questions that you're getting,

48:02 or pick the model based on what questions you're getting.

48:05 So that is another place where I will see if I can bring in some efficiency.

48:09 But those are the two things that I

48:10 can think of where efficiency can come into play.

48:13 And then the agents multi agents is a dream

48:18 for a lot of engineers who are orchestrating systems.

48:21 But at the same time, look at the use case.

48:23 If multi-agent is not something that you need,

48:25 if you're only dealing with text and you can get away with images and voice,

48:28 then look at models that are smaller in size.

48:31 Gemma for has a two billion parameter model as well.

48:34 A smaller footprint and try to see if it is doing the job for you.

48:38 I'm sure the person who's asking this question, he knows this answers already,

48:42 but I'm just reinforcing your thought that these are

48:45 the places where if I was you.

48:47 I will explore of how I can optimize my token.

48:51 Thank you.

48:52 I think that's some really good advice.

48:55 Does anybody online have any softball questions they want

48:58 to throw out or me to keep them coming.

49:02 It's OK.

49:04 Fantastic I don't see any additional ones right now.

49:08 So I'm curious.

49:10 We'll see if there's any more that want to come in.

49:12 Where are you using agents like in your personal life.

49:16 How are you using it for you.

49:19 Everywhere everywhere.

49:20 OK, now I'm really curious.

49:22 Enlighten me.

49:23 Please yeah, sure.

49:25 So I recently orchestrated a lot of skills under the hood Where at NVIDIA,

49:32 we are a email heavy company.

49:36 So most of our important communication comes through emails.

49:38 So you need to keep an eye on emails that are coming in.

49:41 So I have AI have a brain of myself who will

49:44 go and analyze all those emails and give me a summary,

49:48 you need to orchestrate that these keywords are important.

49:51 These kinds of use cases are important.

49:53 These kinds of use cases are important.

49:55 So you can orchestrate that.

49:56 And it comes as a news digest every day in the morning.

50:00 So when I'm on my desk at 8:30 AM, I go through them.

50:04 And then I see that these 10 emails are

50:06 very important for me to address in the first hour,

50:08 and I'll go back and figure those things out.

50:12 So that is my email observation.

50:14 Other places where I have agents orchestrated for myself is there are

50:20 a lot of great community leaders out there and they publish a lot.

50:25 They put a lot of applications, use cases on their GitHub portals.

50:31 So what I try to do is I have the handles,

50:35 and I've orchestrated on a skill or an agent that can

50:38 create a news digest for me every day in the morning.

50:42 And then there are certain githubs that I would

50:45 like to see there is a code update happen,

50:47 or if there is a feature that got introduced and I

50:50 would like to be the first one testing it or experimenting with.

50:53 So I have those kinds of agents orchestrated as well,

50:56 or skills orchestrated within my agent that will

50:58 go and create that report for me.

51:00 Hey, today in simulation, MuJoCo has provided this feature.

51:06 Mujoco is from Google DeepMind.

51:08 So MuJoCo has given out, created or submitted this new feature.

51:13 Would you want to try it.

51:14 So at least I'm keeping myself up to date

51:16 with the latest and greatest information coming out model releases,

51:19 so I have an agent for that as well,

51:22 where I follow the top model providers and trying to see if a new one comes out,

51:27 I at least I'm at least aware of it.

51:29 So that is where I use agents.

51:31 And then, of course, for use cases that me and my team members we

51:35 work on, because our goal is to create these kinds

51:37 of applications and showcase it to the developer community so

51:41 that they can get inspired and build something for themselves,

51:43 either in production, for their company, or for themselves.

51:46 So we do build a lot of those things,

51:49 and we do use a lot of agents reason those things with agents.

51:54 And very important one of the person asked.

51:56 The token burn.

51:57 We are efficient.

51:58 We don't go and use the biggest model out there.

52:01 We will have small models deployed

52:03 on our small memory footprint on our desktops,

52:05 and we will try to reason with that model.

52:07 Once we have some information, we'll scale it up to bigger API calls.

52:10 But token burn, we do take that into consideration.

52:13 Token doesn't comes free to us either, even though we are at NVIDIA.

52:19 That's a good point.

52:20 No, I love the inbox summary too.

52:23 I use that built into Workspace for the top

52:26 five tasks that I need to accomplish.

52:28 And like yesterday, it caught.

52:30 I needed to return.

52:31 I got a brand new work desktop here, but I needed to return the old one

52:35 and I didn't think the box had been delivered.

52:37 So they handed me the UPS, tracking and everything that I needed to find.

52:41 Or I swear, sometimes it takes forever to find

52:44 that right tab or that the presentation that you have.

52:48 I never show my tabs.

52:49 That's too much looking into how I work,

52:51 but finding that right presentation to be able to get a task done,

52:55 I think sometimes can take a while.

52:57 So when it pulls those resources that I need to complete the task,

53:01 that's the most useful thing that I've found lately.

53:05 I totally agree with you.

53:06 I'm like you as well.

53:07 I have all my tabs open,

53:09 and I know some of my team members are used to how do you manage this.

53:12 And like, I figured out a way.

53:14 I don't know exactly.

53:18 Exactly I know exactly where I left it.

53:21 Yep great.

53:25 I don't see any other questions coming in.

53:30 But I guess I'll just say, how can people get started here.

53:35 I think we went into some really great ways to optimize your token usage,

53:40 but if you're just somebody looking

53:41 into the space and getting started with agents,

53:44 do you have any advice for them on how they

53:46 can just kick the tires and start implementing on their own.

53:51 That's my favorite question.

53:52 Oh, I'm glad I can maybe end on that one

53:55 unless somebody else comes up with another question.

53:59 No, I think that is our goal as well.

54:02 My me and my team, we work on that, evangelizing how we can

54:06 bring these kinds of use cases and motivate developers where we can get started.

54:11 There is plenty of content out there produced by Google, NVIDIA.

54:16 There is plenty of content out there.

54:18 We do joint creation of content as well

54:22 so folks can join the Google Developer program.

54:26 Similarly, we have NVIDIA Developer Program as well.

54:28 Folks can join that too.

54:30 To keep yourself up to date on the latest and greatest coming out.

54:33 Other places where I'll start is if there are community

54:38 leaders out there who are creating these kinds of use cases,

54:42 application content.

54:43 There's a lot of that available on YouTube.

54:45 There's a lot of GitHub material available as well.

54:47 If you open Gemini in your browser and ask, how can I create a single agent.

54:52 I'm sure you will get 10 to 15 links,

54:55 which are really good places to get started.

54:58 So beyond Google, NVIDIA, we produce content to enable the developer ecosystem.

55:04 At the same time, there are so many

55:06 community leaders out there who are producing that content,

55:09 and it depends on which area you are trying to create agents in.

55:14 So you're a beginner, there's a lot of content out there.

55:16 If you are going to intermediate to advanced

55:18 stages where you are looking at agents

55:20 for specific use cases like the Flipkart one that you mentioned the voice agent,

55:28 one that Mason has created.

55:30 They will have applications or use cases

55:33 of this very well orchestrated available in form

55:36 of code that you can use out of the box and try it out first,

55:39 and then you can go modify it further for the use case that you're working on.

55:43 So that's my high level view.

55:45 There's plenty of information out there.

55:47 If you Gemini app Google it is a old term.

55:51 Now, if you Gemini app it,

55:52 you'll get all of that information in your first top 10 links.

55:56 Pretty easy.

55:58 I love you making Gemini app it a term.

56:01 That's great.

56:02 We're going to just modify that.

56:04 Just Google it.

56:05 I Gemini app for everything, but well thank you everybody for tuning in.

56:12 If there was something that was useful please put it in the comments.

56:15 I would love to hear what you thought and that will help

56:17 us shape what next week's live stream is going to look like.

56:20 So appreciate you tuning in.

56:23 Thanks all for joining.

56:25 We'll see you soon.

56:26 Thank you all.

56:27 Yep hope it was helpful.

56:29 I think so.

56:30 Thank you.

56:30 Yeah bye bye.