Build intelligent Android apps with Google's AI

0:00 [MUSIC PLAYING] JOLANDA VERHOEF: Let's get started.

0:07 My name is Jolanda Verhoef, and I will be joined later by my colleague Thomas.

0:13 Our approach to AI on Android is based on three core pillars.

0:18 So first of all, the Android operating

0:21 system is evolving into an intelligent system,

0:24 and you can expose your app's capabilities to that system.

0:28 Second, you can bring intelligence

0:30 into your app using Cloud and on-device models.

0:33 And finally, you can use developer productivity

0:38 to use intelligence to build your apps.

0:40 Now, together, these three pillars represent our commitment

0:44 to embedding intelligence throughout the Android ecosystem.

0:48 And while all three of them are crucial to the future of Android,

0:52 today, I want to focus on this middle pillar.

0:55 So together with Thomas, we're going to make sure that you

0:58 learn everything about in-app intelligence today.

1:01 Now, if you want to learn more about the other two pillars,

1:04 you can watch "What's new in Android" to learn more about the first pillar,

1:09 the intelligence system.

1:11 Or you can watch "What's new in Android development

1:13 tools" if you want to learn more about developer productivity.

1:18 Now, for today, we're going to, at a high level, cover five different areas.

1:22 So I will start with on-device inference, and then Thomas will be here to talk

1:28 about cloud and to talk about hybrid inference.

1:30 And I'll be back again to talk about agents running

1:33 in the cloud and how you can connect to them.

1:35 And then finally, we'll have a sneak preview into running agents on device.

1:41 So how are we going to talk about those?

1:45 Well, what we want to do is we want to take a travel app called Jetpacker,

1:48 which our team has been building over the past weeks/months.

1:52 And we're going to start with a very basic travel app.

1:55 And throughout the talk,

1:56 we're going to make that more personalized, intelligent, and agentic.

2:01 And so right now, the app already looks quite nice,

2:04 but it isn't yet using the newest intelligence APIs.

2:08 So I actually have the app running on my device.

2:11 So without further ado, let's have a look at it.

2:15 So let's have a look.

2:17 We should see my phone.

2:17 There we go.

2:19 And as you can see, we have the Jetpacker app running there.

2:22 So let's open it up.

2:23 And you can also see in the right top

2:25 of my screen that I'm actually currently in airplane mode,

2:28 so I don't have any internet connection.

2:30 And as you can see, I have my current California trip right here.

2:34 We can look at our upcoming trips.

2:36 I'm going to Ireland later.

2:38 And then later this year, I'll be in Paris.

2:40 And we can have a quick look at our past trips as well.

2:43 So here, we have an overview of all our trips.

2:46 We can go to our Itinerary screen to have

2:49 a deeper look at what each of the days looks like.

2:52 And then we can also go to the Detail screen as well.

2:55 So far, so good.

2:57 Basic app, but it isn't yet using any form of intelligent APIs.

3:02 So let's go back to our slides and have

3:04 a look at the first feature that we want to build.

3:09 So during today's session, we're going to walk you through not one,

3:13 not two, but seven different intelligent features.

3:17 And so I think we should just get going with the first one.

3:21 So for our first feature, we're going to look at our Itinerary screen.

3:24 We know from our users that that screen can actually be quite overwhelming,

3:28 especially when you have a long trip coming up.

3:30 There's a lot of stuff happening.

3:32 So what we want to do,

3:33 we want to create a card at the top of the screen that summarizes the upcoming

3:38 trip and that includes some information on how

3:40 the user can best prepare for that trip.

3:43 So for this, what we'll do is we'll access a model

3:47 and we'll give it some input and we will get some outputs.

3:50 So in this case, we will give it the full trip itinerary.

3:53 And then as outputs, we will get a short summary and we will get

3:57 some tips on things to pack and phrases to learn.

4:01 So this setup is actually a great candidate for using an on-device model.

4:07 It has shorter inputs.

4:09 It has shorter outputs,

4:10 making sure that that on-device model can run really fast.

4:15 Now, when you use an on-device model, of course,

4:17 you have no additional cloud inference costs,

4:21 and we don't have to worry about internet connectivity,

4:23 since the model is going to be consistently working even when we

4:26 have spotty internet or if we have no internet connection at all.

4:29 And given that we're traveling, that's quite a nice feature to have.

4:34 So if we're going to want to build with on-device features,

4:37 we're going to have to use Gemini Nano.

4:40 Gemini Nano is Google's most efficient

4:43 model that is optimized for mobile devices.

4:46 We first introduced Nano a few years ago, and by now,

4:50 we have over 140 million devices running versions of Nano.

4:56 And our latest version, Nano 4, is based on the same architectural foundation

5:01 as the recently released Gemma 4 model,

5:04 and then got further optimized for maximum battery and performance efficiency.

5:10 Now, Gemini Nano 4 is coming to consumer devices later this year,

5:13 but as a developer, you can already preview and prototype with it.

5:19 And so for today's demos,

5:21 we're going to use Gemini Nano 4 for our use cases and show

5:25 you how capable that model actually is in a whole range of different scenarios.

5:31 So let's dive right into our code using the ML Kit GenAI Prompt API.

5:36 This API gives us access to that Gemini Nano model that runs on-device.

5:42 So with this API, we simply take our trip

5:45 itinerary and then we add it to our prompt,

5:48 and we ask our model to generate a summary of our trip,

5:51 along with tips on preparing for the trip.

5:54 Now, here, what I did is I just wrote a prompt myself.

5:57 I just sat down and thought, what would I ask this model?

6:01 And that's great, but we kind of want to iterate on that.

6:04 It's always good to test out a prompt before you bring it to production

6:07 and kind of see how you can make it fit well with your use case.

6:11 So let's switch back to our phone and let's do just that.

6:15 So in order to do that, I'm going

6:17 to actually close my Jetpacker app for a second,

6:19 and I'm going to go into the AICore app.

6:22 So as you can see, there's an app here that's called AICore.

6:26 And you can get that app if you join the AICore developer preview.

6:29 So this is not something that every Android user sees,

6:32 but if you join the developer preview, you get access to this app.

6:36 It is available on all devices that support AICore.

6:39 And if you don't have a physical device with that model running,

6:42 what you can do is you can actually also go

6:45 to Android Studio and you can use the Gemma 4 model there.

6:49 So let's have a look at the AICore app.

6:51 As you can see, we have Gemini Nano 3 there,

6:54 but we also have Nano 4 Fast and we have Nano 4 Full.

6:58 So here, we can choose which of the models.

7:00 Now, I already downloaded them just to make sure

7:02 that this talk isn't taking an hour on this Wi-Fi.

7:06 So let me open the fast model and have a look.

7:09 So let's start with the prompt that I thought of myself originally.

7:13 So I just copied it over and I put in my Paris trip.

7:18 So when we do this, as you can see,

7:20 this is generating a response, but it's generating quite a long response.

7:27 This is taking quite some time.

7:29 There's a lot of information.

7:30 And we were trying to make things easier

7:32 to parse for our users, not more difficult.

7:35 So we want to make sure that we're

7:36 not spending this much time generating this much text.

7:40 So this is where we start iterating.

7:42 Now, I did some of that iteration and I actually came up

7:44 with the final prompt that might be useful for us to look at.

7:49 So here, what I did is— let's first take a look.

7:52 We got a very fast response.

7:54 And as you can see, I added a little bit more guiding for the model.

7:58 So I told it to use one sentence, less than 15 words, et cetera.

8:02 And this is kind of a good way to guide

8:04 the model into the response that we actually want to have.

8:07 And with that, you could see that it started immediately generating a response.

8:11 And it took only, what is it?

8:13 1.6 seconds to generate it.

8:15 Now, let's have a look at that in our Jetpacker app.

8:18 So I'll go back to the app.

8:20 I'll go to my secret debug menu, turn on the feature,

8:24 and I'll go to my upcoming Paris trip.

8:27 And pay good notice, because this is going to be fast.

8:30 So there we go.

8:33 We immediately have it coming up.

8:35 [APPLAUSE] Amazing.

8:40 Let's switch back to our slides.

8:44 With that, we can mentally prepare for our trip

8:47 and we can move on to our next feature.

8:50 And for me, if I'm traveling,

8:52 one of the things that I really don't like to do that much is budgeting.

8:57 And it's managing how much I'm spending on souvenirs,

8:59 on food, on all those things.

9:01 So I usually end up with a whole bunch of receipts

9:05 that are all in my backpack and in my pockets,

9:08 and then I have to sort through and tally all of them.

9:11 So for such a tedious task,

9:13 wouldn't it be nice if we had AI to help us with that?

9:18 And since receipts might contain sensitive

9:20 information like credit card numbers and addresses,

9:22 this is another great use case for using

9:25 an on-device solution so users can be confident that private

9:29 information will actually be processed locally on the device

9:32 without any of their data being sent to the cloud.

9:36 Nano 4 excels at multimodality,

9:38 and so this especially includes image understanding

9:41 for tasks like OCR and visual data extraction.

9:46 So naturally, it will excel at extracting information for our receipts as well.

9:51 So let's have a look at how this works.

9:55 In our last example, we only passed in a text prompt.

9:59 But in this example, we'll pass in both a text and an image to be processed.

10:05 And then for our text prompt, we will specify that we want three

10:08 pieces of information here based on the receipt.

10:11 We want a generated title.

10:12 We want to get an amount and a category.

10:16 And to make sure that the returned result is consistent,

10:19 we'll also tell the model that we want it to return a certain JSON object.

10:25 So let's see this feature in action before we continue with the slides.

10:29 Let's go to our phone, and let me quickly turn on this extra new feature.

10:35 I have to shake my phone in order to go to the debug menu.

10:38 There we go.

10:39 I'll turn on the expense management and I'll go to my current trip.

10:43 So we'll go to the California trip.

10:45 And actually, this morning, I was very,

10:48 very tired because I'm from the Netherlands and I got jet lag.

10:51 So I got someone to bring me a coffee and I actually got the receipt.

10:55 So let's take an image of that receipt.

10:58 So I, here, have my expense menu.

10:59 I already went for dinner.

11:00 I already did some stuff while I was here in San Francisco.

11:04 But now, let's take a picture of our receipt.

11:08 Here we go.

11:08 Say hi, everyone.

11:09 Yay.

11:11 And let's take a picture.

11:12 There we go.

11:15 And this is taking a little bit longer because

11:17 for this, I'm actually using the Gemini full model.

11:20 So you saw that we had a fast and we had a full model.

11:23 And in this case, we want to have that full image recognition,

11:26 so we're using that more complex model for us.

11:29 And as you can see, it recognizes that we

11:32 did an espresso blend purchase, which was $5.94.

11:36 That is correct.

11:37 Amazing.

11:38 Cool, let's go back to the slides.

11:40 [APPLAUSE] I'm always so happy when demos actually work.

11:47 [LAUGHS] This one was the scariest one, but hey, there we go.

11:50 And as you saw, this is already working pretty well, right?

11:53 But you might have noticed that we had to explain

11:55 to the model what kind of fields that we need,

11:58 and we had to give it a JSON example of what that then actually looks like.

12:03 And then after we wrote that whole prompt,

12:05 we then had to go and parse the return data,

12:07 which we hope is that JSON, into a data class that we can use in our app.

12:13 This is a lot of manual steps, and all of those manual steps can go wrong.

12:17 So it's kind of scary.

12:19 And this is why we're introducing a Structured Output API.

12:23 This API is going to land soon,

12:25 but let me give you a quick sneak preview into what it will look like.

12:30 So with our Structured Output API, we can mark any data class as generable.

12:35 And then we can further describe the class

12:38 using guides for each of the properties.

12:41 So these annotations basically guide the model and tell

12:44 it what kind of data it should generate.

12:48 If we do this, we can actually now, for our request, simplify our prompt.

12:54 We don't have to guide it anymore there.

12:57 And then we can use the method called generateTypedContentRequest that takes

13:01 the parsed receipt class name to restrict what the expected output should be.

13:06 Let's make sure I don't lose my receipt.

13:10 And now, our response simply contains this parsed receipt instance,

13:14 or it will return null if it wasn't able to find one.

13:18 So with structured output, we can be more confident about consistent outputs,

13:22 and our code is much, much simpler, with Kotlin data classes

13:26 that integrate seamlessly with the Android code base.

13:31 Now, during this talk, we prototyped some pretty cool features with on-device

13:34 APIs using the latest Gemini Nano 4 preview model, but this is just the start.

13:40 We already have seen lots of apps using the Prompt API in production

13:44 to leverage the power of Gemini Nano for use cases such as classification,

13:49 summarization, content inspiration, those kind of things.

13:54 If you want to learn more about how you can get the latest in on-device Android,

13:59 you can check out this talk, where we are going to go more in depth

14:02 on how you can ship on-device use cases to production.

14:06 Next, I will hand it off to Thomas,

14:08 who will walk you through the next couple of features.

14:11 [APPLAUSE] THOMAS EZAN: Thank you, Jolanda.

14:15 So yeah, Gemini Nano 4 is pretty impressive,

14:19 and it's really exciting to see how on-device

14:22 models are getting better and better every year.

14:25 So we talked about Gemini Nano that is

14:28 basically optimized to run on your Android device efficiently,

14:32 but sometimes you have use cases that require an AI model

14:37 that has either a greater world knowledge or a much larger context window,

14:43 or sometimes even greater ability in handling complex tasks.

14:48 So let's take a look at what we can build with these cloud models.

14:53 So we are going to use them to build our next feature in the Jetpacker app.

14:58 So we are going to build a museum assistant.

15:01 So it will be a chatbot that is powered by Gemini Flash

15:04 and enables you to ask questions to plan your next visit to, let's say,

15:10 the Louvre Museum in Paris.

15:13 So first, you need to add the Firebase AI Logic to your project.

15:18 And once you do that, with just a few lines of Kotlin code,

15:23 you can directly call the Gemini models that are running in the cloud.

15:29 You will start by instantiating a generative model,

15:32 and you will pass the specific version of Gemini that you want to use.

15:37 And then you basically just have to call start chat to create a chat.

15:43 And the cool thing about the SDK is

15:45 that it will actually manage the chat history for you.

15:50 So the communication between your app and the model

15:53 is actually fully handled by the Firebase SDK.

15:57 There are three series of Gemini AI models

16:01 that are running in Google Cloud that you can use.

16:04 So there is Gemini Pro, which is really good for complex tasks.

16:08 There is Gemini Flash, which has a very low latency.

16:12 And Gemini Flash Lite, which offers a really good intelligence cost ratio,

16:18 and also has a very low latency.

16:23 So we want to build a chatbot that is really knowledgeable about the museum.

16:28 And earlier, I mentioned that cloud

16:31 models store massive amounts of world knowledge.

16:34 So you can assume that Gemini already knows quite a lot about the Louvre,

16:40 but it doesn't necessarily know everything there is to know about it.

16:45 So for example, if you ask the model about

16:48 the current temporary exhibitions that are happening at the Louvre,

16:52 the chatbot will basically tell us to go

16:55 to the website and search for it ourselves.

16:58 So it's not super helpful.

17:01 So what we can do to prevent this is use a technique called grounding.

17:06 So in the context window, which is the full input that is passed to the model,

17:10 we are going to add some relevant content

17:13 to help the model answer the user's query better.

17:17 So grounding techniques are also used

17:19 in the context of Retrieval-Augmented Generation, or RAG.

17:25 So you can implement it yourself,

17:27 but we also made it easy for you with the following APIs.

17:32 There is URL grounding,

17:34 which enables you to use a document at a specific URL, there is maps grounding,

17:41 which uses the data from Google Maps,

17:44 and there is web grounding that is using real-time web data.

17:49 So let's see how we can use them.

17:51 We are going to start with URL grounding.

17:54 So let's imagine that you want to prioritize some data

17:58 from a specific page that is hosted on the web.

18:01 So you will start by adding the URL context tool

18:05 to the list of tools when you instantiate the model.

18:09 And then in the user prompt, we will tell the model to look at the specific

18:14 page at the URL to grant the response.

18:18 And so you can see that here, we are actually using the FAQ URL of the museum.

18:23 So here, we are only using one URL,

18:26 but you can actually use up to 20 URLs per request.

18:31 And so now, if we ask the chatbot about,

18:34 for instance, criteria to get discounted tickets,

18:38 for example, Gemini is going to actually use

18:41 the content of the FAQ page to answer.

18:43 So if I say, for example, if having a cat named Mona Lisa grants you

18:48 a discount to get a ticket at the Louvre,

18:51 you will see that basically, the chatbot is saying no.

18:55 And as accurately outlined in the FAQ, that is not enough to get a discount.

19:02 If we want to give the chatbot

19:04 the ability to retrieve some information from Google Maps,

19:07 we can use the Google Maps tool.

19:10 So that will, for instance, enable the chatbot to search for cafes

19:15 or restaurants around the museum after our visit.

19:19 So if, after adding the tool, when I ask,

19:23 what are the best restaurants that are good for children next to the Louvre,

19:28 we will actually get a pretty comprehensive response from the chatbot.

19:32 It will query the data from Google Maps and return a list of restaurants,

19:39 and it's actually anchoring on restaurants that are

19:43 clearly known for having a children's menu.

19:47 And so we can also use web search grounding to, for instance,

19:51 let the chatbot search for more recent information on the web.

19:56 So we now have a pretty competent museum

19:59 chatbot to enable us to prepare the visit.

20:03 And so the cool thing is that we built

20:05 all of that with Gemini Flash in Firebase AI Logic.

20:10 And actually, the Gemini Pro and Gemini Flash models are perfect to build,

20:14 let's say, the first iteration of your AI feature.

20:18 So whether you need multimodal understanding,

20:21 JSON generation, function calling, or even image generation,

20:26 you can quickly build everything in Kotlin without

20:29 really needing any server code or any backend resources.

20:35 And the cool thing is that if your feature

20:37 actually doesn't rely on your own backend infrastructure,

20:41 you can really go to production with it and scale it.

20:44 You can use AI monitoring features in the Firebase AI Logic SDK.

20:50 You can use server prompt templates and App

20:52 Check and basically scale to your whole user audience.

20:58 And if you think about it, you probably already have some features in your app

21:01 that are good candidates to use Gemini in Firebase AI Logic,

21:05 so I strongly encourage you to look into it.

21:10 So it's actually used already by hundreds of apps in production.

21:13 So for instance, the redBus application, which is a ticketing platform,

21:18 uses the Gemini audio understanding capabilities to create a voice-first,

21:24 multilingual review system.

21:27 It's also used by Karrot, which is a marketplace app that basically uses

21:32 it to translate messages between buyers and sellers.

21:37 Gratitude, a journaling app, is using it to suggest journal entries.

21:43 And finally, UltraHuman, which is a fitness tracker app,

21:47 is using it to build a food

21:50 logging feature using the Gemini image understanding capabilities.

21:55 So a lot of interesting use cases.

22:00 OK, so let's take a step back for a second.

22:03 So we saw that many of you want to build

22:06 with Gemini Flash and Gemini Pro using cloud inference.

22:10 And earlier, Jolanda showed all the cool

22:12 things you can build with Gemini Nano on-device.

22:15 But we also heard some feedback from you.

22:18 You basically need to support users on devices

22:21 that don't necessarily support Gemini Nano just yet.

22:26 So what if we could just combine both, right?

22:29 That's actually the promise of hybrid inference.

22:32 And I think I have a really good use case for it.

22:35 So reviews are a pretty key element in most Android applications today.

22:41 So in your app, maybe you're asking your users to write other users,

22:46 or write a business,

22:47 or even a piece of content, like, I don't know, a song or a show.

22:52 And so you're often asking your users to basically write a review,

22:56 or sometimes you're even asking users to post

22:59 a review for your application in Google Play.

23:02 And it's important for you, but it can be sometimes tedious for your users.

23:06 So we think Gemini Nano can actually be super helpful for that.

23:11 But you also need to have this feature available

23:14 on devices that don't support Gemini Nano just yet.

23:19 So in the Jetpacker app, earlier, the chatbot museum recommended restaurants.

23:25 And so what if we were to implement a restaurant review feature in the app?

23:31 So here, you can see that we have a list

23:33 of topics to choose from about how we felt about the restaurant.

23:38 And we will let the model draft the short review for us.

23:42 And the user is always going to be able to edit it later before posting it.

23:47 So here, you can see we are selecting some topics.

23:50 And once we are done, we are clicking on generating the review.

23:55 And basically, if Gemini Nano is available on the device,

24:00 the review will be generated by Gemini Nano.

24:04 But if not, it will default to Gemini Flash Lite in the cloud.

24:07 So it's completely transparent for the users,

24:10 and it's also very easy for you to build as a developer.

24:14 So to build this, we are actually using the latest API from Firebase AI Logic.

24:20 So to do that, you need to add

24:23 the on-device Gradle app dependency to your project.

24:27 And then— so you can see here that we basically generate

24:30 a review with a prompt that is interpolating the selected topics.

24:37 And then when we instantiate the model,

24:40 you can see that we are passing an onDeviceConfig object,

24:43 where we define the inference mode that we want to use.

24:47 So here, you see that we are saying prefer on-device,

24:51 so we will prioritize on-device inference.

24:55 So this API supports four inference modes.

24:59 So there is prefer on device, prefer Cloud.

25:02 And there is also only on device and only cloud.

25:07 And it's actually available today, so you can try it out and you

25:11 can learn more about it in the Firebase documentation.

25:16 So we just talked about a very simple routing logic.

25:20 Here, it's basically two criteria.

25:23 Is Gemini Nano available on the device?

25:25 And is the device online?

25:28 But sometimes you want to build something that is a little bit more complex,

25:33 so you can imagine having a pretty advanced custom routing logic.

25:38 So for instance, you can monitor the system health of the device,

25:43 and you can decide to use a cloud model if,

25:46 let's say, the CPU is overloaded or if

25:51 the battery of the device is below a specific threshold.

25:56 And another thing worth checking is also the latency.

25:59 So if the latency is above a specific threshold,

26:02 you can decide to focus on on-device.

26:06 Another thing, too, to keep in mind,

26:07 is that not every single query requires a very

26:12 advanced model running in the cloud, and some do.

26:17 So it can be interesting sometimes to use a lightweight LLM, like for instance,

26:22 the Gemma 3 model that is only 270 million parameters,

26:28 to basically route those queries based on their complexity.

26:32 And at the end of the day, I'm just listing three criterias here,

26:35 but there are many other criteria that you want to take into account based

26:40 on your app's needs or also the user experience that you want to provide.

26:47 So if we go back to the Jetpacker app,

26:50 let's say that we now want to implement instant messaging translation

26:53 for a chat feature between the hotel staff and the customers.

26:58 So the customers and the hotel staff

27:00 will basically use their own language to chat,

27:03 and we'll use the AI for translation.

27:06 So let's see how we can use hybrid here.

27:09 So let's imagine that we want

27:12 to prioritize on-device to save on inference costs.

27:15 We also want low latency.

27:18 So we provide a really real-time experience to the users.

27:22 And we also want to make sure that we

27:26 basically support as many languages on as many devices possible.

27:32 So we know that on-device translation capabilities

27:35 are getting better with Gemini Nano models,

27:39 but we also want to be able to fall back

27:41 to the cloud if the model doesn't support the language.

27:45 And so in this case, we'll use TranslateGemma,

27:48 which is an open-weight model based off of the Gemma

27:51 model that can be potentially fine-tuned if you want.

27:57 And so here is the inference stack that we are going to use.

28:00 So for on-device inference,

28:02 we will use Gemini Nano models through the ML Kit Prompt API.

28:07 And for the cloud inference,

28:08 we will use TranslateGemma that will be hosted on Google Cloud.

28:12 And we will also use the on-device ML Kit Language Identification API

28:16 that will be able to detect the language used in the conversation.

28:23 So we will implement the appropriate routing logic based

28:27 on the version of the model that is available on the device,

28:30 the language that is used, and also the latency of the network.

28:35 And so here, in this diagram,

28:37 we are basically implementing the routing logic in code.

28:40 But there are already experiments to use AI to actually define the routing.

28:45 And in a minute, Jolanda is going to show

28:48 us how the agentic approach can help us with that.

28:52 So let's see the translation feature in action.

28:55 So let's say that I'm typing in Dutch,

28:58 but I'm basically talking to a hotel in France,

29:01 and the staff speaks mainly in French.

29:04 So I will send my message and I will receive

29:08 a response that basically enables me to translate— using hybrid inference,

29:16 translate the message in Dutch.

29:18 And on the other end, the staff can also have the same setup,

29:22 where they can basically translate the message

29:26 that was sent by the user to their language.

29:28 And so here, the translation will either be on the device or on the cloud,

29:33 based on the criteria that we talked about before.

29:36 It will be, again, completely transparent for the users,

29:39 but we'll have a very high-quality translation

29:42 feature that is running via hybrid inference.

29:47 So this custom hybrid inference approach is

29:50 already used by several applications in production.

29:53 For instance, the Gboard app is using that.

29:56 It's also used in the Pixel Screenshots application.

29:59 And the Kakao mobility app also started using it.

30:04 And so with that, I will pass it back to Jolanda to talk about agentic systems.

30:10 JOLANDA VERHOEF: Thank you, Thomas.

30:12 [APPLAUSE] OK.

30:15 For our next feature, we want to take our app to the next level

30:20 and we want to connect it to our cloud backend.

30:24 So we have a Jetpacker backend running

30:27 in the cloud that is able to reserve flights,

30:30 hotels and activities on our user's behalf.

30:33 Now, in our app, we simply want the user to be able to go to the booking screen

30:37 and then automatically kick off that booking process.

30:40 And it will automatically book all the things that that user wants to do.

30:44 And then the booking process will be running fully autonomously.

30:48 And our Android app is just showing the progress

30:50 of all of these different booking agents working.

30:54 And it will let the user provide input when needed.

30:58 So before we show how we can build this, let's

31:01 go back to the phone and look at it in action.

31:05 So in order to run this, let me first turn off my airplane mode,

31:09 because we have a backend this time around.

31:12 So we have internet going on, I hope.

31:15 And we'll open our Jetpacker app again, and we'll turn on the Booking Assistant.

31:20 There we go.

31:22 Now, if I open my current app, let's say, or my current trip,

31:26 I can go to the booking screen and I can start our booking.

31:30 And as you can see, it is connecting to our booking server.

31:35 Now, while this is connecting, as you can see,

31:37 this is taking a little while because I need to connect

31:40 to this server that is running that is doing a whole bunch of stuff.

31:44 I do hope it will connect in time, before the end of the talk.

31:48 It might be that my Wi-Fi is not as strong as I was hoping it would be.

31:52 Look, there it is.

31:52 OK, amazing.

31:54 So as you can see, what we did is we basically just sent

31:56 our itinerary to that server and it started up a session for us.

32:00 And it's then kicking off all of these different booking agents for me.

32:04 So it's kicking off four agents to book activities.

32:08 So here, I see my lunch and my dinner and all kinds of bookings there.

32:13 Same for my hotel, and then the same for the flights.

32:16 But all of those need some kind of information from me.

32:19 So for example, let's say that for my SFO to Amsterdam flight,

32:23 I will confirm that indeed that is the right time frame for me.

32:27 I can confirm the selection and I get to choose a chair.

32:31 However, in the meantime, while all of these agents are running,

32:35 if I'm not doing anything else, I quit the app.

32:37 I go do something else, and I come back tomorrow.

32:40 And I come back into the app.

32:41 I can reconnect to that existing session.

32:43 So now if I go back to my ongoing connection, as you can see,

32:48 it will just reconnect and it will find all of those agents that were running.

32:52 And surely, it should show all of them.

32:56 It is, again, maybe not the fastest Wi-Fi here.

32:59 But in the meantime, as you can see,

33:01 it's still asking me— it still remembers where I was in the process,

33:04 so it's still asking me for that seat number.

33:07 So I can confirm that seat number.

33:08 I can say I confirm my whole booking.

33:11 And there we go.

33:13 As you can see, I think my internet is fully out now,

33:15 but this would normally show the full confirmation and we'd be done.

33:20 There's not a full talk that you can do without having one demo hiccup, right?

33:24 So this is the one.

33:25 [APPLAUSE] I already assumed it would be

33:30 the one where I'm actually using the internet.

33:34 So this is why we have on-device inference, people.

33:36 [LAUGHS] OK.

33:39 So before, Thomas was already speaking

33:41 to you about Firebase using a managed backend.

33:44 And this is ideal for getting started or when,

33:47 as a mobile team, you want to develop in isolation.

33:50 And it provides and focuses on direct client-to-model inference,

33:54 which makes it kind of, fundamentally, stateless.

33:58 And so for our trip booking, we instead use a custom self-hosted backend.

34:02 This server, which we fully control and gives us full flexibility.

34:07 And so when we book sessions, even when the user closes the app,

34:11 that system can just continue running in the cloud.

34:14 It also gives us complex multi-step orchestration.

34:17 And because it's running in our own backend,

34:19 it gives us access to our company's data.

34:23 So how would we architect such a solution?

34:25 Well, if we go and take it all the way back to its core,

34:28 we basically have a cloud backend and we have an Android app,

34:31 and they somehow communicate with each other.

34:34 Now, we want that to be, basically,

34:36 a server or an event-driven communication that goes both directions.

34:41 So some ways of doing that transportation would

34:44 be to use server-sent events or to use WebSockets.

34:47 Then, we have our connection set up between the two,

34:49 but then what do we run in the backend?

34:52 So we wanted to have a complex, multi-agent booking system,

34:55 and we chose to use the Google build ADK, or Agent Development Kit,

35:00 which is a flexible and open-source

35:02 framework for developing and deploying AI agents.

35:06 Now, for Android developers like you and I,

35:08 it might be a little bit scary to look at backend systems,

35:12 and it might be a little bit intimidating.

35:14 But just to give you an idea,

35:15 let's actually look at the implementation of the backend that we wrote.

35:20 So we can define an agent by setting their name, giving them the model that they

35:25 should communicate with, giving them an instruction,

35:28 and then giving them a set of tools.

35:31 That's all we need to set up one simple agent using ADK.

35:35 Then we can make a top-level orchestrator

35:38 that can delegate work to several subagents.

35:41 So here, for the restaurants, for the hotels and for the museums,

35:44 I'm making a bunch of different subagents.

35:47 So when we have such a backend set up, there's one question that remains.

35:51 It's running in the clouds.

35:53 We have an Android app.

35:54 How do we communicate?

35:55 What kind of messaging protocol do we use?

35:58 Now, we could be making our own.

36:00 We could set up our own kind of communication pattern,

36:03 decide what are the requests that we're sending, what are the responses?

36:07 However, once you start adding more complexities to such a backend system,

36:11 it can become quite unwieldy and difficult to deal with.

36:16 So instead, we can use the AG-UI protocol.

36:19 This standardizes communication between agents and UI clients.

36:23 So the protocol predefines a set of message types

36:27 that have been designed to best represent agent-to-client interactions.

36:32 So the agent can inform the client about, for example,

36:35 its lifecycle events, text messages, tool calls, state management, et cetera.

36:39 And the other way around,

36:41 we can have our Android app that is sending text messages,

36:44 providing tool call results, and sending custom events.

36:49 Now, most agent frameworks have an AG-UI adapter module.

36:54 So that allows them to interact using that AG-UI protocol.

36:57 And similarly, there's also several AG-UI libraries for clients.

37:01 So for example, for Android, there's a Kotlin SDK that gives you a great

37:05 Kotlin API to communicate with any AG-UI-enabled backends.

37:10 So as an Android developer, at that point,

37:13 all that you need to do is set up the connection to your backend and then

37:18 decide how you want to visualize those AG-UI

37:21 messages that are coming in from your server.

37:24 So in our use case, we're parsing the messages that are

37:26 coming in and we're showing them in our nice UI.

37:31 This is already quite great, but I believe we can do better.

37:36 So if we're deciding— so the step two that we just mentioned,

37:40 deciding how to visualize the events coming in, that is actually quite

37:44 a lot of work and quite a lot of kind of figuring out, right?

37:49 It doesn't really scale that well

37:51 and there's some version dependency complexities.

37:53 Like, what if my backend is updated

37:55 but my Android app is still older, et cetera?

37:58 So what if, instead,

38:00 we could have our app describe a set of UI components that it is able to render,

38:05 and then let our AI model speak UI instead?

38:10 So what do I mean with that?

38:12 So A2UI is a protocol that allows agents to render UI in a client.

38:17 It works really well together with AG-UI as a transfer protocol,

38:21 but the client simply informs the agent which components it can render,

38:25 and then the agent that's running on the server decides,

38:28 what is the best UI component that I can use, the best tree, the best hierarchy,

38:33 to then do what I want to do,

38:35 or ask the user for input, or to reach my goal, basically?

38:40 And as an Android developer, that means that the only thing that we still

38:43 have to do is define our set of components,

38:46 which might be composables that you already have in your app,

38:49 provide them to your server, and then render them.

38:54 So let's have a look of an example of what that communication looks like.

38:58 You can see the component tree being passed from the server

39:01 to the client to be rendered at a given point in time.

39:04 And you can see that A2UI supports both basic components,

39:07 like we have column and text here, but it also supports custom components.

39:12 So for example, here, I added support for a my progress spinner.

39:16 You can also see that we're not actually passing actual data at this point.

39:21 Instead, the server is just passing variable names.

39:24 And then in a separate message, after it's sent the components,

39:28 it can actually gives you data associated with those variables.

39:32 So all of this is pretty cool.

39:33 But at this point, we're just passing JSON, right?

39:36 It's just JSON coming in, JSON going out.

39:38 What does that actually mean?

39:40 That's not UI on a screen yet.

39:43 So how are we going to render that, specifically on Android?

39:46 So on Android, we are currently building a Jetpack Compose Renderer.

39:52 It will allow you to automatically render those A2UI

39:55 messages that are coming in from your server.

39:59 [APPLAUSE] This A2UI renderer has a set of basic components

40:05 that will give you great out of the box experiences,

40:08 so you don't have to do anything special here.

40:11 And when needed, you will also be able to provide

40:13 your own custom components based on the composables in your app.

40:18 So that's what we did in the app

40:19 to constrain our model to generate our custom booking-related components.

40:24 So now we can build beautiful UI

40:26 and let the agent show specific components when,

40:28 for example, we're choosing a seat.

40:32 Now, in our code, we create an A2UI

40:34 message processor and pass our custom catalog to it.

40:37 From that processor, we then retrieve active surfaces.

40:41 And in our UI, we simply retrieve the surfaces that are coming

40:44 in and display them in a column using the material A2UI surface composable.

40:51 So to summarize, we have A2UI on the one hand,

40:53 which is our transport layer communication protocol,

40:56 and it provides an event-based

40:58 communication pipeline between agents and clients.

41:02 And A2UI is our independent protocol that helps an agent speak UI,

41:06 which can then be rendered into native components on the client side.

41:11 Now, A2UI is just message passing and it works over any transport.

41:14 So it could also use REST, MCP, A2A, or AG-UI.

41:19 Now, we're at the start of these protocols, so please,

41:21 please do give a look at a2ui.org to learn more and stay posted.

41:29 OK, let's get to our final, final feature.

41:32 So we already saw how we can connect to an agent running in the cloud.

41:35 What if we want to instead run an agentic system on our device?

41:40 Now, this is very, very, very early and experimental,

41:43 but I did want to give you a heads-up of what we're

41:46 thinking about and what we're developing

41:47 and what we're moving towards as Android.

41:51 So let's see if we can build an agentic system

41:53 that hosts multiple agents that all communicate with each other,

41:57 orchestrated from within our Android app.

41:59 Let's switch to our app and quickly show it in action.

42:04 Hoping that the Wi-Fi is fixed by this point.

42:07 So I'll turn the feature on.

42:09 This is called the Genius Assistant.

42:10 And our Genius Assistant is going to help

42:12 us during our trip to understand what's going on.

42:16 Now, it will first go to our cloud,

42:18 and it will go and try to understand what's going on in the trip.

42:22 So it parses the itinerary,

42:23 and then it understands that I have a hotel and a car reservation.

42:27 So let me check my hotel reservation.

42:30 What it will do is it will

42:32 actually understand that there is multiple agents running,

42:35 and it can understand that it can transfer to our hotel agent.

42:40 Now, our hotel agent is a subagent that runs on-device,

42:43 because I have my personal data running on my device as well.

42:47 So I have some personal information here.

42:50 I have my passport and I have the hotel reservation.

42:53 And the model will actually compare the two.

42:55 So here, there's multiple agents running.

42:57 And I'll show you later how this is set up.

43:00 But this is pretty cool.

43:01 And we have a way to understand that for my hotel reservation,

43:04 actually, I provided the wrong passport number.

43:08 So there's a lot going on here.

43:09 And I'll show you how that actually works.

43:12 So let's go back to the slides and dive into it.

43:17 So one way of doing this on Android is using ADK for Androids.

43:24 [APPLAUSE] This is a brand-new library.

43:28 We just released version 0.1 this week.

43:32 And it takes all the learnings that we had from ADK on the cloud,

43:35 which we've seen in our previous section, and it applies them to Androids.

43:40 ADK manages all the complex orchestration,

43:42 context handling, and error handling for you.

43:45 It allows you to build an agent with just a few lines of code,

43:50 easily swap out models, et cetera.

43:53 Now, for our trip assistant feature,

43:55 we first designed a main orchestrator that uses Cloud Intelligence.

43:59 It then calls tools to get information about the trip,

44:02 chooses to offload work to a subagent that we run fully on-device,

44:06 and then each of the subagents runs a set

44:10 of agents in a sequence and shares the data between them.

44:14 So we only ever return the final results

44:16 to the orchestrator that are stripped of any privacy-sensitive data.

44:20 So in our code, what this looks like is we have our trip assistant orchestrator,

44:24 which uses a cloud model for its intelligence and has a set of subagents.

44:31 And we give it a tool, as we saw before,

44:34 and we give it some top-level instructions so

44:36 it knows when to offload to which agent.

44:39 Then each subagent is defined as an on-device sequential agent,

44:45 and we have various retrieval agents that can use the device

44:48 Nano model to retrieve the correct data from our user's document.

44:53 Separately, we have that last validation agent that we saw

44:56 that can compare the data coming in from those retrieval agents.

44:59 And together, we have a whole set of agents

45:01 set up with just a few lines of code, using both on-device and cloud,

45:07 working to build something that is highly intelligent for your user.

45:12 Now, as I said, ADK for Android is just starting,

45:15 and we're looking forward to see what you build with it.

45:17 Check out adk.dev for more.

45:21 Now, with that, we completed all seven features

45:23 and we made our app so much more useful.

45:26 So while we're working on making

45:28 the Jetpacker source code available to you soon,

45:30 in the meantime, please go to developer.android.com/ai to learn more.

45:36 And with that, thank you for joining, and have a great I/O.

45:38 [APPLAUSE] [MUSIC PLAYING]