Building the most intelligent agents with the latest knowledge sources | BRK318

0:00 Hello and welcome everybody.

0:03 My name is Hardik, I am a group product manager at Microsoft.

0:07 I work on copilot Studio and my team along with Eli and Anjali,

0:10 we focus on making knowledge better for all

0:13 the agents that you build on copilot Studio.

0:16 We are going to talk today about how you can build

0:19 your agents with the latest innovations we have made in knowledge.

0:23 We are going to share with you some learning, some best practices,

0:25 and for the first time take you through the inner workings of how our rag works.

0:29 And later we'll hear from Nick who leads the IT at Gilead Sciences.

0:34 He is the head of IT Innovations and we are going

0:38 to hear from him how they have built an agent called Genie,

0:41 which helps their employees be more productive as well as more informed.

0:47 So let's get this started.

0:49 You know, before we go into the exciting demos,

0:52 I do want to take a few slides to tell you about Knowledge.

0:56 Let's unpack knowledge for a minute.

0:58 You know, there are a few

0:59 characteristics that every enterprise would resonate with.

1:02 When we talk about enterprise knowledge,

1:03 the 1st is, you know enterprise knowledge is vast.

1:07 Every company uses multiple products for various business needs, right?

1:12 We at Microsoft also use multiple products and all the knowledge

1:16 that your employees create are in all these different products.

1:19 For example, the documents are in SharePoint, your emails are in Exchange,

1:24 your Teams messages, your conversations happens on Teams.

1:27 And maybe for IT services you are using ServiceNow, right?

1:30 And so your enterprise knowledge,

1:31 when you think about it, it's spread across these products.

1:35 The second thing about knowledge is it's complex.

1:38 Like even talk about SharePoint, it's not a single knowledge source.

1:43 SharePoint has entities, it has site pages,

1:45 it has document libraries, it has lists.

1:49 Even in document libraries, you can have URLs,

1:52 files, folders as well as images, right?

1:55 So when you think of knowledge, remember that it is complex.

2:00 Well, the hard truth about knowledge is it does not follow a structure.

2:04 No matter how hard you try to enforce a structure,

2:07 overtime knowledge does get messy, right?

2:10 Say for this example here the onboarding

2:12 guide for remote employees is actually misplaced.

2:16 It should be under that folder.

2:17 There are duplicates, like there are two folders for United Kingdom.

2:20 And knowledge is not stale, it is constantly updating.

2:24 So you have multiple versions of a policy,

2:26 some is in draft, some is already published, right?

2:30 And then it begs the question, you know, if knowledge is vast,

2:33 if knowledge is complex and it doesn't follow structure,

2:36 how can one define knowledge?

2:38 Well, there is no canonical definition of knowledge.

2:40 But working with multiple organisations,

2:42 helping them build agents and ship it to production,

2:45 we have realised, you know, briefly or at a high level,

2:48 you can say knowledge is a logical collection of one

2:51 or more entities that serve a specific purpose or an intent, right?

2:55 So I'm just going to show you what I mean here.

2:58 You know, when you add knowledge to an agent, here are some best practices.

3:02 So first is organise, just to make sure you have all your documents organised

3:06 so that the agent can access it in a meaningful way.

3:10 Second is clean.

3:11 This includes removing duplicates, getting rid of stale information.

3:15 Third is modularise.

3:19 You know, we have worked with many organisations and we ask them,

3:21 hey, can you show us your knowledge?

3:22 They typically just pointed to a whole SharePoint website.

3:26 Well, yeah, I mean, SharePoint website can have a lot of things,

3:29 but a better way of adding your knowledge is like this.

3:31 Like the first box here is about onboarding and boot camp,

3:34 the second is about benefits, the third is about expenses,

3:37 and the fourth one is to answer questions about when is my next holiday, right.

3:42 So please do not throw the kitchen sink at your agent

3:45 because what we have seen is when knowledge is ambiguous,

3:48 your agent might hallucinate or it might just ground it on the wrong data.

3:54 With that, I'm going to invite Eli,

3:56 who's going to take you to us through a process of building

4:00 an open enrolment advisor agent

4:01 and for a fictitious company called Northwind Care.

4:05 This company is building agents to make

4:07 their employees more informed and more productive.

4:10 Welcome, Eli.

4:12 Thanks so much.

4:15 Thanks so much.

4:16 And let me make sure, of course,

4:18 that the demo machines are actually on and and ready to be used.

4:22 Great.

4:23 So thanks Arctic.

4:25 Now like a lot of you,

4:27 I got an e-mail earlier this month that looked something like this.

4:32 And one thing that I learned when I became

4:35 an adult is that there are three certainties in life,

4:39 Death, taxes and open enrolment.

4:41 And if you ask me which one of those I'm the most

4:44 afraid of, I'm not sure I could give you a straight answer.

4:47 Now, this is doubly true this year because three months ago I had a son.

4:52 And so my decision impacts not only my health, but his health as well.

4:57 Now I promise this isn't just an elaborate

4:59 excuse to show off baby pictures at the office.

5:02 Maybe it is.

5:03 But rather, I want to show the gravity that open

5:06 enrollment carries and why it's tough not just mentally,

5:09 but emotionally as well.

5:12 But generative AI is starting to change that.

5:15 Choosing a health plan is an exercise in researching

5:19 and retrieving and synthesising things that an agent excels at.

5:24 And that's why many of our customers chose to create

5:27 agents to guide employees through these tricky and overwhelming decisions.

5:31 And today, I'm going to show you exactly what one of those agents looks like.

5:37 Now, I'll show you the agent in just a moment.

5:39 But first, let me try to remember what plan

5:42 I had in 2024 to give us a starting point.

5:45 They always ask that.

5:46 Now I remember it was a plan with a low deductible,

5:49 with vision coverage and with dental coverage.

5:52 And I remember my wife saying that she thought Doctor Krentast,

5:55 her dentist, would be great with kids.

5:58 So anytime that we test a query in Copilot Studio,

6:02 we first need to define what a good response looks like.

6:06 And we use assertions just like this.

6:09 So we assert that the answer must be grounded in this file.

6:13 It must mention this plan, and it also must give the details of the plan,

6:18 including the deductible,

6:19 vision and dental coverage status and whether Doctor Krentst is in network.

6:23 So let's try it out.

6:30 So let's start in this agent.

6:32 And I've already added this, excuse me, we'll actually start in this agent.

6:37 And I have already added this knowledge

6:40 source and it's a SharePoint knowledge source.

6:44 And I'll just quickly show you what it looks like.

6:46 So you can see there are 18 plans here across years and states.

6:51 And if we go into one of those, we

6:53 can see it's a, it's a very complex document, right?

6:58 It contains long tables with the coverage status of different medical events,

7:03 and it also has a large table of in and out of network providers.

7:10 So part of the reason that I chose the assertions that I showed

7:14 you is because these assertions require

7:16 information that's spread across the document, right?

7:19 So at the top we have the overall deductible.

7:22 To find vision coverage, we have to look in the middle of the document.

7:25 And to find the specific providers, we have to go all the way to the bottom.

7:31 And there's my wife's favorite dentist, Doctor Krentst.

7:33 So the plans are set up so that for 2024,

7:37 based on those assertions that I showed you,

7:40 only Washington Plan 001 is going to meet all of my criteria.

7:45 So why don't we start by asking a question.

7:50 Let's see if we can get the agent to remind

7:54 us what was that health plan that I had before.

7:59 So I'm just going to put this in.

8:06 Perfect.

8:07 So behind the scenes the agent is looking

8:12 through all of these different 2024 plans from last

8:16 year and you can see that it was

8:21 able to successfully retrieve this NWC Washington 2024 plan.

8:27 So that's great.

8:29 Now, one of the most the common questions that I

8:32 hear is what about the ability to compare documents, right.

8:37 Well, California, it's been a pretty good time being in San Francisco.

8:41 My wife's been talking about maybe moving here.

8:43 So let's see what happens when we

8:45 compare this health plan with the California plan.

8:48 So we could say, can you pair

8:52 this with California plans maybe in a table format?

9:08 So beautiful.

9:09 As you can see, the agent successfully compares these full documents.

9:13 And I'll, I'll make this a little wider so the table shows up better and it

9:17 notices the difference that appear from the beginning

9:19 all the way to the end of the document.

9:22 It even gives me this personalized table, right?

9:25 Remembering what were those aspects that I said were important to me?

9:31 So you're probably asking what did we change

9:34 to improve our quality and unlock scenarios like this?

9:42 To answer that, for the first time ever,

9:45 we're going to pull back the curtain and give

9:48 you an inside look into our rag stack.

9:51 And the clicker isn't working,

9:52 so I'm just going to go with the old fashioned way I suppose.

9:58 Or perhaps somebody in the back can advance me to the next slide.

10:02 Beautiful.

10:03 So to give a quick refresher on what a basic rag stack looks like,

10:08 there are three basic steps, right?

10:10 First, the agent takes the user's query and tries

10:13 to understand what is the user trying to do.

10:16 Second, the agent sends an API request to the configured knowledge source

10:19 asking to retrieve the file snippets that are relevant to the user's query.

10:24 And then finally, those return files are fed into the lol prompt,

10:28 which generates an answer.

10:31 We'll see if our clicker is working.

10:32 Beautiful.

10:34 And so the challenge with all of this is

10:37 it relies on a single shot retrieval process.

10:41 And this means that, you know, if you don't retrieve the right files,

10:45 you're ultimately going to be out of luck and you're

10:49 going to be unable to deliver a high value response.

10:53 So we know that LLMS are trained on human data.

10:56 So for retrieval, we looked at how do humans search for information.

11:02 So if you asked me the same exact query that we asked copilot,

11:06 here's what I would do.

11:08 First, I would look for any files that seem like they might be relevant.

11:12 I'd take a quick read of the title and the executive summary.

11:15 I would pick the irrelevant, filter out the irrelevant ones,

11:19 and and only focus on what's relevant.

11:21 If it's a short document, like a couple pages, I might read the whole thing.

11:25 Or if it's a long document and I can't read the whole thing,

11:29 I might control F and look for specific keywords

11:31 like deductible or vision or dental or doctor crentest.

11:35 Now if I encounter big tables with lots of structured data,

11:38 I might do some quick analysis, maybe with a Python or a Ruby script.

11:43 But most importantly, if I reach the end and I don't

11:47 have enough information to give a high confidence answer,

11:52 I return to Step 1 and I start over.

11:55 Now it turns out that we can create the same workflow in the rag stack itself.

12:01 In the agent world, we think of these boxes on the left.

12:05 On the left, that is, as tools that the agent can select.

12:10 So in the Copilot Studio RAG stack, those three basic steps, understanding,

12:14 search and summarisation remain the same.

12:18 However, now, in addition to knowledge,

12:21 the RAG system also has access to a variety

12:24 of different tools that Copilot can use to search for files,

12:29 read entire documents, target key snippets and passages,

12:32 and write code for data analysis,

12:34 which you'll hear more about in just a moment from my colleague Anjali.

12:40 But once again, by far the most important part of all

12:44 of this is the fact is the fact that we have replaced

12:48 the single shot search step with a retrieval loop and it can

12:53 iteratively call knowledge and tools until it can deliver a great response.

12:59 And all of this, all of this means, you know,

13:03 better quality that both on the retrieval side as well As for final answers.

13:11 And so how do we know that this is better quality?

13:14 Well, we know because copilot Studio is obsessed with evaluations.

13:19 Every single day we run thousands of test cases across

13:23 many different domains to understand what does our quality look like.

13:28 And I can tell you that between Build and Ignite,

13:31 we have seen a 20 percentage point gain in our SharePoint quality benchmarks.

13:37 That's huge.

13:38 And a lot of our customers have given us the same

13:41 feedback that that they're seeing those gains in their evaluations as well.

13:47 So when we started Copilot Studio,

13:50 we believed customers wanted AI to do everything

13:53 from configuring and selecting knowledge to delivering great answers.

13:57 But as we've learned more about how probabilistic systems behave,

14:03 we've realised that great answer quality requires that we empower makers

14:09 with the tools to deterministically

14:11 control and evaluate our probabilistic systems.

14:17 So I'd like you to consider one

14:19 of the most common problems that makers face stale documents.

14:24 Keeping organizational docs organized is incredibly challenging,

14:27 especially given all of the decentralized,

14:30 unstructured knowledge challenges that Hardik shared earlier.

14:35 When new documents are uploaded, we can't expect makers to go back and delete

14:39 every stale document that exists in their entire company.

14:43 And that's why today I am thrilled

14:45 to announce metadata filters in Copilot Studio.

14:49 With this feature, we are giving makers deterministic

14:52 control over which documents should be retrieved and when.

14:56 Let me show you how it works.

14:59 So earlier I showed you a scenario where

15:02 we were retrieving documents from last year 2024.

15:06 But what happens when we use a knowledge

15:08 source that contains both 2024 and 2025 documents,

15:11 and those documents are almost identical?

15:14 So the only change between 24 and 25 is now Doctor Krentis has swapped

15:19 from being in network on Plan 1 to being in network on Plan 2.

15:24 So now the correct answer, as you can see through our assertions,

15:27 is Plan 2, even though it has a slightly higher deductible.

15:30 So let's see if Copilot can recommend the correct plan for me.

15:41 So ideally, the LLM should be smart enough

15:44 to pull all of the most relevant documents.

15:48 But until then, let's not leave it to chance, right?

15:52 So let's create a filter here and let's

15:55 only look for documents that have been modified.

15:59 Let's go on Or after today.

16:03 And today's the day that we uploaded these 2025 plans or yesterday.

16:08 So you can see as well that we could use if we wanted to power FX

16:13 and global variables to change these filters

16:16 at runtime based on who the end user is.

16:20 And then there are also a bunch of other attributes that you can use to filter.

16:24 And we are continuing to add more of those every single week.

16:29 So we'll, we'll save that, we'll start a new test session.

16:33 And 1st off, let's see if Doctor Krantis is

16:37 even in network for the 2025 version of my plan.

16:41 So for that, let's just ask if he's there.

16:49 And so now when we run this search in the activity pane,

16:52 we'll be able to see exactly which documents it's reviewing.

16:56 And we can see that before when it was only looking at 2024 documents.

17:00 Now we have a list of search results that all are from 2025.

17:05 So it looks like that metadata filter is working,

17:08 but unfortunately it looks like my wife's

17:11 favorite dentist is no longer in network.

17:15 So this is the correct answer, but maybe we can see if we can find a plan

17:18 that's going to be a little bit better of a fit.

17:22 So let's ask for a recommendation.

17:26 I want to plan kind of with all

17:28 these pieces and one that has Doctor Crunchest in it.

17:32 What is going to be the right plan?

17:38 All right.

17:41 And just like we expected, the agent comes back with this Washington Plan 002,

17:46 where we now know that Doctor Krentis is in network at Happy Smiles.

17:51 And once again, it goes through all the different elements that I

17:54 requested and shows me why this is the right plan for me.

17:58 So metadata filtering isn't the only place that we've

18:01 given makers more deterministic controls over their knowledge.

18:06 I wish I had time to show you everything, but I don't.

18:08 So I'm just going to hit on a couple of highlights.

18:13 So first, one of the most common requests is

18:17 the ability to dynamically configure the knowledge connection at runtime.

18:23 So let's take web for a second and let's say we're adding,

18:29 you know, northwindcare.com/enus.

18:31 Now I could add our English website, but that would be hard coding it.

18:36 What about all of our employees in other

18:39 countries that also want access to personalized results?

18:42 Many of our multinational customers have complex websites

18:45 with localised content and Northwind Care is no exception.

18:49 So now you can add variables right into the web

18:54 URL to customise content for any person or location.

18:59 And, and so the way I would do that would just be by adding this variable

19:03 that I created earlier around language and country

19:05 and then I could configure that knowledge source.

19:08 But we won't do that right now.

19:11 Instead, let's look at these knowledge sources.

19:13 So looking at these two, I realised I might have a duplicate.

19:17 And many times, you know you want

19:19 to use knowledge depending on certain conditions, right?

19:22 And now you can, you can activate

19:25 different knowledge sources depending on these criteria.

19:28 So for example, I could go in here and I could select

19:31 this Power FX formula and if I didn't want to use this knowledge source,

19:35 I could just change it to false.

19:38 Or of course, I could, I could make it more complex,

19:41 but since it looks like it's a duplicate,

19:43 let's just go ahead and deactivate it for now.

19:47 So last, let's look at one of our top customer requests,

19:53 which is knowledge instructions.

19:57 So if we go into this file group with a set

20:00 of planned documents that we were looking at earlier,

20:04 how would the agent know which one of these it should pick from?

20:09 Well.

20:10 What we can do is we can say please choose

20:15 the right file based on and we'll say global user plan ID.

20:23 And I'll, I'll show you in just a second how we, how we created that.

20:27 So now, right, let's just once again, we're in California.

20:30 So why don't we take this one and we'll start with our plan ID.

20:37 And so now it's just asking how it can help.

20:40 So let's just ask a very ambiguous question that it will

20:45 need to to already know kind of what file it's looking for.

20:50 And that's going to be summarize my plan.

20:53 So in case you're not familiar with setting variables at runtime,

20:57 let's just jump into topics.

20:59 And so this agent uses the on conversation start

21:03 system topic and basically just sets this variable the language

21:08 1 to Enus and then allows the user to say

21:12 what their plant ID is and use it that way.

21:17 But this is really just for the the demo.

21:20 You can bring in all sorts of end user context

21:23 information here from identity providers like Antra or Power automate Flows.

21:28 So it's a a really rich experience here.

21:32 And so going back to the the response, we can see that the agent did pick

21:37 the correct plan and and summarised it very well.

21:41 So it picked the correct document even

21:44 though I asked a totally ambiguous question.

21:47 And that's the power of end user context.

21:50 So earlier I gave a sneak peek at code interpreter as a tool for data analysis.

21:56 To dive deeper, I'd like to hand it off to Anjli,

21:59 who drives all of our investments in structured data and complex analysis.

22:04 Please welcome Anjali to the stage.

22:08 Thank you.

22:09 Thanks.

22:11 Hi, folks.

22:13 Who's ready to build another agent?

22:17 All right, I think the PowerPoint is stuck or it's not.

22:31 It was just a little delayed, maybe didn't have its coffee.

22:35 I know I did.

22:36 I have, I'm caffeinated up.

22:37 So the next agent that we are going to build is the policy insights agent.

22:42 And this is going to be targeted to an employee

22:45 at Northwind Care who's responsible for making policies.

22:49 So they are going to need deep insights into what the available plans are,

22:54 what the premiums look like and what tweaks I can make.

22:58 So this agent is going to need access to a lot of data.

23:03 As Hardik had mentioned, datas are data is complex, it's vast and it's messy.

23:09 And what sort of data does this agent need access to?

23:12 It needs access to text data.

23:14 So text can be Word and Notepad files.

23:18 It needs access to structured data.

23:21 Structured data could be tabular data in Excel sheets, in CSV,

23:26 in backing databases like Dataverse, Fabric, SQL,

23:29 and it could also be just images.

23:33 It could be PNGJ, P/E, G files, or it could be image images that are

23:37 embedded in our unstructured and structured data.

23:41 Take an example of PowerPoint.

23:43 PowerPoint has text, it has tabular data and images.

23:46 And our agent needs access to it,

23:48 be able to reason over it to be able to provide accurate responses.

23:53 So you know, Eli mentioned this slide on the new Agentic rag.

23:59 One of the newer pieces here as well is the structured data piece

24:02 and the ability for agents to be able to handle complex analytical queries.

24:08 We do so by generating like a Python script

24:11 at runtime and executing it in its own secure container.

24:17 So what does this enable our agents to do?

24:20 It enables our agents to answer business

24:23 critical queries that require complex data analysis.

24:26 These could be precise look UPS.

24:28 These could be computations and visual generations like charts and graphs.

24:33 How do agents do it?

24:35 They do the by analysing the query the that the user has provided.

24:40 It writes the code, it refines the code and executes

24:44 the code at runtime to be able to answer these queries.

24:48 And the code is is available for the maker to see.

24:52 So that ensures that you know the code is accurate and we

24:56 are able to just generate trust between our makers and users.

25:01 So we're going to take a look at it in action.

25:12 Here is this policy Insights agent.

25:15 And let's take a quick overview.

25:17 I built this using the natural language to agent.

25:20 We have used the default model.

25:22 You you can see that the instructions are really elaborate.

25:25 I didn't have to do any of it.

25:27 The the natural language component took

25:30 care of curating these very detailed instructions.

25:34 I also have knowledge and I have two Excel SX files in my knowledge

25:37 for customers and financials that we'll take a look at in a moment.

25:42 And the most important piece of this agent now is in the settings.

25:45 If we Scroll down to the file

25:48 processing capabilities in the generative AI section here,

25:51 you can see code interpreter that is

25:54 the new capability available to you in preview today

25:57 we have ensured that this capability is

26:00 on and let's take a look at it in action.

26:03 So here I've started a new test session and I'm

26:06 going to ask the agent a fairly simple question.

26:10 So what I've asked is create a bar graph

26:13 with the distribution of members on all the plans in 2025.

26:19 Now what the agent has to do is, as Eli articulated,

26:22 go through the search to see what file has this data and ensure

26:27 that it understands that the right file is being passed on to the code tool.

26:33 The code tool will then generate that Python script to execute.

26:37 Let's take a look at the data that the agent has access to.

26:41 So here we have customers data.

26:43 This is just, you know, synthetically generated data.

26:45 We have customer IDs, first name, last name, phone numbers, addresses.

26:50 We have genders for them, what region they are in, whether they're in gold,

26:55 platinum, silver or bronze plans,

26:57 what the monthly premium is, if they're on auto pay,

27:00 the type of plan that they have signed up

27:03 for, whether it's individual or group sign up date,

27:06 active or cancelled and so on and so forth.

27:09 Similarly in the Northwind Financials Excel sheet,

27:12 as you can see it's a multi tab sheet here,

27:15 we have five years of data broken down by quarter.

27:19 The average premiums, which is you know this with the AVGPMPM nomenclature,

27:24 what the claim costs are,

27:27 operating expenses and we have them by quarter as well as the plan year totals.

27:33 So we have some pretty comprehensive data that the agent has to look at.

27:37 So let's take a look at what the agent comes back with.

27:40 And here in the middle of the screen,

27:42 you can actually look at the code that was generated.

27:46 So here you have all of the import of the various

27:49 files that the agent needed to execute this code,

27:52 what files was passed on and so on and so forth.

27:55 And it's curated a bar graph for me.

27:58 If I open this in a new tab,

28:00 we can see that the bar graph talks about the bronze, gold, platinum and silver.

28:05 And I have a good understanding of what

28:07 the distribution of members across these various plans are.

28:11 So now in the interest of time, I have pre generated a query here.

28:15 And I also asked the agent to not only create that bar graph,

28:19 but to add a line graph in the same

28:22 visual to show the average monthly premium as well.

28:25 Because, cool, I have a distribution of the members,

28:28 but what does it mean in terms of my revenue?

28:32 So the agent has created this bar graph for me that we'll

28:35 take a look at and it's also provided me with some insights.

28:39 And here it's curated this graph where I see that, hey,

28:43 bronze is my start up plan with about $250 monthly premium,

28:47 obviously platinum upwards of six, $700.00.

28:50 That's where like, you know, my, my most lucrative plan, so to speak.

28:55 And I've curated this nice little graph.

28:58 Now remember, this agent had access to the raw data in the Excel files.

29:03 There were no formulas, no macros,

29:06 none of that that I had to build as the policy maker

29:10 here I get all of this data for me curated by the agent.

29:15 The agent also provided some insights in here that we'll take a look at.

29:19 You know, it's said that the visual clearly shows

29:22 the inverse relationship between the planned cost and member enrolment.

29:26 Bronze and silver are the lower cost options.

29:29 Platinum is the least popular but the most expensive 1.

29:32 So we've got some pretty good insights here as well.

29:37 And you know, I'm happy I'm the maker of this agent.

29:40 I'm happy I've tested this up a little bit.

29:43 What I'll do now is I will publish this with one Click to any channel

29:49 so I can here publish it to Teams M365 copilot to SharePoint, my own website.

29:56 And you know, if I wanted to do like more business to consumer kind of an agent,

30:00 I could also publish it to Facebook and WhatsApp all through here.

30:04 So I have published this channel to M365 copilot and here

30:08 I have the policy insights channel and now I'm this policy maker.

30:14 I'm in charge of the mergers and acquisitions for Northwind Care.

30:17 So I'm, you know, considering,

30:20 I'm considering like the acquisition of a file called of a company called Aurora

30:25 Health and I have some questions that I want the agent to analyse about it.

30:31 So I upload the financials for this specific

30:34 Aurora Health and I'm just asking the agent about,

30:37 hey, what are the average premiums for each plan

30:41 in Aurora Health and curator table and provide insights.

30:45 Now remember, the agent has access to Northwind

30:48 care financials but not Aurora health financials.

30:52 But that still doesn't stop me from utilising this agent

30:56 to be able to ask certain queries as well.

31:00 So the Aurora health over here is a similar a similar financials file,

31:05 which it has, you know, the same distribution of numbers.

31:10 But here you can see that the plan names are basic, standard, premium and elite.

31:15 So I'm going to click over here.

31:17 It does seem like the agent did hallucinate.

31:20 I have it backed up right here.

31:23 So what I what I asked the agent was that the about the Aurora health

31:28 and it curated a table for me along

31:31 with the average premiums and provided some insights.

31:35 Not only can I upload a single file,

31:37 I can also ask the agent on how do the average

31:41 premiums across Aurora Health and Northwest Care compare and contrast.

31:47 Now remember this is a trick question for the agent right?

31:50 Because the plan names don't quite compute.

31:52 But the agent tried to like put together

31:55 this plan and set up like the corresponding numbers.

31:59 It also gave me insight here that hey,

32:01 there was no directly matching plan names between the two pre 2 providers.

32:05 So each plan is listed and it curated

32:08 ACSV file for the comparative analysis as well.

32:12 So with that, I wanted to talk about how

32:17 our team ensures that you're getting the highest quality.

32:23 And by that, for that, we just do evals, evals and more evals.

32:28 So we use open source and synthetic data and curate data sets

32:32 with thousands of queries to be able to handle pretty complex questions,

32:36 but that are still based in real

32:39 world by working with customers like yourselves.

32:43 So for example, we already talked about like

32:45 how we could have multiple sheets within a file.

32:49 We also have examples of inconsistent column names, missing column names.

32:55 Like for example, the demo that we just did had

32:58 an AVGPMPM which is that really premium or something else?

33:02 And the agent is able to understand what

33:05 the intent was and what the data suggests.

33:08 We also have merged columns, merged rows, nested tables, etcetera as well.

33:16 That is all part of our data sets

33:17 to be able to ensure that there is high quality.

33:21 And finally, we also have unstructured data that has tabular data.

33:26 So for example, in this in this visual,

33:29 we have the table that's broken across multiple pages even though it's embedded.

33:34 And the agent is able to read through all of this.

33:37 We have, we just saw the functionality for code

33:41 interpreter that's enabled directly at the agent level,

33:45 but code interpreters also available in prompts.

33:48 So if you go into tools and you look at prompts,

33:51 you can enable code interpreter there as well and generate code and even

33:56 pin it to ensure that that deterministic code is always what's run.

34:00 And those prompts can be attached to agents or workflows or can be stand alone.

34:07 So with that, it's my great pleasure to invite Nick Taylor on the stage.

34:13 He is the head of IT Innovations at Gilead and has been

34:16 working very closely with us for the past year and a half.

34:20 And they've done some amazing things that I'm sure you're going to really love.

34:25 So welcome, Nick.

34:26 Thank you.

34:31 Thank you.

34:31 Alrighty, let's get started.

34:33 So I'm Nick from Gilead Sciences.

34:39 Insert joke here.

34:41 Excuse me, I'm reading the prompter.

34:44 So for those less familiar about Gilead, right, we're a leading biotech company.

34:50 We have over 20,000 employees plus contractors, maybe 25,000.

34:55 We're known for advancing life saving therapies.

34:58 We have HIV drugs, oncology drugs, other viral infectious disease drugs.

35:07 So we're not a software company, right?

35:09 We're not a software company.

35:10 We are looking for platforms that let us move

35:12 fast building without having to build things from scratch.

35:16 So what we've built without building from scratch

35:20 is the Gilead Enterprise Navigation and information engine, Genie.

35:27 There are lots of Genie bots out there, but ours is mine, right?

35:31 So the idea for Genie is nothing crazy.

35:33 You've already seen it here before.

35:35 You have this issue.

35:38 I'm sure we have multiple enterprise systems that are authoritative,

35:41 that are the sources of truth, that do different things.

35:44 And I have to go to service now,

35:46 and I have to dig down 15 layers to go do this and then find my PTO over there.

35:51 You know, what's the status of my ticket, those kinds of things, right?

35:54 Well, you know, we've been working.

35:57 Excuse me.

35:59 What we wanted is that single point of entry.

36:02 Again, this is nothing super revolutionary,

36:03 but the way we did it makes me very, very proud.

36:06 We had the single connected intelligent enterprise agent.

36:10 That's what we wanted.

36:12 We needed fast answers, consistent results,

36:14 fewer tickets and the non negotiables

36:17 for something like Gilead in a regulated environment,

36:20 security, privacy, responsible AI.

36:23 So we found that really in copilot studio very, very early.

36:27 I started working with it with just

36:30 me Copilot Studio was announced at Ignite 2023

36:33 and and it's just amazing that we're here

36:36 2025 showing what we built with Copilot Studio.

36:40 But it really goes to show

36:41 the collaboration we've had with Microsoft working really

36:43 close with us on a, on a vision that was that was very clear.

36:48 So there's no code low code platform.

36:51 It's, it's great native in Teams.

36:55 We're putting it in the Microsoft 365 as a, as an agent

36:57 that you can @and there are a lot of possibilities there.

37:00 We needed secure connectors in Genie.

37:02 We have no Phi, we have no PIIIP, those kinds of things.

37:06 It's it's based.

37:08 The idea is any employee, regardless of their role, they do certain things.

37:14 Genie should be able to help them with that.

37:16 Would you use Microsoft Graph to enhance

37:18 the AI search and foundry and governance by design?

37:21 So here's our here's our architecture.

37:25 You know, Genie is the orchestrator,

37:28 orchestrates enterprise sub agents intended to be owned by their domain.

37:32 So we want, we want HR to own

37:34 the data quality of their particular agent Corp OPS, so so on and so forth.

37:41 And we have the Genie team building that orchestrator.

37:45 I have my Microsoft architect over here,

37:48 Fairmont and great job on this thing here we have Microsoft

37:52 Graph or kind of low level what's what's the company holidays,

37:56 those kinds of things.

37:59 But where we need really, really high accuracy.

38:01 We used Azure AI search to index these things, policies,

38:04 a lot of our intranet where we

38:07 couldn't be that deterministic articles, knowledge bases.

38:10 We wanted people to trust the data Sops and we use Active Directory,

38:16 right, data gateway.

38:17 We have lots of on Prem things, AWS things,

38:20 so like our inventory system, tickets and records from SAS.

38:25 And again, this is where people can find it.

38:31 I'm going to show you this because I'm not brave enough to do a demo.

38:34 That's Microsoft's job to take that risk and roll that dice.

38:36 So what I have here are some some screenshots from from Genie

38:40 Genie dev that we're we're going to be pushing the prod on Monday.

38:46 So can you tell me what has chicken on Monday at Bayside,

38:50 which is one of our cafes, right?

38:53 I think we've done some really interesting things here, right?

38:56 You can get very specific and it'll look through the menu,

38:58 it'll find what you have, what has chicken on it, it'll tell you.

39:01 And then it'll say, hey, would you like me to tell you about,

39:04 you know, Tuesday's chicken options or the full weekly menu?

39:07 So asking follow up questions right here and in there, intent recognition.

39:13 We have our feedback and then my legally mandated disclaimer.

39:16 It's not that long, but there it is.

39:19 What are my open tickets?

39:20 This is custom for ServiceNow.

39:24 We have these adaptive cards.

39:25 This thing can open tickets, update tickets,

39:27 find tickets, you know, close tickets.

39:29 But just as importantly, when I say open tickets, it's I have a problem.

39:33 It says, hey, do you want to open this ticket after trying to resolve it?

39:39 And then it'll take the context from the conversation

39:41 and put it into that ticket so you don't

39:43 have to have your user write it again because

39:45 they're already probably irritated with IT in the first place.

39:49 This one's really neat, too.

39:51 This was built by the guy, I call him the Maestro of Mississauga, Canada.

39:56 His name's Don.

39:57 He has asked about his PTO balance up here, right?

40:02 Went to work day, gave him that number.

40:03 Unfortunately, he spent too many days, you know,

40:06 ice fishing or whatever they do up there.

40:08 Doesn't have any more, right?

40:09 So Jeannie has a more empathetic tone, saying you know,

40:12 I'm sorry, you're probably disappointed, but how else can I help you?

40:17 We want to have it, you know, a little bit of a little bit of empathy there.

40:20 These here are more examples of queries,

40:23 but also just showing that we can put it

40:26 inside of M365 as that declarative agent say at Genie,

40:30 you know, what software do I currently have and it'll go to our on Prem

40:35 inventory and then it will and then you can send it away, right?

40:40 And then just ask another agent or copilot like I need training

40:44 for something that I didn't even know I had until I asked.

40:48 So the the one on the left hand side is really nice too.

40:52 It knows where you are, it's localized, personalized.

40:54 So Don is in Canada, it tells him the Canadian holidays, it knows who he is.

41:00 And then it offers to check your PTO balance because you're asking for holidays.

41:04 And to kind of wrap this up is the agentic nature of it.

41:07 So you can ask one query and we can pull across domains,

41:10 across intents or based on the intents and orchestrate that.

41:14 So we have the the query about, excuse me,

41:18 he wants to book a holiday before Christmas.

41:23 So it checks Don's balance, it looks up,

41:25 his manager offers to write him an e-mail,

41:28 and then we have some personalization stuff.

41:31 Yeah, the ice fishing thing, it was like,

41:33 hey, you know, it's perfect time for that.

41:36 So yeah, takeaways, scale and scope.

41:37 We launched it to everybody at Gilly Head, all contractors, everybody.

41:40 We have about 6000 unique visitors or users per month.

41:45 That's about 25% of what we have is primarily IT right now.

41:49 We're moving to other domains, reduce the ticket volume.

41:52 We have about 200 per day that are just they don't tickets aren't

41:57 coming out of them because Jeannie's answering

42:01 it 16% increase in one survey question, which was great over the last year.

42:06 And our user satisfaction employee survey, 16% increase over last year.

42:11 I can find information at Gilead and I think,

42:14 I think we did it part of that velocity we had,

42:17 we're a ragtag bunch of volunteers, right?

42:19 I had this idea.

42:20 I found these people who wanted to work on it.

42:23 We got this thing to pilot in 10 weeks.

42:24 It was, it was nuts.

42:26 Govern governance, distributed ownership, that's the idea.

42:29 Expansion to HR, finance, OPS policy.

42:32 We really have built something that is extensible.

42:34 And with everything they talked about today,

42:36 I think people are just going to get happier and happier

42:39 with the results without us having to do Azure AI search on everything.

42:43 So with that, I'm going to pass it back over to Hardik.

42:46 I do appreciate it.

42:49 Thanks buddy.

42:53 All right, So we saw how we have made improvements in knowledge,

42:56 how Gillard was able to use some of the latest

42:59 advancements and some of the numbers Gillard share were amazing.

43:02 Like something that stand out to me

43:04 was 16% increase in their employee satisfaction.

43:07 They are able to reduce the resolution time

43:10 for tickets and 10 days from inception to production.

43:13 And there by the way, also rolling it out to 24,000 employees coming Monday.

43:17 Here is a portfolio of all the knowledge sources Copilot Studio supports.

43:20 You can see we have M365 line of business data.

43:23 You can upload files, you can connect your Azure indexes.

43:26 You can use the power of Bing.

43:27 If your data lives outside of Microsoft systems,

43:30 you can connect it with three P connectors or MCP connectors, right?

43:34 And a few key takeaways from the station today, guys.

43:38 So we are committed to delivering the best

43:42 knowledge quality for agents to build right.

43:46 And as you saw, we have introduced agentic Rack,

43:49 which is ability for your agents to find information like humans do,

43:53 giving it the power of all the tools based on the data.

43:56 One such tool is code interpreter.

43:58 So now your agents can answer analytical questions.

44:01 And you also saw how we have made it easier for makers to surgically

44:05 configure the knowledge sources so

44:07 that your agents can leverage the most accurate,

44:09 relevant and up to date information.

44:12 If you haven't seen our talk and sessions on evals,

44:15 please do take a look at that.

44:17 They will be recorded.

44:17 You can find them on the on the Ignite website.

44:19 Because guys, there is no way of just throwing all your knowledge source.

44:26 I usually call it a kitchen sink to your agent and expecting

44:29 the best answers you have to evaluate for which we have evals.

44:32 And with that, I would love to bring everybody on stage.

44:36 Thank you all for attending our session.

44:38 That's all from us.