Azure Cosmos DB Conf 2026 | Live Stream

0:03 Hey everybody.

0:04 Welcome to azure Cosmos DB conf.

0:06 Hello.

0:07 Hello.

0:08 Hi, welcome.

0:09 We've got so much great stuff to show you.

0:11 Hi.

0:12 Hi.

0:12 Hello everyone.

0:13 Hey.

0:14 Hello, hello, hello.

0:16 Welcome.

0:16 Welcome to azure Cosmos DB conf 2026.

0:30 All right.

0:30 Hello everybod.

0:32 Welcome to Azure Cosmos DB Conf 2026,

0:34 a virtual event for developers building modern apps with Azure Cosmos DB.

0:39 I'm Patty Chow, product manager for DocumentDB

0:42 and I'm joined by my amazing co host,

0:45 Jay Gordon, Senior Program Manager for Azure Cosmos DB.

0:48 You know what today is all about?

0:50 Practical guidance that you can use right away.

0:53 So we're talking about, architecture patterns,

0:56 best practices and real stories from teams building

1:00 production systems with Azure Cosmos DB, DocumentDB and AI.

1:05 Yeah, that's right.

1:06 You'll see a mix of deep technical sessions,

1:08 quick interviews and demos and this will

1:11 cover everything from getting started to performance,

1:14 scaling and building AI powered experiences on Cosmos DB and DocumentDB.

1:20 And you know what, whether you're joining us live,

1:23 and I, wanna say thank you to our live audience.

1:25 I'm gonna give you all thank you.

1:29 Thank you.

1:29 Whether you're joining us live or watch Demand later, I know I will be.

1:34 We're excited to have you as part of the Azure Cosmos DB community.

1:38 And without our community, we can't have an event like this.

1:43 But before we get started, let's get the big hand emoji.

1:47 We want to say a big thank you to amd.

1:49 Yeah.

1:50 AMD is sponsoring Azure Cosmos DB Conf.

1:53 This year and their partnership has helped us significantly expand the show.

1:57 You know what that means, Patty?

1:58 What does that mean?

1:59 Well, you know what that means?

2:01 More sessions, deeper technical content,

2:03 which I know is important to all of you.

2:06 And, we'll have a broader set of speakers

2:08 and topics that we'll be covering today.

2:10 Yeah.

2:11 And this whole expanded programming really

2:13 wouldn't be possible without their support.

2:15 So thank you, amd.

2:17 Yes, thank you so much.

2:18 We really appreciate it.

2:19 Your support has just really made this into something huge.

2:24 I'm loving just seeing the LED wall.

2:26 I love seeing all these people here.

2:28 Thank you so much, amd.

2:29 Yeah.

2:34 Now what you're seeing live today is just part of that experience,

2:38 you know, that's right.

2:39 You know what, Patty?

2:40 There is a full library of sessions available live today and on demand.

2:44 You'll see them all go live around 1:30pm, Pacific.

2:49 And they're going to be covering everything

2:51 from AI architectures to design patterns, performance tuning.

2:55 We all need to do that.

2:56 And real world cases.

2:58 Yes.

2:59 So let's say that we're presenting here and if we can't catch everything today,

3:03 or you can't catch everything today,

3:05 you can go deeper and explore it all at your own place, at your own pace.

3:09 And you know what else we're doing, Patty?

3:11 And this is a big one.

3:12 We're also running the Azure Cosmos DB Conf.

3:15 Cloud Skills Challenge.

3:17 This is something.

3:18 You know what, Patty, I know you're excited about.

3:20 Yeah, I'm really excited about this, Jay,

3:22 because it is designed to help you build with hands

3:25 on skills using Azure Cosmos DB and it runs through May 8th.

3:29 Sounds put in your calendar.

3:30 Yeah, yeah, absolutely.

3:32 And I just hope that everybody who is watching you take advantage

3:36 of it because the first 500 people to complete the modules will get.

3:40 And here we go.

3:41 Patty, this is my favorite part.

3:42 What's your favorite part?

3:43 It is a, 100% free voucher to take the DP-420 exam.

3:50 So you know what?

3:51 Get certified.

3:51 Get certified for free.

3:54 Get learning.

3:55 Yeah, you know, I love free too.

3:57 And if you want to get a head start, visit AK Mscosmos CB CompChallenge.

4:03 That's right.

4:03 And like I said, our community,

4:05 the huge part of what we do to make sure that you're all involved,

4:09 you're all included and you know, you can take part in this event.

4:13 Yeah.

4:13 I mean, yeah, truly.

4:15 I think one of the things that makes Cosmos DB and Cosmos DB Conf.

4:19 So special is the people behind it.

4:22 And that's what makes it so special.

4:23 Like you all in our audience, thank you for being part.

4:29 And so we want to thank our incredible speakers,

4:32 our customers and partners who are sharing real world experiences today.

4:37 And that includes teams from OpenAI, Vercel, Walmart, AMD,

4:42 of course, the ODP Group, Southworks, Next, and many, many more.

4:48 And you may have gone on one of these sites today.

4:50 You might have bought something.

4:52 They're part of everyday life.

4:54 And these teams that we'll be,

4:56 sharing today that are building production systems at scale.

4:59 Yeah.

5:00 Today they will be sharing what's been working for them.

5:04 Sure.

5:04 And if you want to explore everything today that's happening,

5:07 you can head over to aka Ms.

5:11 Azurecosmos dbconf.

5:13 Yeah.

5:13 Just a quick note.

5:15 The YouTube description has all of today's links in one place.

5:18 There are over 30 sessions total,

5:21 with every session going live before the end of today's show.

5:24 Plus, if you're in the YouTube chat,

5:25 please tell us where you're watching from and ask questions as we go.

5:29 Sorry, I was working at my own pace.

5:30 No, it's okay.

5:31 We want to hear from you.

5:32 We want to hear from you too.

5:34 Well, you know what?

5:34 I think we've set you all up and it's now to get you to our, opening keynote.

5:39 Yeah, that's right.

5:40 Here's Kirill Gavrylyuk, Vice President of Azure Cosmos DB at Microsoft.

5:45 Developers.

5:45 Developers.

5:46 Developers.

5:48 Welcome to Azure Cosmos DB Conference 2026.

5:52 We are really grateful to have you here.

5:54 And let's take a look back at the year since

5:57 the last conference and what a world year has it been.

6:02 Of course we're all building AI apps at amazing pace, right?

6:07 And for Cosmos DB perspective, we invested heavily in semantic search.

6:11 That is core part of any AI app you have now.

6:14 Full text search, hybrid search, vector search,

6:17 semantic data, ranker, plethora of capabilities.

6:20 With confidence we can say if you're using Cosmos DB,

6:22 you do not need a separate search system,

6:24 just use built in capabilities and take advantage of cost efficiency,

6:29 performance, scale reliability and precision.

6:33 And of course more than half of our customers

6:36 are using coding agents today to build apps on Cosmos.

6:40 And for that we offered great agent kits

6:44 with skills to make your coding agent world class expert.

6:48 We added MCP servers and other capabilities to make this possible.

6:53 Now, AI does not remove the need for reliability,

6:57 security, performance and that is our priority number one.

7:01 This is where we invest the most of our energy.

7:04 We've provided throughput management,

7:06 fleet management for customers with large fleets of Cosmos DB.

7:10 We've invested with simplified partitioning with global security indexes,

7:14 hierarchical partition keys.

7:16 We've invested in per partition auto failover.

7:19 And now Cosmos DB is the only database that offers you 5,

7:22 9 reliability for any consistency mode.

7:25 And of course security is number one priority

7:28 and a number of improvements in security space.

7:32 Last year we've added second NoSQL database to Azure DocumentDB.

7:38 Unlike Cosmos, DocumentDB is fully open source.

7:41 It's based on an open source project at Linux foundation where Microsoft,

7:45 Amazon, Google and 12 other vendors are collaborating.

7:49 DocumentDB has a number of advantages over other document databases

7:54 and notably more than 40% cheaper than any other document database,

7:59 including MongoDB Atlas on AWS.

8:01 So please come over to Azure, use DocumentDB,

8:04 take advantage of performance and low cost.

8:08 Now AI transformation is on top of everyone's mind, right?

8:12 And for databases really means, in my opinion three things.

8:16 One is AI increase the importance of flexible data.

8:22 The data AI processes and emits is all semi structured, right?

8:26 It's prompts, it's memory, context, that's what AI is focused on.

8:30 And Cosmos DB provides the first class support for semi structured data.

8:37 It proudly presents flexible data model.

8:41 Now, AI accelerated the pace at which we develop applications.

8:46 Flexible data is paramount for you to enable the space.

8:50 Developers cannot be locked down by strict schemas,

8:53 they need schema flexibility which is again number one capability of Cosmos DB.

8:58 That's why we built a database is

8:59 to provide you schema less flexibility in data modeling.

9:03 Now AI brought semantic search as a first class operator

9:09 into queries and it's true for any database that matters.

9:13 And of course in Cosmos DB we invested far and beyond of that.

9:17 We added semantic search,

9:19 full text search vectors, hybrids, semantic grid anchor.

9:23 We invested in GraphRAG and we're about to announce

9:26 agentic retrieval which takes a graph rack to the next

9:29 level where the graph knowledge graph can be inferred

9:32 by LLM for you without you having to manually construct it.

9:36 Now the third important thing about AI transformation is

9:39 we all now are accompanied by our coding agents.

9:44 That's our favorite friends now, right?

9:46 Coding agents speed up development and for coding agents

9:50 to be efficient world class experts in databases, they need skills.

9:54 And that's why we invested in Cosmos DB Agent Kit.

9:57 We'll talk about it later.

9:59 That provides skills to your coding agent no matter which coding agent you use.

10:04 Along with MCP servers plugins for any favorite coding agents,

10:09 that you may choose.

10:11 Now let me introduce our guest speaker, Guillermo Rauch,

10:18 founder and CEO of Vercel, a popular AI app development cloud.

10:23 And Guillermo Rauch knows a thing or two about what AI apps need from databases.

10:30 In order, to have with us CEO and founder of Vercel, Guillermo Rauch.

10:37 Guillermo, you have built libraries like Socket.IO and Next.js

10:42 that form the modern web as we know it today.

10:45 On top of it, you've built a highly successful company, Vercel,

10:49 that has seen a really good growth of developers, that provides developer cloud.

10:57 What do you credit Vercel?

10:59 Success.

11:01 I think if I look back on the principles that we

11:04 created Vercel on, it was really about the ease of use.

11:08 At the time it was ease of use for humans.

11:10 Now lately I've been calling it agent ergonomics or ease of use for agents.

11:15 It's about, you know,

11:16 you have an idea and you need to bring it online and you actually

11:20 want it to scale and work really well and be a really high quality product.

11:25 So if, if the, if it was, if it could be summarized as something simple,

11:28 it would be the focus of ease of use,

11:31 but in the service of quality, not just, you know, getting slop as we call it,

11:36 this taste online that crashes and pages you at midnight makes sense.

11:41 As you transition to agents as your customers.

11:45 What, surprised you is what's happening in the industry.

11:49 It's really the scale.

11:53 As an entrepreneur, as a founder,

11:55 you start doing market sizing estimates when you're preparing

11:59 your early decks for your investors and would be investors.

12:04 The common wisdom was that even being generous,

12:08 Maybe there were 30 million people that could deploy applications because

12:12 we targeted the easiest programming language that we could think of, JavaScript,

12:17 TypeScript, trying to make the cloud super accessible.

12:20 But it was still tens of millions of people.

12:23 And Next.js also lowered a barrier to entry to react into this universe.

12:29 But when you think about agents,

12:31 they're empowering everyone on the planet to create software.

12:35 They might not even be realizing that they're creating software.

12:38 They might just ask for a solution to a problem.

12:41 And in the process, the agent writes an application and deploys it on our cloud.

12:47 So a lot of the growth that we're seeing these days is literally,

12:50 by the way, happened yesterday.

12:51 A high school friend of mine came to visit me.

12:53 He'd never written a line of code in his life and he's like, oh, by the way,

12:57 I know what Vercel does now because Claude code deployed

13:01 a Vercel and told me the application was now online.

13:05 So I could have never imagined that we would

13:08 see an increase in, let's call it a hundred times,

13:12 the addressable market of creators of software.

13:16 Maybe even higher.

13:18 I think we'll be soon in the future where, software is like water,

13:23 like it's just a normal thing to create, consume, sell, and it's everywhere.

13:30 That's amazing.

13:32 And as the platform that is, so frequently used by the agents,

13:36 what do you expect of the modern database?

13:40 How can we help you?

13:42 Well, I'm very thankful.

13:43 I always tell the team, because part of growing up as a company is

13:46 telling the stories and scars of the early days.

13:50 We started out using a database that was very managed by me in the early team.

13:56 And it was a horrifying experience.

13:58 And I reached out to you and I'm very thankful,

14:02 Carol, through an introduction from Nat Friedman,

14:04 because I was looking for a solution that would help

14:07 me focus on growing Vercel and not being a dba.

14:11 We had very few people on staff.

14:13 This is literally the garage story and one of the convictions that I had

14:18 that helped me choose Cosmos and be savvy about this because I'll tell you,

14:23 some of the less familiar developers look at Query languages.

14:27 And they might have a little bit of culture shock sometimes,

14:30 but one of the things that convinced me is that compute was becoming serverless.

14:35 The value of Vercel was you deploy it scales to zero,

14:40 it scales to billions of people.

14:42 I literally am supporting applications these days that were vibe

14:45 coded and go viral all over the Internet and have traffic

14:49 that even teams of engineers would have never dreamed of and can

14:53 now be sustained by someone that barely knows how to prompt.

14:57 And the magic there is that it was that bet on serverless.

15:00 Because I also tell you the reality is

15:02 that a lot of this software is quite ephemeral,

15:05 that's being written by the agents.

15:07 Maybe it's useful for a day, maybe it's useful for a presentation.

15:10 We're seeing a lot of sales engineers, including people at Microsoft,

15:13 use v0 to vibe code an app for a sales pitch,

15:18 for a demo for pre and post sales to demonstrate an integration of a product.

15:23 And so if software is ephemeral, serverless,

15:26 scale to zero and flexibility there is extremely useful.

15:29 So I'm very thankful that we made the right bet.

15:32 The other thing that I wanted to be super mindful

15:35 of that serverless compute helps a lot with is fault isolation.

15:41 So I didn't want to get a machine hot that could create noisy neighbor problems.

15:46 Imagine if the misbehavior of one tenant could impact the reliability,

15:51 uptime, availability of other of other customers.

15:54 But what I just described is the everyday database.

15:57 It's kind of nuts that we still.

15:59 There's so many people that kind of live in the shadows,

16:01 I think because they put systems online whose behavior they cannot predict,

16:07 they might become a wave of queries or users or whatnot that just ruins

16:12 the day for everybody and people start citing slow query logs and all that junk.

16:16 And so I wanted a system that gave me an economical

16:22 thinking where the developer writes a query and they understand its cost.

16:26 We can project out its cost.

16:29 A lot of what I think has made Cosmos successful is

16:31 that it kind of makes sense in the token world, right?

16:35 Like when I use a coding agent,

16:38 I know that certain traces of reasoning cost more compute.

16:42 When I make a cross partition query in Cosmos,

16:44 I get back an RU count of like how much I spent on that query.

16:48 And so it allowed me to scale Vercel from a few

16:52 developers that didn't want to have a DBA on staff

16:55 and to actually hundreds of engineers that are supporting

17:00 a growth in usage that would have never imagined with deployments,

17:06 which is one of the key collections that we Host on Cosmos Having grown,

17:10 our weekly Deployment count is 3x'd in a few months.

17:14 And I think there's probably no database system

17:17 in the world that I could have homegrown,

17:20 for example, that could have absorbed that kind

17:23 of growth that agents have gotten us.

17:27 Thank you.

17:27 Well, we're very grateful for you to be our customer.

17:31 We have many developers online right now watching this that are early in career.

17:37 What would be your advice to them?

17:41 Well, one of the things is that the agent is the new computer and this is

17:46 advice that I give myself every day is that I really have to update my priors.

17:51 Every day I give a presentation to the whole company saying look,

17:56 these are the things that I used to believe AI couldn't do.

17:59 And I was wrong.

18:00 I'm coming out and saying it like I

18:03 thought AI couldn't contribute to large code bases.

18:07 I was wrong.

18:08 I thought AI couldn't have, couldn't do good designs.

18:11 I was wrong.

18:12 And so it's very important to realize that the things

18:17 that you've learned you have to hold on to.

18:20 Of course you have to appreciate the skills that you've developed and be proud.

18:24 But also just be very light on your feet is my advice.

18:29 I like to think of the engineer of the future.

18:33 I'm someone that is very full stack in their understanding of the business.

18:38 When I hire an engineer I don't expect them

18:40 to just be typing code or even prompts for that matter.

18:43 I wanted to understand the full problem.

18:47 Again it's a little bit like I mentioned that economy system that Cosmos

18:51 has given us where I was able to shift the burden of understanding,

18:55 you know, what our database performance and cost

18:58 is going to be to the everyday developer.

19:01 They didn't toss the problem out to.

19:03 Oh, some other team will worry about scaling my database, right?

19:06 No, you are participating in that process.

19:08 I think the original ethos of DevOps was very much in that direction.

19:12 I think the difference now is that everyone is super full stack.

19:16 I do design, I generate videos, I generate SVGs,

19:22 I create front end code bases, I create backend code bases, I write clis.

19:27 During the holiday break I vibe coded my own Swift menu bar app.

19:32 And so I think the antidote here or like the way to really

19:38 stay relevant is to be super open minded and super full stack.

19:45 Thank you Guillermo.

19:46 Really grateful for us to have you here.

19:49 Appreciate everything what you're doing, love your posts,

19:53 love everything that vercel companies likewise.

19:55 It's been a great Journey and excited to continue collaborating

19:58 with you all and do more with Vercel in the future.

20:12 Thank you Guillermo.

20:13 Now Guillermo just covered few of the reasons why they chose Azure Cosmos DB.

20:17 But as these are the same reasons why thousands of other

20:20 companies choose Cosmos as a database for their agentic apps.

20:24 Its schema free flexibility enables your team to run fast iterate fast.

20:29 It's search built in into your query engine so that you

20:33 don't have to deploy different systems for search for database et cetera.

20:38 It's reliability, low latency and serverless

20:42 elasticity to enable your software to spike,

20:45 to enable your app to grow with demand without you touching anything.

20:50 Of course it's 5, 9 reliability that makes your sleep sound

20:56 while your software runs and serves millions and billions of people.

21:01 Now when it comes to AI use cases we have many

21:05 but the common ones with Cosmos DB of course include vector database.

21:09 Right.

21:09 That's why we invested in vectors, multimodal vectors,

21:12 the speed and cost effectiveness of vector search, semantic retrieval,

21:18 billing on top of vectors offers you

21:20 ability to do full text search, hybrid search,

21:23 precise increased precision of your retrieval,

21:26 taking advantage of the knowledge graph of your system.

21:31 But a recent trend is new use case that tripled

21:35 in size over the last six months is agentic memories.

21:39 We have more and more systems choosing Cosmos DB as agentic

21:43 memory for their apps and there is a reason for it.

21:46 Memory is semi structured, memory is dynamic,

21:50 memory needs to be reconciled over time and all

21:54 of these capabilities are built into Cosmos so you

21:57 can focus on what you app needs to do

21:59 and less about the infrastructure that it runs on.

22:05 Now let's get a couple words about the search in Cosmos DB.

22:10 The vector search is based on DiskANN algorithm built by Microsoft Research.

22:15 And the two key things about it is one, it's extremely cost efficient compared

22:19 to the traditional separate standalone search systems.

22:23 It's up to 50 times cheaper as you can see on the charts.

22:28 It is also low latency and scales no matter the scale of your app.

22:32 It provides you the same millisecond latency

22:35 for your vector search and full text search queries.

22:39 It scales well from few vectors to billions of vectors in your data set.

22:45 Now when you think about an AI app built on Cosmos,

22:48 of course OpenAI ChatGPT comes to mind.

22:51 That is well known.

22:53 It's a tremendous one of a kind application.

22:56 It executes more than 1.4 trillion transactions daily against Cosmos DB.

23:02 It stores more than 45 petabytes in Cosmos by now and at this scale,

23:07 it's the fastest growing app on the planet.

23:08 It still grows tenfold every year.

23:12 And to dive deeper into what enables OpenAI

23:15 to scale that fast and maintains a trap, it's a pace of innovation.

23:23 Let me invite John Lee, staff engineer from OpenAI,

23:26 to share a few thoughts with us.

23:30 Well, hi John, it's really great to have you here.

23:33 Would you mind introducing yourself?

23:36 Yeah, of course, it's good to be here.

23:38 My name's John, I'm on the online storage team at OpenAI.

23:42 Awesome.

23:43 Well, OpenAI is a unique company in many ways.

23:47 Right.

23:47 But one of the things that everyone knows about it is

23:50 that, and it's really visible is that it moves really fast.

23:54 You guys are doing many things concurrently and very fast.

23:59 What does OpenAI look for from databases

24:02 to enable users fast pace of innovation?

24:06 Yeah, that's a great question.

24:08 I think there are lots of standard database things that we care about, at scale.

24:13 So predictability, being able to scale to petabyte sizes,

24:16 being very flexible with how our use cases can be built,

24:21 and then of course observability is very important.

24:24 But I think from the scale perspective and being able to move really fast,

24:29 the most important thing here is being able

24:34 to scale from zero to millions of QPS,

24:37 being able to scale from zero bytes to petabytes,

24:39 features are launched on the regular, and they go immediately from no usage

24:45 to being used by hundreds of millions of people, on a daily basis kind of thing.

24:49 Right.

24:49 You know, so the way that you might design a normal database,

24:54 and then scale it out as your use case grows,

24:56 you know, that just doesn't work at this kind of velocity.

25:01 And I think we have thousands of developers that are actively building products.

25:05 So with Codex now things are iterating even faster.

25:09 It's really important to make it easy to onboard to databases really fast.

25:13 So we have thousands of tables, that we have to back by Cosmos DB.

25:20 And we have a system in front of Cosmos that helps

25:24 you be able to do sort of schema less design,

25:28 and allow these product developers to onboard very quickly.

25:33 Amazing.

25:35 Could you share any of the interesting patterns

25:37 that you guys have done on top of Cosmos DB?

25:40 Yeah, I think predictability is really key for how we use Cosmos DB.

25:46 We have a very well defined API that we have and expose

25:52 and all of these clients they build on top of this.

25:56 And that allows us to keep the system stable.

26:01 We don't want Every client to be writing their own Cosmos DB queries,

26:04 creating their own Cosmos DB accounts, creating their own tables.

26:08 So we try to build a multi tenant system on top of Cosmos, interestingly,

26:12 sort of at this global scale where we have

26:15 a lot of Cosmos accounts in a lot of regions, region failure is a big problem.

26:20 Right?

26:21 It doesn't happen often, but with dozens of regions it will happen.

26:26 And so we definitely use Cosmos DB multi region write.

26:31 We replicate our accounts, so that we can always shift reads

26:34 and writes to other regions in case of outages.

26:38 But generally also for good performance,

26:40 you need to be able to allocate your data

26:43 in Cosmos DB accounts that are close by.

26:46 One particularly interesting feature that we use for Cosmos db,

26:49 is that we have tiers of Cosmos DB accounts where some

26:53 of these accounts that are being used to store globally accessed metadata.

26:57 So like things like account, information and user settings,

27:01 we actually replicate that to dozens of regions,

27:05 as opposed to just a single region.

27:09 That's fascinating.

27:11 You've done a lot of innovations on top of Cosmos DB.

27:13 What's your favorite?

27:18 I would say that, being able to move data across Cosmos DB.

27:24 So we have a lot of products that will start,

27:30 on a single Cosmos DB account, and in a single container.

27:35 But as they grow or as their needs change,

27:38 iteration is very important at OpenAI.

27:41 As needs develop, maybe we realize

27:44 that they have much higher performance, constraints.

27:48 And so being able to transparently migrate their data from one Cosmos

27:52 DB account to a different Cosmos DB account that is globally replicated,

27:55 that is something that we've spent a lot of time trying to build around.

28:00 And the Cosmos DB team has given us a lot

28:03 of the building blocks that we need to be able to do that.

28:06 That's awesome.

28:07 We should at some point join the publishers.

28:10 A lot of viewers would love to see us as well.

28:14 Well, we have many app developers that are

28:17 early in career or potentially are, still in college.

28:21 What would be your advice to them?

28:24 Yeah, you know, I think this is evident also

28:26 in the way that our system has been built, to accelerate product teams.

28:31 There's a big culture at OpenAI about shipping things,

28:35 and shipping things quickly.

28:37 I think it's important to have like a vision of what you're working towards,

28:40 but don't over build solutions to problems.

28:43 Try and get something working into the hands of your users,

28:46 and then you can evolve and iterate on it.

28:49 Almost Never.

28:51 You will get the first iteration correct, but you'll learn from that experience.

28:57 Thank you so much John.

28:58 And thank you so much for being our customer

29:00 and thank you for joining us at this conference.

29:05 Thanks for the invite me.

29:10 Thank you, John.

29:10 Now what you noticed John mentioned Codex use in OpenAI,

29:15 everyone uses coding agents, right?

29:17 As I said, based on telemetry,

29:19 more than half of our customers are now using coding agents.

29:24 And for you to get started, let's say you don't know about Cosmos DB yet, right?

29:31 The best thing is to install our Agent Kit.

29:34 This is a set of the skills that gets you agent

29:37 up and running and becoming the world class expert in Cosmos DB.

29:40 It has 80 plus skills in growing encompassing best

29:44 practices with Cosmos DB across the entire app lifecycle.

29:48 From data modeling to partitioning to queries,

29:51 indexing, throughput management, high availability and monitoring.

29:56 Now to see it in action, let me invite our fearless product leader,

30:00 Andrew Liu to show Agent Kit in a demo.

30:05 Hi Kirill.

30:08 So to set the stage, I am building an app here.

30:12 It's a travel planner.

30:13 It's a multi agent Cosmos DB application and it's going to help me plan a trip.

30:19 So here I'm planning a five day family trip

30:22 to LA that includes my 30 year old daughter.

30:24 Now what's important about this is that it's not just helping me plan the trip.

30:28 It remembers where I left off.

30:32 This is done through by storing memory in Cosmos DB.

30:35 This includes short term memory, long term memory, and vector.

30:38 Now here's the thing.

30:39 It's not about the application, but rather, I'm new to building AI apps.

30:43 And the thing I constantly question myself is did I build it right?

30:47 Because at scale, getting it not right gets expensive.

30:52 What if I pick the wrong partition key?

30:54 What if I have the wrong data model, the wrong index?

30:57 I mean that's burning RUs in production.

30:59 That's the kind of thing that you don't want to surprise you when you launch.

31:03 And scale.

31:05 So let's jump over into what this looks like in vs, code.

31:10 First, thing I want to show you is

31:11 actually our new VS code extension or relatively new.

31:14 Many of you guys have seen the portal.

31:17 We get it.

31:18 We've also gotten a lot of the feedback on.

31:19 We really want to go and improve the desktop tooling.

31:21 So we've been putting a lot of effort into the VS code extension.

31:25 What you'll find here is you'll get a lot of the same operations but you

31:28 can do it now in line with your code without having to context switch.

31:32 So here I have my Cosmos DB account, and it works just the same way.

31:36 I can go and query the database right behind this application and what

31:41 you'll see is I'm storing a bunch of data in Cosmos DB.

31:44 I have my short term memory in the form of messages.

31:46 I have my long term memory, in the form in this memories container where I

31:50 have things like my declarative memory which is durable facts,

31:55 procedural memory where I have behavioral preferences and episodic memory,

32:00 which is going to be things that are a bit more trip specific.

32:03 Now let's go into the magic on Agent Kit.

32:08 Did I even get this right?

32:10 So installing this is actually very easy.

32:13 Just1 CLI command away.

32:15 You can use npx skills add and go and add the Cosmos DB Agent Kit.

32:20 It's going to go through a few different questions,

32:23 like setting it up for the project scope or doing it globally on your computer.

32:31 But when it sets it up, what it's going to do is it's going to go

32:34 and create this on the project scope a agents container,

32:38 or rather directory and it'll load all of my skills in here.

32:44 Now that I have my agent skills set up,

32:47 let's go and go check out the quick application.

32:54 So I am going to go take a look here and I'm going

32:56 to have it just go review my Cosmos DB data model for performance issues.

33:02 Now this isn't just a generic LLM looking at the code.

33:05 The Agent Skills is a package set of specialized expertise modules.

33:09 Right?

33:09 We're taking a lot of context about Cosmos

33:12 DB and bringing that into modules, just for this.

33:17 Now there's no services, no accounts.

33:19 It's a repo of skills so the agent gets smarter about Cosmos.

33:22 It's teaching the agent how Cosmos DB actually works.

33:25 It'll include data modeling,

33:27 best practices, partitioning, indexing, RU economics,

33:30 and make sure that that actually matches up with my data access patterns.

33:34 These are things that it took us decades

33:36 to learn over building real world Cosmos DB applications.

33:41 Now you can get that directly in the form of agent skills.

33:47 Now this is what's really cool about this.

33:50 It's not just doing some generic pattern

33:52 matching and giving me generic NoSQL advice.

33:55 It's reading my actual code.

33:57 It's taking a look at My code, my infrastructure as code, Bicep templates,

34:01 correlating my partition key, my index policy,

34:05 my document shapes, to the different data access patterns.

34:09 It knows what good looks like.

34:12 Specifically for Cosmos DB,

34:14 this is expertise that used to live in one senior architect's head on my team.

34:20 But it was hard to get a hold of those individuals.

34:23 I had to book them a week out.

34:25 Now it's just in my editor here on demand.

34:30 So let's take a look at the summary here.

34:32 It was able to go and find an order of priority and highest roi.

34:38 Oh shoot, I missed some composite indexes for my order by query

34:42 that can shave off a bunch of RUs for each of my different queries.

34:46 And when I'm running at scale with high QPS, that's really going to add up.

34:50 It's also going to go and prioritize additional things like hey,

34:54 I should go and fix my memory's partition key once again.

34:58 Hey, if I get the wrong partition key,

35:01 as this application scales to many partitions,

35:03 if I have a lot of cross partition queries that's going to go add up.

35:07 So what I'm really happy about this is

35:09 it's catching this before I hit live production.

35:16 So there's two key takeaways I'd like

35:19 to for everyone here in the audience number one, if you're a new developer,

35:24 Cosmos DB doesn't have to feel foreign or scary like

35:29 all the cognitive load of partitioning data modeling right here.

35:33 I can go and interact here with my Cosmos DB Agent Kit and go and get a live

35:39 code review of my system and make sure

35:41 that what I'm doing is well architected by default.

35:45 Also if you're an existing Cosmos DB developer,

35:47 well this also lets you go and scan your code,

35:51 scan your past projects and go look for opportunities for optimizing

35:55 the performance as well as cost characteristics of that application.

35:59 Now back to you Kirill.

36:07 Wow, thank you Andrew.

36:08 I think the last point resonated with me so much.

36:11 This is not just allows you to increase your productivity and build apps faster.

36:16 It's actually world class expert in Cosmos DB as your code reviewer.

36:21 You can plug it in into your CI CD processes in your code,

36:25 review flows and it's right there catching any bugs that you

36:29 may introduce or helping you tune your code to be more efficient.

36:34 Now we have great coding agents skilled on Cosmos DB.

36:38 We have fantastic AI apps,

36:41 but with all of that we need systems to continue to be reliable,

36:45 secure and performant.

36:46 And Cosmos DB is built for that.

36:49 That is the number one priority for Cosmos DB as a database.

36:53 Cosmos DB is a database that can take your applications anytime,

36:56 from gigabytes to petabytes,

36:58 from hundred transactions per day to trillions of transactions per day.

37:02 You don't need to worry about it if you hit on overnight success at some point.

37:08 Cosmos DB is the only database in the cloud

37:10 that offers you guarantees not only five nines down,

37:14 uptime across any consistency modes, including the strong consistency.

37:20 It also provides you money back guarantees for zero data loss.

37:23 Thanks to the consistency SLA's financially backed.

37:29 Finally, it's the only database that gives you full autonomous resiliency,

37:33 zero touch for your system.

37:36 Now this scale and reliability is made

37:38 available to you not only just by software, but also by an awesome hardware.

37:43 And to discuss how we run the fleet of Cosmos DB,

37:47 I would like to invite Steve Berg,

37:50 Corporate Vice President and General Manager of the Service

37:53 CPU Cloud Business Group at AMD on stage.

37:56 Welcome Steve.

37:57 Thanks for having me.

37:59 Glad to be here.

38:00 Great to have you.

38:03 Now, Steve, from your perspective,

38:04 how does AMD and Microsoft partnership impact how customers experience Azure?

38:09 That's a great question.

38:10 Before I answer the partnership piece, let me give you a little bit

38:13 of history about our working together with Microsoft.

38:16 Back in 2017 we launched our first EPYC processor, Naples,

38:20 and Microsoft was actually the first cloud provider

38:23 to offer it to make it available in the cloud.

38:26 That created a really unique bond between us and Microsoft.

38:30 Over the past decade,

38:31 we've worked together to launch five more generations of of EPYC instances

38:35 that are available in about 60 different compute families across the globe,

38:39 with our most recent one being Turing, which is an excellent product.

38:43 So we're constantly working to optimize our hardware

38:46 along with your software to be more performant,

38:49 more efficient and provide new capabilities with each generation.

38:53 We do this with a deep engineer

38:55 to engineer collaboration on making roadmap changes

38:57 that improve our products and help them

39:00 get tuned better for Microsoft Azure needs.

39:03 So some of these examples include things like confidential computing,

39:06 3D V cache and high performance bandwidth that's used on some of your VMs.

39:13 This partnership has led to more Azure first parties adopting our hardware.

39:20 And that is what the partnership provides is Cosmos DB.

39:24 And users on Cosmos DB get all the benefits of decades of work

39:28 that we do together and all the hard work that happens within our collaboration.

39:32 I could agree more.

39:34 We actually use AMD quite heavily in our Cosmos DB fleet and that's what

39:38 enables us to scale with demands like

39:40 OpenAI and other super fast growing applications.

39:43 Yeah, exactly.

39:44 OpenAI and all those just get all the benefits

39:46 of us working together with everything under the hood.

39:49 So it's really, that's what the partnership really brings.

39:53 That's awesome.

39:53 Now, in today's world with global situation, energy efficiency is top of mind,

39:59 how should builders think about balancing

40:01 the performance needs and sustainability and energy efficiency?

40:05 Yeah, we absolutely focus on that.

40:07 Our product teams and our engineers are

40:09 always looking at what's energy efficiency and sustainability.

40:13 And fortunately, builders who choose to run Azure get all of this benefit

40:17 without even really having to think much about it at all.

40:20 Because we do all of the for them.

40:22 We work to deliver generation after generation EPYC CPUs with more performance,

40:27 better efficiency per watt and improved TCO,

40:30 which directly translates into users and builders being able to get

40:35 benefits of cost and maximizing their performance per watt per dollar.

40:40 That's the critical metric.

40:42 So when you consider the global reach

40:43 of azure with over 70 regions and counting,

40:46 it's absolutely imperative that we maximize performance

40:49 for those users in a sustainable way.

40:51 So focusing on that performance per dollar is where we concentrate on.

40:57 And when Cosmos DB team rolls out new improvements to your SaaS offerings,

41:01 builders implicitly get all those benefits

41:04 just by using the modern infrastructure.

41:06 And when end users roll their own infrastructure as a service,

41:11 if they gravitate towards amd, epyc, vms, they also receive those benefits.

41:16 That's amazing.

41:17 That's exactly what we all need to do, right?

41:19 We don't want to think about problems that we can't solve.

41:22 We want you to solve them for us.

41:23 We want to make it easy for you.

41:24 Yes.

41:25 And semiconductors has always fascinated me

41:27 as a really tough industry because you

41:29 have to plan ahead five years ahead for what CPUs customers might need.

41:35 How do you do that?

41:36 Yeah, it's interesting you asked that because I don't think anyone could

41:41 have predicted what was happening with AI in the past year or so.

41:45 And it's really fascinating to see what you're doing with agentic AI today.

41:49 I think the rapid pace of AI took everyone many by surprise and no

41:54 one would have guessed the importance that CPU provides the role in this market.

41:58 Right.

41:58 CPUs are becoming what I like to refer to as the new bacon.

42:02 The inference, workloads, orchestration, pre and post processing, agentic AI,

42:07 general purpose, all continue to use CPUs in a very greater detail.

42:14 And so cloud infrastructure will continue to move from general

42:18 purpose Commodity type machines

42:19 to increasingly customize differentiated platforms.

42:23 Now we're going to focus on what's important,

42:25 that performance per dollar per watt that is so critical in our infrastructure.

42:30 So we already do a lot of this custom work today.

42:32 I provided some of the with Microsoft,

42:34 I presented some of those examples earlier and we'll

42:37 continue to expand in this space in the coming years.

42:40 So if I leave you with a prediction

42:42 of what's going to happen in the five year span,

42:44 I can tell you that we're going to launch

42:47 more and more VMs with Azure and Cosmos DB

42:51 and user developers in the planet will be able

42:55 to get benefits from AMD and our collaboration together.

42:59 That's exciting.

43:00 So Cosmos DB customers can look forward to more transactions per

43:03 dollar thanks to the work that we at a better cost efficiency.

43:08 Absolutely awesome.

43:09 Well, thank you so much Steve.

43:10 Thank you for our partnerships.

43:12 Thank you for sponsoring this conference

43:14 and made this wonderful sessions available.

43:16 Appreciate you having me and thank you all for the time.

43:20 Thank you.

43:27 That was amazing.

43:29 Now Cosmos DB is not the only NoSQL database we have on Azure.

43:34 Last year we launched our second NoSQL database, Azure DocumentDB.

43:38 And unlike Cosmos, DocumentDB started from an open source project.

43:42 This is a DocumentDB project at Linux foundation that Microsoft,

43:46 Amazon, Google, Yugabyte and 12 other partners are collaborating on.

43:53 Both Amazon and us are offering fully managed services.

43:57 Amazon DocumentDB and Azure DocumentDB on top of this open source project,

44:01 this is a fully managed document database that has

44:06 a number of advantages over other document databases.

44:09 It's faster, it has faster AI search for example.

44:12 You can deploy it anywhere in any cloud or on premises.

44:17 And it's more than 40% cheaper than any other document database on the cloud,

44:23 including AWS MongoDB Atlas, for example.

44:27 So let's take a look at an example of this.

44:30 Let's say I want to send 1 million

44:32 writes per second to my database on Azure DocumentDB.

44:37 A single instance, single shard 80 allows me to do that no sweat.

44:45 An equivalent scenario,

44:47 equivalent setup on AWS with MongoDB Atlas with the same SKU M80

44:53 would only give me 800k transactions per second according to AWS documentation.

44:59 We have the codes if you want to run it and see for yourself what it does.

45:02 But according to docs, it's only 800k and it cost me 60% more.

45:09 So all in all, I can get twice as many

45:14 transactions per dollar with Azure DocumentDB for every SKU.

45:19 It's at least 40% cheaper on skewer SKU basis overall,

45:23 as you scale your application load more data,

45:26 you get more than 85 reduction in TCO for your application.

45:31 So DocumentDB is really cost effective.

45:35 Take it for a spin and get the cost efficiency.

45:39 Now when should I use which?

45:43 If you are having a lot of simple queries,

45:46 if you want to have a mission critical application,

45:51 if you want to scale limitlessly, if you like serverless,

45:55 form factor, you like global distribution, Cosmos DB is for you.

45:59 This is the only database in the cloud that can do all that.

46:02 But if you prefer open source, if you like MongoDB API compatibility,

46:06 if you want to potentially port your application to other

46:10 clouds or on prem and you like the core architecture,

46:14 then DocumentDB is a good choice for you.

46:19 Multi cloud, open Source, Complex Queries,

46:24 DocumentDB, Lots of Scale, Serverless,

46:27 5, 9 Reliability, Cosmos DB now what if I want both?

46:32 What If I want 5, 9 resilience and I want cross cloud portability?

46:39 As of today you can do it both.

46:42 We created an open source project, we launched it today called MultiCloudDB.

46:48 It's fully open source,

46:49 MIT licensed and it provides you cross cloud portability across Azure AWS

46:56 and GCP by providing one data access layer on top of Cosmos DB AWS,

47:03 DynamoDB and Google Spanner, we provide you cross cloud disaster recovery.

47:14 We will provide you cross cloud replication with sync

47:16 as part of this project you don't have to choose.

47:20 You can have five ninth availability and cross cloud portability

47:24 at the same time using this project and with Cosmos,

47:28 Dynamo and Spanner from each of the hyperscalers.

47:34 Now these are just some of the topics we will cover in the next few hours.

47:38 And while we have your attention, join us for Skill Challenge to win free Cosmos

47:44 DB certification and stay tuned for more exciting sessions today.

47:50 Install Cosmos DB Agent Kit, look for more sessions,

47:54 build an app and maybe you can talk

47:56 about it at Our next Cosmos DB conference 2027.

48:00 Enjoy the rest of the event.

48:06 Thank you so much Kirill.

48:08 Hey, wasn't that a great keynote?

48:12 Thank you so much Kiril.

48:15 Thank you Andrew.

48:16 Yeah, thank you so much.

48:17 There is so much to take in, from AI to real world applications.

48:23 Even with the interview with AMD that was so interesting.

48:27 There's so much that you can do with Azure Cosmos DB.

48:30 Yeah, and you know what, the demo really brought it to life.

48:33 I love AgentKit.

48:34 I want to say hi to Theo and Saji.

48:37 They're doing such amazing work.

48:39 Yeah.

48:39 And if you're enjoying the event so far, please let us know in the YouTube chat.

48:43 We love to hear what you think.

48:46 Up next, we've got a quick story from one of our partners.

48:49 Let's take a look.

48:53 Hi, my name is Mick Feller,

48:54 I'm a distinguished software engineer for the ODP Group.

48:57 But you might better know us as Office Depot.

49:05 We are currently using Cosmos DB mainly, in our homegrown AI,

49:10 application that we built for our enterprise called the Personal Assistant.

49:13 We use it for time series to do analytics,

49:16 but more importantly we use it for, for HR profiles.

49:19 So we store each employee's profile so we can give more context

49:23 to the AI models to be able to answer questions better in contextual form.

49:32 The main benefits we've gained from using Cosmos DB is the hands off management.

49:38 It scales automatically.

49:39 We have full control over how much we use, how much we don't use.

49:43 And it's been just amazing around the whole management of the actual

49:46 tool because the scalability we can control cost to fine grain level.

49:51 And that's why we chose Cosmos DB.

49:58 We've seen almost 100% year over year increase

50:01 in usage and big driver of that was Cosmos DB.

50:05 Because it scaled so flawlessly with our needs,

50:08 we were able to seamlessly scale to that level of usage for the customers.

50:13 That's our success story with Azure Cosmos DB.

50:16 And I can't wait to hear about yours.

50:21 Yeah, welcome back and if you're just joining us,

50:25 yes, you missed a fantastic keynote.

50:29 I can't believe they missed it.

50:30 I know, but don't worry, we've got you.

50:33 Everything from today will be on demand,

50:35 including exclusive sessions so you can catch up at any time.

50:40 So grab the full playlist here at aka Ms.

50:44 Cosmosconf26 playlist.

50:47 Let's give some shout outs.

50:51 Do you want to just say to anybody who's watching,

50:53 I saw some great people from Sweden.

50:56 Yeah, I saw people from the uk, Mexico, all around.

51:01 So tell us where you're watching from.

51:03 We want to give you shout outs if you got

51:06 some interesting stories also about what you're doing with Cosmos DB,

51:09 we may just throw here.

51:10 Oh, we've got a few of our.

51:12 So, well, we've got Mike, from Norwich, thank you so much.

51:18 And then one of our great influencers, Ravi Jain.

51:21 Thank you so much for all the support.

51:23 Yeah, no, thank you so much for being

51:25 here and please keep it coming in the chat.

51:28 So if you can tell us what you're building with Cosmos DB,

51:31 we'd love to shout you out.

51:34 We promise we Won't ask for your partition key unless you bring it up, which.

51:39 Okay.

51:39 Hey, that's why we have that Agent Kit, right?

51:41 Yeah, exactly.

51:42 Andrew showed us.

51:43 If you want to keep connected, aside from the YouTube chat,

51:47 after the event, you can join us on the NoSQL channel on Discord.

51:52 You can go to aka Ms.

51:55 Discord Cosmos Conf.

51:57 Yeah.

51:57 And also the Cloud Skills challenge.

52:00 Yeah, absolutely.

52:01 We want you to get, that ability to win a, 500.

52:07 Excuse me.

52:07 The first 500 of our entrants will be able to enter

52:11 to win one of 500 free certificate vouchers for the DP-420 exam.

52:16 So don't miss out on that opportunity.

52:18 Yeah.

52:18 And if you want to get started, visit AKA Ms.

52:22 aka.ms/cosmosdbconfchallenge.

52:24 And you know what, quick feedback helps a ton.

52:27 So make sure that you fill out our evaluation survey.

52:30 We want to know what you thought, so go to aka Ms.

52:33 Cosmosconf2026survey.

52:36 So mouthful of all those URLs I know.

52:40 All right, well, it's back to keynote programming and it's the last part of it.

52:44 And then we're going to get into all the other sessions.

52:46 But we wanted to do a really great featured session.

52:50 I used to watch the Jetsons as a kid

52:52 and this one kind of near and dear to my heart.

52:55 Jade, stop this crazy thing.

52:56 Cosmos DB Best Practices A lesson from Spacely Sprockets.

53:01 Oh, wow.

53:02 Wait, that sounds amazing.

53:04 Here's Sid Anand from Walmart and welcome to my talk.

53:09 My name is Sid Anand.

53:11 I will be talking today about

53:13 Cosmos DB Best practices and architectural patterns

53:16 in the context of building a fictional E

53:19 commerce site for a company called Spacely Sprockets.

53:23 First, a little bit about me.

53:26 My name is Sid Anand.

53:27 As I mentioned earlier, I'm a technical fellow in the data platform at Walmart.

53:32 I've spent about 25 years in the SF Bay Area working on end

53:37 to end data infra at scale at a variety of companies big and small.

53:44 Let's dive into a, product overview of the site we're trying to build.

53:49 First of all, since it's an E commerce site,

53:51 we expect to see some core shopping experience features such

53:55 as a homepage with some sort of integrated search landing page.

54:00 On this page you'd expect to see some product carousels and a search box.

54:05 A user may search for items or click on one of the personalize

54:09 recommended items and that will take him or her to a product details page.

54:13 If the user likes what he sees,

54:16 he can click on add to cart to add that item to his or her cart.

54:21 And later on he or she might decide to check out and that would

54:26 kick off order processing flows that would result in delivery of this item.

54:32 Last but not least, after the user has bought multiple items from the site,

54:38 he or she may want to go back and look at current and past orders.

54:43 And also, like other modern e commerce sites,

54:46 we'll also have other features like profile and preference,

54:51 payment and billing, authentication and account management.

54:54 All of these things are important to have,

54:56 but they're outside of the scope of this talk.

54:59 We're going to focus primarily on the core shopping experience.

55:04 So let's start with a hundred foot view of what we need in engineering.

55:08 We'll start with a classic three tier

55:11 architecture made up of a presentation tier,

55:14 an application tier and a persistence tier.

55:17 There are many options and choices

55:19 for stacks for presentation tiers and application tiers,

55:23 so we'll just go with what we know best.

55:25 We'll pick React Vue and Angular for the presentation tier,

55:29 and we'll pick a spring boot based, platform for the application tier.

55:36 The next big question we need to ask ourselves is what will be

55:40 the wire protocol and format for data

55:42 that is transacted between our different tiers?

55:46 When making a decision like this, we often have to consider multiple things.

55:51 For example semantic richness.

55:53 Does the wire format, have a rich type system that is natively supported?

56:01 And usually if it does have a rich type system,

56:04 it comes with schema enforcement and schema evolution.

56:10 But sometimes schema enforcement and evolution go against developer velocity.

56:15 So we have to decide which of those is more important for us.

56:18 And because we'll have incidents in production,

56:21 we'll need a way to debug the data flying back and forth between microservices.

56:26 So we'll want to also ensure that we have

56:28 a rich tool and ecosystem to support whatever we pick.

56:32 Most often teams are given two choices or they're faced with two choices.

56:38 Either they go with gRPC and protobuf, which is much more efficient,

56:42 then the other option which is REST over HTTP with JSON.

56:47 However, the rest case with JSON is more widely used

56:51 and it is what we'll pick for our talk today.

56:55 So now we've picked our top two tiers and the interchange format between them.

57:01 We have to make a decision about what

57:03 to pick for our database in the presentation tier.

57:07 While making this decision, we can consider either a tabular,

57:13 data model or a JSON based document model.

57:17 If we pick The JSON based document model.

57:19 It simplifies our mental model and our cognitive load.

57:23 It reduces our cognitive load because what we're transacting

57:26 between microservices is also what we're storing in the database.

57:30 We don't have to concern ourselves

57:32 with flattening from JSON into a tabular format.

57:38 And we also want to keep the SQL

57:40 as our query language because it's very expressive.

57:44 Therefore, we need a database that provides both

57:47 the flexibility of JSON and the expressiveness of SQL.

57:52 Luckily, Cosmos provides both as part of its NoSQL API.

57:57 So now we have a full stack.

58:00 What are some of the other benefits of Cosmos?

58:03 Well, we get scalability thanks to its support for horizontal partitioning.

58:09 It's globally distributed, it provides a range of consistency options.

58:15 So we get tunable consistency and through this we get support for transactions,

58:20 which is very important.

58:22 Finally, it gives us a flexible cost model.

58:25 Now being an enterprise database, it also supports change,

58:28 data capture, snapshot and point in time recovery and other features.

58:36 But this is all very good and what we need in a database.

58:40 Now that we have picked our stack, let's talk about the product.

58:45 While we have five different things to pick from, we're

58:48 going to focus on the three most challenging ones.

58:50 The shopping cart, order processing and also viewing a customer's order history.

58:56 Let's start with the shopping cart.

58:58 What are our engineering requirements for the shopping cart?

59:02 First of all, we need it to be globally distributed.

59:05 If a region has an outage, we don't want an outage in our shopping cart.

59:10 We want people to be able to add to their cart

59:13 and view cart no matter what is happening in a given cloud region.

59:18 Now that we have picked a database that works in multiple cloud regions,

59:22 we also care about the data being consistent.

59:26 If a user makes a change to a cart in one region,

59:29 he should see that exact same change reflected in other regions.

59:34 And last but not least,

59:35 we need all of these interactions to be low latency because any

59:39 type of latency friction will cause a drop off or conversion problem.

59:46 Luckily, Cosmos DB provides all of these.

59:49 So we can create a cart container

59:51 that operates in multiple regions and uses session consistency.

59:56 This provides lower latency and better

59:58 availability for writes than globally strong consistency.

1:00:03 Let's look at this in detail.

1:00:05 Let's say the user wants to add an item

1:00:08 to his cart after he's called that add to cart.

1:00:12 The presentation tier forwards the request to the application

1:00:16 tier which will call upsert on the Cosmos DB instance.

1:00:22 Once it's successful, the database will return,

1:00:26 200 or 201 error code, response code, along with a session token.

1:00:32 And that session token will flow all the way upstream back

1:00:35 to the user where it will be stored in a cookie.

1:00:38 Now let's say after this set of steps is finished, the region goes down.

1:00:44 Like we have a total region, one outage.

1:00:46 But now the user wants to view his cart

1:00:49 because we stored the session token in the cookie.

1:00:53 The user will present that session token when he calls View cart.

1:00:58 That will go to the presentation tier and get forwarded to the application tier.

1:01:03 The application tier will then call query

1:01:05 against the cart collection presenting that session token.

1:01:09 If the data has been replicated from the first region

1:01:13 to the second region by the time this call is made,

1:01:17 the item will be found and returned to the user

1:01:19 so the user can view a consistent cart.

1:01:23 But if there are any delays in replication,

1:01:26 rather than showing the user an inconsistent cart,

1:01:29 what will happen is the application tier will continue to retry until it,

1:01:34 syncs with the initial region, at which point it will show the final cart.

1:01:41 The design provides a globally consistent low latency shopping cart.

1:01:45 And it's exactly what we need for our application.

1:01:49 Now that we talked about shopping cart, let's talk about order processing.

1:01:53 Order processing starts when we're, checking out.

1:01:57 We convert the cart that we check out with into an order.

1:02:01 An order is made up of one or more order line

1:02:04 items where each line item encapsulates a product and its quantity.

1:02:09 Let's look at an example.

1:02:11 Let's say the user had a cart with ID 123 and checked out.

1:02:16 As you can see, in this cart there are three items.

1:02:19 The first one is for the foo sprocket, the second one is for the bar sprocket,

1:02:24 and the third one is for the monkey sprocket.

1:02:26 And each of them come with different quantities.

1:02:29 After checking out, the cart will be converted into an order with ID

1:02:34 A and we can see the order line items are carried over.

1:02:40 So what are the engineering requirements for order processing?

1:02:44 When an order is inserted into the database,

1:02:47 we want the order and all of its line items to be inserted, atomically.

1:02:54 It should also be possible to upsert line items

1:02:58 independent of each other and of the order itself.

1:03:01 On the read side, we should make it really easy for an application

1:03:06 to read and order and all of its line items in a single query.

1:03:10 But they may also want to read either

1:03:12 the order or align item independent of one another.

1:03:16 To achieve this, in Cosmos, we have two options for data models.

1:03:21 Option one gives us a Denormalized model,

1:03:24 also known as embedded or a single document model.

1:03:28 Option two gives us a normalized AKA reference, AKA multidoc model.

1:03:33 Let's have a look at these.

1:03:36 Let's say I pick option 1.

1:03:38 In option 1 I have an order JSON.

1:03:41 I see that the partition key is the order ID,

1:03:44 which happens to also be the ID of the document.

1:03:50 There are other, properties in this document that are relevant and important.

1:03:53 For example the status of the order.

1:03:56 Also, the shipping address and payment address are included,

1:03:58 as well as the customer's id.

1:04:00 And if you notice there is an array type called orderline items.

1:04:04 In this array we have each of the orderline items that are part of the order.

1:04:11 In option two, what we do is we create a parent order document and its

1:04:17 children order line item documents and we

1:04:20 store them together in the same container.

1:04:23 Because they use the same partition key,

1:04:25 they are also part of the same partition.

1:04:29 The order line items have their own line item IDs,

1:04:32 but they are all stored with its parent order in the same partition Now,

1:04:39 how do you choose which model to use?

1:04:42 Well, if you have orders that don't have too many line items,

1:04:47 if you know reliably that there will be less

1:04:49 than 50 of those, option one is a good choice.

1:04:52 It's also a good choice if your dominant access pattern

1:04:55 is always on the order with all of its line items,

1:04:58 such as reading the entire order.

1:05:01 And also if you want simplicity and lowest read ru cost,

1:05:05 option one is a good one for you.

1:05:08 But if the orders can be unbounded,

1:05:11 have unbounded number of line items, you must choose option two.

1:05:16 And option two gives you some other benefits as well.

1:05:18 It allows you to update individual line items.

1:05:22 Because you may want to update the fulfillment status,

1:05:24 for example it's been delivered, and also the quantity.

1:05:27 If you amend your order, you also may want to independently query the line

1:05:33 items from each other from the parent order.

1:05:35 And so the multi doc approach gives you this.

1:05:39 And how does it work?

1:05:40 Well, if you use option one whenever you want to read an item,

1:05:44 in this case read an order, you just call readitem for the order and it'll

1:05:49 return the order and all of its nested line items.

1:05:53 If you want to, modify or delete the order and its line items,

1:05:59 you can also issue point operations.

1:06:01 For those with option two, you get more flexibility.

1:06:05 For example, if I want to, read a parent and all of its children together,

1:06:11 I would use the query shown here.

1:06:13 I would select star from order where the partition

1:06:16 key is in this Case the order id.

1:06:18 But I'm also given the ability to do point reads and read

1:06:22 a single document like a single line item and update its quantity, for example.

1:06:27 And last but not least,

1:06:29 whenever I'm doing any changes across the order and its line items,

1:06:33 I can use transactional batch to either update them

1:06:36 or delete the whole order and the line items.

1:06:41 Now that we've solved two of the hard problems, let's get to the third one.

1:06:44 In this case, a customer wants to see its order history,

1:06:48 which means all of the orders that it

1:06:52 has completed and their associated line items.

1:06:55 And we want to do this with low latency and low cost.

1:06:58 Now, as the order container is partitioned by order id,

1:07:02 querying by the customer ID results in a cross partition query.

1:07:08 Cross partition queries incur higher cost and latency than point queries.

1:07:13 The greater the number of partitions,

1:07:15 the greater the latency and cost of cross partition queries.

1:07:20 Now, does Cosmos provide a way to efficiently query by customer id?

1:07:25 Yes it does.

1:07:26 And it's called Global Secondary Indices.

1:07:30 Let's look at how this works.

1:07:32 Let's start with our normal container, the order container.

1:07:36 As we mentioned earlier, it is Partitioned by Order ID,

1:07:39 but it contains other JSON properties like Customer ID.

1:07:43 Now, I define a new index container called CustomerOrders,

1:07:48 which is defined by a covering query shown here.

1:07:51 Select star from order and a partition key.

1:07:55 In this case, the partition key will be customer id.

1:07:59 Once I set this up, the customer ID will get

1:08:02 live updates from the order via the full fidelity change feed,

1:08:07 filtered by, the covering queries seen below.

1:08:13 Now, the customer orders container is called an index container.

1:08:17 It's a read only container.

1:08:19 Apps cannot write to it,

1:08:21 but writes can only come to it through the order container.

1:08:25 And the beauty of this is a user can query the order container

1:08:29 by order ID or against the customer

1:08:31 orders container by customer ID at both times.

1:08:35 They are point queries, so they're cheap and fast.

1:08:37 We avoid expensive cross partition queries.

1:08:42 Now that we've covered three of the main core pieces, of functionality,

1:08:46 let's talk about some optimizations in terms of latency and cost.

1:08:51 There are three types of optimizations worth considering.

1:08:54 We've already talked about the first one which is to use a gsi.

1:08:58 The second one is about partition scoping.

1:09:02 First of all, what is partition scoping?

1:09:05 Let's look at an example.

1:09:08 Let's pretend that we have a user collection.

1:09:11 That user collection stores documents as shown here.

1:09:14 User JSON and it's backed by a POJO class shown here on the Right user job.

1:09:22 If I want to select like George Jetson from the database,

1:09:27 I would provide a point query as shown here.

1:09:30 Select star from C where C ID equals a bind param.

1:09:34 There's one problem here.

1:09:36 Every time I issue this query, there's an extra round trip that is used

1:09:41 to parse the query and generate a query plan.

1:09:47 If, I execute this query every time,

1:09:51 if I use this code every time I execute the query,

1:09:55 the query will incur an extra round trip.

1:09:58 But if I add the line shown here where I

1:10:01 set the partition key on the Cosmos query request options,

1:10:05 then the first time will be the only time that the query will be,

1:10:11 will generate a query a plan,

1:10:12 and then it will be cached in whichever pod this application

1:10:16 is running in so that subsequent queries avoid an unnecessary round trip.

1:10:21 And this can cut your point query latency by up to 50%.

1:10:27 Last but not least, let's talk about payload size reductions.

1:10:32 What I'm showing here is how RU costs vary as a function

1:10:37 of of the payload size and also based on the type of operation you're doing.

1:10:44 As you see, as I go to larger and larger documents in the payload,

1:10:51 the cost in terms of RUs increase,

1:10:53 but they increase more dramatically for point reads.

1:10:57 Point reads, double in cost, as a power of 2 of the document size.

1:11:05 However, you'll notice that under 32 KB it's cheaper than point queries.

1:11:10 But because point queries have a very flat,

1:11:14 slope over time, they end up costing less.

1:11:19 So let's look at the example from before, where we had an order I.D.

1:11:24 in model one with its array of line items.

1:11:27 In this case, let's pretend I had 100 items in my cart when I checked out.

1:11:34 If, I want to save space, what I can do is compress the order

1:11:38 line items using an algorithm like zstandard and base64,

1:11:43 encode it so that I can store it as a string field, a string property.

1:11:48 I've also shown a, property called compression,

1:11:50 which has a bunch of metadata, but that is not needed.

1:11:53 Assuming we kept that metadata, what will happen?

1:11:57 If we use Zstandard, with its default compression level of 3,

1:12:03 we can save 75% of space,

1:12:06 and this will result in almost, up to 75% in RU reduction.

1:12:13 At this point, I'm, at the end of my talk and I

1:12:15 want to thank the following people for their guidance and support.

1:12:18 Thank you.

1:12:21 Hello everyone.

1:12:22 My name is Chander Dahl and I'm the CEO of Kasden,

1:12:26 an elite AI consulting firm delivering high impact Enterprise grade solutions,

1:12:32 including custom built AI chatbots and advanced

1:12:36 automation systems tailored to complex business needs.

1:12:41 Today we're going to talk about one of the most important concepts,

1:12:45 which is agent memory.

1:12:48 And by the way, it's the same for your chatbots.

1:12:51 And why is agent memory so important and how to get it done right.

1:12:58 So today I'm going to show you four strategies,

1:13:01 starting from a strategy in which we talk directly to LLMs.

1:13:06 And let's not forget LLMs are stateless

1:13:08 and they do not have any memory whatsoever.

1:13:12 And then we'll use the other three strategies,

1:13:15 sliding window, hierarchical and entity graph.

1:13:19 And one of them will lead to the highest precision

1:13:23 and that will be the high precision or recall winner.

1:13:27 However, that doesn't always mean that it's going to be hundred percent recall.

1:13:32 With that particular strategy,

1:13:34 we'll show you different use cases and from that you

1:13:38 can decide what's the best for your particular use case.

1:13:43 We will start with 60 seed messages,

1:13:46 and the reason we chose 60 seed messages is we saw

1:13:50 that in a lot of companies when they do AI testing,

1:13:54 especially on their chatbots,

1:13:56 a lot of times they go up to 10 to 15 conversations only.

1:14:01 And guess what?

1:14:03 All the three strategies provide 100% recall up to 30 turns in most cases.

1:14:10 And that's why by the time you end up in production,

1:14:13 you may not even know that you have less than 100% recall.

1:14:19 And that's a, big no, no.

1:14:22 We also will show you 10 recall questions.

1:14:26 And the first five questions are really easy

1:14:29 and all the three strategies get them right.

1:14:33 However, the next six questions are more nuanced

1:14:36 and that shows how you need to change your testing strategies,

1:14:41 especially if you're leaning more towards

1:14:43 the easier questions during your AI POCs.

1:14:48 So let's have some fun and define the problem.

1:14:52 If you started working like me before ChatGPT was launched on AI applications,

1:15:00 you may have noticed the context limits really small.

1:15:05 The token window limit would be the most important thing.

1:15:10 Before you even started creating these applications,

1:15:13 you would notice right up front that these LLMs actually are stateless.

1:15:19 So after ChatGPT was launched,

1:15:22 a lot of people assumed that LLMs are actually stateful.

1:15:26 And in some cases they even thought that it has a memory.

1:15:30 A lot of that has to be done by engineers like you and I.

1:15:34 You might have heard of the famous quote, I told you my name,

1:15:37 my budget, my team, my deadlines, and you forgot everything.

1:15:42 And that is why memory happens to be

1:15:44 the most important conversation in enterprise applications worldwide.

1:15:50 So now that we know that LLMs have this problem

1:15:53 that they are stateless and every API call starts from scratch.

1:15:58 How do we solve this problem?

1:16:01 Well, very easy.

1:16:02 We can send the full conversation history every single time.

1:16:07 Yes, except we know all LLMs have a limit.

1:16:11 All right, then maybe you can just summarize it.

1:16:15 Let's try it out in case you send the entire

1:16:19 conversation and let's assume that's within the limit an LLM expects.

1:16:26 In that case you're still having costs that are going to explode.

1:16:32 You're also going to slow down the LLM.

1:16:35 What if you start summarizing parts of it?

1:16:39 In that case you still have that issue

1:16:41 that you may be losing some context, maybe some fact,

1:16:47 anything real world that you really care about

1:16:50 that got compressed because all summarizations have a compression problem.

1:16:56 And then finally users will have to repeat themselves quite a bit.

1:17:02 So why does memory matter?

1:17:04 One of the most important things to remember is that more

1:17:07 than 70% of enterprise projects will require multi time agents.

1:17:13 And the number one gap cited by developers happens to be long term context.

1:17:19 And by the way, if you go with the LLM alone,

1:17:21 which really has no memory, and you go with the best memory strategy,

1:17:26 we're going to compare today,

1:17:27 you may be looking at a 20x token cost difference between these strategies.

1:17:33 And that's why memory isn't just a feature checkbox.

1:17:36 It's an architectural decision that determines cost,

1:17:39 recall quality and user experience.

1:17:42 So the first strategy happens to be sliding window which

1:17:45 keeps your recent messages and then summarizes the messages before it.

1:17:50 An analogy would be a security camera that keeps the last eight hours

1:17:54 of footage and it writes a one page summary of everything before that.

1:17:59 That's really easy to implement and that's a strength.

1:18:03 And it's also going to have a low token cost, in this case roughly 1100 tokens.

1:18:09 And it's great for short conversations, especially less than 30 turns.

1:18:14 And this is something to keep in mind.

1:18:16 Just because it's great for that particular use case does not mean it's going

1:18:22 to be great for another use case where we actually require a real world fact,

1:18:27 even the one that was hundred turns ago or a thousand turns ago.

1:18:33 In that particular case, recall is going to be lower.

1:18:36 And in today's demo it's got about a 60% recall.

1:18:40 Let's not forget you will have 100% recall if it is less than 30 turns.

1:18:46 So remember, it isn't about the strategy, it's more about the use case.

1:18:50 And based on that use case, we need to pick the strategy.

1:18:54 Strategy 2 hierarchical memory think of this as three different tiers,

1:18:59 the first year being hot.

1:19:02 For example, a company's knowledge management system will

1:19:05 have your team's messages as your first tier

1:19:08 and that's part of a conversation and you

1:19:10 want to send them as it is uncompressed.

1:19:14 Number two is your weekly meeting notes,

1:19:17 let's call that tier two and kind of compressed

1:19:21 summaries because tier two not as important as tier one.

1:19:26 And then finally the least used out of the three which could be

1:19:30 your company wiki and those are extracted facts in your long term storage.

1:19:36 So better recall than sliding window.

1:19:39 And of course it can do better than it, but not as good as something that will

1:19:45 always present the right fact every single time.

1:19:48 Because tier three, as you can notice it

1:19:51 will miss on certain facts here or there.

1:19:54 Finally we got entity graph and I love this analogy,

1:19:57 it's like a detective's case board where you have all

1:20:00 these different flags and then you got linkages between them.

1:20:03 Right as every single card with connections between them.

1:20:07 And that's entity graph.

1:20:09 So what does it store?

1:20:10 Stores all the entities, Anything real world,

1:20:13 A name, organization, anything real world.

1:20:17 Any entity that is real world gets stored facts.

1:20:21 These are key value pairs for that particular

1:20:24 entity and you can have any amount of these.

1:20:26 And finally embeddings you may have heard

1:20:29 of vectors and vector databases got created simply because

1:20:34 of these LLMs wanting to search basically do semantic

1:20:38 search on your data and get you that particular retrieval.

1:20:44 Now in this case what I love about Cosmos DB

1:20:46 is that you have the native vector search inside Azure,

1:20:50 Cosmos DB and then finally relationships which are connections between entities.

1:20:56 Now why do you get 100% recall in this particular case?

1:20:59 I mean the reason is simple because we don't really have any compression.

1:21:03 We're doing the vector search,

1:21:05 we're getting those entities and those entities have

1:21:08 the facts and we can get all of that back.

1:21:11 So in this particular case, for this particular data set,

1:21:14 we are able to get 100% recall except we're also using average 1660 tokens.

1:21:21 That's pretty much 50% more than the first strategy.

1:21:24 That's sliding window and remember that's cost.

1:21:27 So that's another consideration.

1:21:30 So even though I would say there are four approaches,

1:21:32 you might have noticed I start with zero.

1:21:35 Why is that?

1:21:36 That's because I'm an engineer and as engineers

1:21:39 we use arrays and we start with zero.

1:21:41 That's not the case.

1:21:43 I really don't think it's an apples to apples

1:21:45 comparison because LLMs really have no memory, no context.

1:21:49 What you're trying to do unless you provide it.

1:21:52 So for that reason I don't think it's a good comparison.

1:21:55 But I still want you to see that you may be able to get 92 tokens,

1:21:59 but you got a 0% recall.

1:22:02 However, the other three strategies, very important, as you can see,

1:22:06 we go from 1100 tokens all the way to 1660.

1:22:09 And in this particular scenario, you go from 60% recall to 100% recall

1:22:14 at a, one and a half times the token limit.

1:22:17 But here's what I love about Azure Cosmos DB.

1:22:20 Same database, same SDK, same partition key, different recall guarantees.

1:22:25 And I can use all these three strategies at the same exact time.

1:22:31 So that leads me to why Azure Cosmos DB For agent memory.

1:22:35 Well, that's because we are in Azure Cosmos DB conference.

1:22:39 That's not the case.

1:22:40 It's really because at the end of the day all my data is in Azure Cosmos DB.

1:22:46 And by the way, for someone like me who's been working on DocumentDB, that was,

1:22:50 I think it was 2016 onwards I started working

1:22:52 on it and then it became Azure Cosmos DB.

1:22:55 And I know we now have another DocumentDB which is

1:22:57 a little different than that DocumentDB that was there long back.

1:23:03 My data and my client's data lives on Azure Cosmos DB.

1:23:06 It used to be a big problem to now have another vector storage,

1:23:11 a completely new database just for the vectors.

1:23:15 Well, not anymore we can actually have that data.

1:23:18 Your embeddings, your vectors live right alongside your data.

1:23:21 I don't actually need to have another native vector search database.

1:23:26 The schema is flexible and we talked about that.

1:23:29 I, love the session isolation.

1:23:31 My partition key in this case happens to be session id,

1:23:34 so I have zero cross partition queries.

1:23:37 And then finally, for this demo, I'm not worried about global scale at all

1:23:42 because all we have is about 60 different messages.

1:23:44 But once I go into production,

1:23:47 I don't have to worry about global scale because now I

1:23:51 can scale as much as Azure Cosmos DB would allow me,

1:23:55 which is literally all over the world.

1:23:58 And that's why Azure Cosmos DB.

1:24:01 So let's talk about the benchmark where we have about 10 recall questions.

1:24:05 You got the core questions, basic recall.

1:24:08 Why do we have that?

1:24:09 A lot of times what we noticed was all these POCs

1:24:12 that people were doing had a bunch of the same kind of questions.

1:24:16 And a lot of times those questions were created by AI itself.

1:24:19 Well, but it was also 100% recall based on questions

1:24:24 that probably don't understand how these strategies actually work.

1:24:27 So what you'll notice is 100% recall

1:24:31 for all three strategies for the core questions.

1:24:35 We also notice that every single data set is different.

1:24:38 We work in healthcare, you know, manufacturing, finance,

1:24:42 fintech, tech, pretty much name any major domain.

1:24:46 And all these bigger corporations worldwide

1:24:48 in those domains have very nuanced data.

1:24:52 That's why I wanted to show you how much difference gets made in just five

1:24:56 questions the moment you start asking nuanced questions

1:25:00 on that particular data and how the Mrs.

1:25:04 Happen.

1:25:05 So for example, in this case,

1:25:06 sliding window only gets one out of those five correct.

1:25:10 So your recall happens to be only 20%.

1:25:13 But if you have the right use case,

1:25:15 sliding window could be at 100% and that's why it gets confusing.

1:25:20 Before we do the demo, I would like for you to ask yourself which strategy is

1:25:25 going to remember a, URL mentioned once 40 plus turns ago?

1:25:32 And I hope you get the answer right.

1:25:34 Let's dive into the demo.

1:25:36 As you can see here, I've got two providers.

1:25:38 Here's Azure OpenAI and you can also click OpenAI.

1:25:40 It'll work with both the keys.

1:25:42 So if you have Azure OpenAI,

1:25:44 assuming you have the same exact keys for whatever model you're trying to use,

1:25:49 here's two different demos.

1:25:50 One is Kasden and another one is default.

1:25:53 And remember, they're both fake data, so please feel free to use your own data.

1:25:58 I'm not going to ask anything from the Direct LLM because

1:26:01 at the end of the day we're not going to get any answers.

1:26:04 Now we're going to compare the other three strategies which is sliding window,

1:26:08 hierarchical and entity graph on the right hand side.

1:26:12 It's just a way to see what the results are and what to expect.

1:26:17 So let's ask the first question and see

1:26:20 what it does while it's answering the question.

1:26:23 I just want to show you that all three of these are going to pass.

1:26:27 So this is actually a really easy question.

1:26:30 No problem whatsoever.

1:26:31 All of these strategies will get you the same exact answer,

1:26:33 which is cast and offerings are consulting, training and recruiting.

1:26:39 Same exact answer pretty quick.

1:26:42 And you'll notice hierarchical was extremely fast,

1:26:45 but entity graph is actually slower still gets you the same three answers.

1:26:51 So now I'm going to skip the next four questions and go to number six.

1:26:56 The reason is because the next four questions we're

1:26:58 going to get the same result from all the strategies.

1:27:03 So here, here's a result which says about us in contact us.

1:27:08 All right, so let's go to round number six

1:27:11 and open this and you notice it's missing Consulting.

1:27:15 Let's go to Hierarchical.

1:27:18 Run this round.

1:27:20 It's got about us and it's got consulting, which is true.

1:27:23 And that's exactly what we were expecting.

1:27:25 On the right hand side, here's your Entity graph.

1:27:32 It's got about us and it's got, consulting.

1:27:34 It's got both.

1:27:36 The answer is correct.

1:27:38 All right, so now let's go back to Sliding

1:27:40 window and we're going to go to number seventh.

1:27:44 And here we're going to open the round and see what we were expecting.

1:27:47 All right, it's got a about, it's got webpage, it's missing products,

1:27:54 and it's also missing from our stored graph context, which is true.

1:28:00 Let's go to Hierarchical.

1:28:05 And as, you can see, it's got pretty much everything,

1:28:08 except it's still missing products and also

1:28:11 missing from our stored graph context.

1:28:15 Let's go to Entity Graph.

1:28:21 And you may have noticed Entity Graph is actually slower.

1:28:24 So that's another thing to keep in mind is what your, use case is.

1:28:30 So you've got, you've got actually everything.

1:28:32 You got products, you've got the About Chander Dahl, you've got the webp.

1:28:36 It's actually perfect.

1:28:38 And it also says from our Stored Graph context.

1:28:41 That's pretty amazing.

1:28:43 Let's go back to the eighth round.

1:28:45 I'm going to pass because they all get the training URLs.

1:28:48 It's the same exact answer for all three of them.

1:28:51 And now let's just show one more.

1:28:53 And let me go back to Sliding Window.

1:28:56 Let's go to number nine, very quick.

1:28:59 And it's got something.

1:29:01 Let's see what it is.

1:29:02 Okay, so it's actually missing presentations, which is true.

1:29:07 Let's go to Hierarchical.

1:29:09 It's got presentations, so hierarchicals got all of those.

1:29:13 And same with Entity Graph.

1:29:20 And then the 10th round, it's a very nuanced round.

1:29:24 All that is getting wrong is not the response.

1:29:28 They have all the responses correct, both of them,

1:29:30 except they're missing from our stored graph context.

1:29:37 All right, so how many of you got that right?

1:29:40 I hope it's all of you.

1:29:42 So what you just saw was directllm has no prior contacts.

1:29:46 We didn't do that part of the demo.

1:29:48 But the idea is you're not gonna get any response if you didn't.

1:29:51 If the LLM does not have that data, unless you fine tune it to have that data.

1:29:56 Whereas, Sliding Window, the first five questions correct.

1:29:59 The next six, it only got one right, which is the eighth one,

1:30:03 which we didn't run but it was really easy and it was going to run.

1:30:07 Now you've got hierarchical, which was three out of ten,

1:30:11 which means in this case about eight out of ten.

1:30:14 So you notice what's happening, even though it was three out of five.

1:30:17 Sorry, not three out of ten.

1:30:19 We had the first five really easy questions and you may think you have an 80%

1:30:25 recall where all you had was a 60%

1:30:28 recall because three out of five were correct.

1:30:31 An entity graph in all the cases was 10 out of 10.

1:30:34 And, one of the things I want you to remember is that if

1:30:36 you had a sliding window which never requires you to go beyond 30 turns,

1:30:42 you would have 100% recall and sliding window.

1:30:45 So it's really not the recall as much as it is the combination of your strategy,

1:30:49 your use case and your data done right?

1:30:52 So here's your scorecard.

1:30:55 If you remove the blue, which is the first five questions,

1:30:57 sliding window is only one question.

1:31:00 That's 20% recall.

1:31:01 Hierarchical, 3 out of 5, that's 60% recall in entity graph, 10 out of 10.

1:31:07 So.

1:31:07 And 5 out of 5 and 5 out of 5.

1:31:09 So that's 100% recall cost versus recall.

1:31:13 The trade off is right here.

1:31:14 And it's important because if your use case is literally less than 30 turns,

1:31:19 you're better off going with sliding window because the cost is

1:31:22 one and a half times more in terms of entity graph.

1:31:25 You're using the least amount of tokens.

1:31:27 You're hoping you get the response also faster.

1:31:30 So that's another cost to keep in mind is performance cost.

1:31:33 Otherwise, if you really care about higher recall, well,

1:31:37 in that particular scenario,

1:31:38 entity graph would be a much better deal altogether.

1:31:42 So an easy way to look at this would be sliding

1:31:44 window for less than 30 turns supports the chat recent contacts hierarchical,

1:31:50 about 30 to 100 turns.

1:31:52 And again, you know, this is more,

1:31:54 it'll depend on your data, but let's just say ballpark,

1:31:57 that's what you're looking for, especially if you're using it

1:32:01 for planning or consulting and you have to have some key facts.

1:32:05 And then entity graph, for example,

1:32:07 you have your CRM bot or something where every fact matters,

1:32:11 like you don't want a response

1:32:13 with a financial number that's actually not correct,

1:32:16 you know, and it can also scale to way more than 100 terms

1:32:20 because at the end of the day you're really getting the fact correctly.

1:32:25 So again, you can start with one, you can also upgrade later.

1:32:30 You can also do a hybrid, which is also not a bad idea.

1:32:34 And in some cases you can have a combination

1:32:36 of, let's say a sliding window as well as Entity Graph.

1:32:39 You know, for 90% of your cases you may have sliding window,

1:32:43 especially for less than 30 turns.

1:32:45 And then you can go to Entity Graph

1:32:48 whenever there's a premium use case behind it.

1:32:51 So again, memory is a spectrum and you need to choose by recall, not by height.

1:32:57 And you gotta remember what's your use case and what your data is like.

1:33:03 What we covered was the problem,

1:33:05 which is LLMs are stateless, they have no memory.

1:33:08 Then we really covered the baseline plus

1:33:11 the next three strategies and then the architecture,

1:33:14 which was really easy by the way.

1:33:16 You must be wondering why five Cosmos containers to remember for Entity Graph.

1:33:21 You will have different containers for Entity Graph.

1:33:24 And that's not at all a bad idea in this particular case.

1:33:28 That's why we did that.

1:33:29 You can, you can do it multiple different ways,

1:33:31 but for that reason we had more than just one container.

1:33:36 And then you've got the live demo which had 10 recall questions.

1:33:40 Then you had the evidence which was 100% recall in terms of Entity Graph.

1:33:45 And then the decision,

1:33:46 which isn't really about the recall numbers that we shared here,

1:33:49 but a lot more than that and we discussed every single of those.

1:33:54 Now just remember this is not production data.

1:33:58 And this is literally fake data.

1:34:00 But this is also not production level code.

1:34:02 However, it's a good POC if you want to run.

1:34:05 And that's your barcode.

1:34:06 If you click that barcode,

1:34:08 it will take you to this blog post that explains the nuances,

1:34:12 how the decisions were made,

1:34:13 and it also explains to you the different parts of the code.

1:34:17 I just want to show you this real quick for your code to remember.

1:34:23 Here's your direct LLM.

1:34:25 And if you're a dot net developer you can look at the right

1:34:27 hand side of the code and if you're a Python developer,

1:34:29 you can look at the left hand side of the code.

1:34:32 Directllm.

1:34:33 It's literally sending the system prompt and then

1:34:36 creating the message chain and then making a call.

1:34:40 Very simple.

1:34:41 That's how you've been making all your calls so far.

1:34:44 Next, the sliding window.

1:34:45 And as you can see here we have

1:34:47 the summary for anything beyond 30 messages and then

1:34:51 just 30 messages where everything is uncompressed and it's

1:34:55 part of the context we're sending to the LLM.

1:34:59 And as you can see here, the window size is 30 and this is your one hour,

1:35:03 which is we just chose that time to live.

1:35:06 And that could change for whatever you're trying to do.

1:35:10 And then you've got the hierarchical memory

1:35:11 and here is where all your tiers are.

1:35:14 And as you can see, here's your tier one size,

1:35:17 your tier two block and the max tier.

1:35:20 Same for both Python as well as Net.

1:35:23 And then you've got the entity graph.

1:35:24 This is actually really easy to produce because

1:35:27 your entity entities and your embeddings are living together.

1:35:32 So one thing to remember is that this is exactly how we anyways code

1:35:37 when we are doing programming because we have the same exact objects now we just

1:35:42 have a way to retrieve those objects and then make it part of of your LLM

1:35:48 answer by giving it the context which it needs to give you the response.

1:35:53 And finally here's your read path on entity graph.

1:35:57 And this is just a small blog post that I prepared only for the demo.

1:36:02 However, a lot more detailed blog post is right here.

1:36:06 If you want to do a more involved code walkthrough,

1:36:09 you can click that link and it goes through a lot more than what I just shared.

1:36:16 So let's not forget,

1:36:17 here's the link to that and I hope you take advantage of it.

1:36:21 Let me know if you have any feedback for me.

1:36:24 I had a blast and hope you have

1:36:26 a blast coding and using this in your applications.

1:36:29 Thank you for having me.

1:36:31 Have a great day and a great rest of the conference.

1:36:37 Welcome back.

1:36:38 Thank you so much to our speakers.

1:36:40 I believe that was Sid or Ann Chander.

1:36:43 They were both great, right?

1:36:45 Yeah.

1:36:45 Honestly that was a very fun and super practical look at Cosmos

1:36:49 DB's best practices through the story of get excited, spacey sprockets.

1:36:54 Jane, get me off this crazy thing.

1:36:57 Right?

1:36:58 Well, at least if you don't have a well architected partition key.

1:37:01 Right.

1:37:03 Always on availability,

1:37:05 low latency across the galaxy and keeping costs under control.

1:37:09 Those are all trade offs we all face when we build real systems.

1:37:14 But you know what, we're going to talk way more about

1:37:17 that and I want to just make sure that we're acknowledging you, the audience.

1:37:21 So let's take a quick look.

1:37:23 We've got some really, really nice comments, but I wanted to just show this one

1:37:29 the thought processes people go through when designing solutions.

1:37:34 You know, there's a lot you have to make.

1:37:36 So many considerations when you're designing a production application.

1:37:40 And you know what?

1:37:41 You all are here to learn about that.

1:37:42 Patty, what do you got?

1:37:43 Yeah, honestly, for me,

1:37:44 I've got nothing but love in this group chat or in the comments section.

1:37:49 So please, if you are around.

1:37:53 We'd love to hear from you.

1:37:54 Feel free to drop a comment, share what you're building or, just show some love.

1:38:00 We're back into the program.

1:38:02 Thank you for the love.

1:38:03 Oh, thank you.

1:38:04 Thank you for the love.

1:38:05 We're shifting into one of the most important

1:38:07 patterns in Azure Cosmos DB in our next session.

1:38:11 One of my favorite teammates.

1:38:12 I know yours too, Justine Cocchi takes us

1:38:15 deep into the Azure Cosmos DB Change feed.

1:38:17 How it works under the hood,

1:38:19 the patterns that hold up at scale and the live demo of debugging and lag,

1:38:24 recovering processing and production is going to be great.

1:38:27 Yeah, you know Jay, Justine is amazing.

1:38:30 And if you love Justine.

1:38:31 Yeah.

1:38:31 And if you haven't used Change Feed, every write becomes a durable ordered event

1:38:36 stream that you can fan out to services, search and analytics,

1:38:39 event driven architecture without staying on a message bus.

1:38:43 So let's hear it all from Justine.

1:38:46 Here's mastering the Azure Cosmos DB Change Feed patterns,

1:38:49 scaling and real world architectures.

1:38:51 We'll see you soon.

1:38:56 Hi, I'm Justine Cocchi and I'm a program manager on the Azure Cosmos DB team.

1:39:00 I'm really excited to talk to you about Change Feed.

1:39:03 Today we'll cover what it is, how to read it,

1:39:05 and some tips for debugging for real world applications.

1:39:10 So first, what is the Change feed?

1:39:12 It's a persistent record of all changes to items

1:39:15 in your container that's ordered by modification time.

1:39:18 As your client app is making writes to your items,

1:39:21 you can then read them in the Change feed with your consumer app.

1:39:25 Your Change Feed consumers are processing changes per partition.

1:39:29 So let's take a look at what that looks like under the hood.

1:39:32 Because Cosmos DB is a distributed database,

1:39:35 your data is distributed across multiple physical partitions in a container.

1:39:40 In this example, I've got three physical partitions.

1:39:43 Each physical partition can have its own independent change feed

1:39:46 reader that's reading the changes in order for that partition.

1:39:50 Order is guaranteed within a logical

1:39:53 partition key in your physical partition range.

1:39:57 Your continuation token tracks progress of that change

1:40:01 feed processing for a given container.

1:40:04 And this means that each of your partitions

1:40:06 can have independent continuation tokens and independent processing.

1:40:09 This is great for efficiency because it really allows you to scale your chain

1:40:13 tree processing even for extremely large

1:40:16 containers with a high volume up writes.

1:40:20 There's a couple of different Change feed modes that you can choose from.

1:40:23 The default Change Feed mode and what's enabled

1:40:26 on all of your accounts is latest version mode.

1:40:29 In this mode you get the latest create or replace

1:40:32 for every item items that are deleted from your container.

1:40:36 No longer appear in the feed, and this means you won't get a notification

1:40:40 for deleted items in the change feed itself.

1:40:43 Any item that still exists in the container,

1:40:45 you can read the latest version of it.

1:40:47 There's infinite retention of all of these changes

1:40:49 in your container and you can go back and reading

1:40:52 even from the very beginning of your container

1:40:54 to get all of the items and their updates.

1:40:58 This is really great for a lot of change

1:41:00 feed scenarios and powers most change feed apps today.

1:41:03 There's also all versions in deletes mode.

1:41:06 With all versions and deletes, you get every create,

1:41:09 update and delete including TTL expirations.

1:41:13 Items for TTL expirations will appear in the feed in order of their purge time,

1:41:18 which may be slightly later than their actual expire time.

1:41:22 Retention of changes in this feed is based

1:41:25 on the continuous backup window for your account.

1:41:27 So either seven days or 30 days, depending on what you've configured,

1:41:31 you can read all of these changes within that window.

1:41:34 This mode is really great for audit logs or real

1:41:37 time processing where you need to react to every change.

1:41:40 Consider an application that is replace heavy where you have a high

1:41:44 volume of updates to a single item within a short period of time.

1:41:48 For all versions in deletes mode, you would get every single update in the feed,

1:41:52 whereas latest version you would only get

1:41:54 the latest version per change feed pull.

1:41:57 Depending on the needs of your application, each mode may be a better fit.

1:42:01 So it's important to consider what your app is trying

1:42:04 to do and choose the mode appropriate for your app.

1:42:09 There's a couple of different ways to actually consume the change feed,

1:42:12 and we're going to cover three of these ways.

1:42:15 The first is the change feed processor.

1:42:17 This works on a monitored container

1:42:20 or the container that you're reading your changes from.

1:42:23 When you create a change feed processor,

1:42:24 you can create multiple instances to handle scaling of all the partitions.

1:42:30 And this is the compute environment that you're

1:42:34 actually deploying your change feed processor on.

1:42:37 So you can imagine this like AKS,

1:42:39 where you'd have multiple pods that represent multiple instances.

1:42:43 The code that's actually being executed when a batch

1:42:46 of changes is processed is your handler delegate.

1:42:49 This is like the business logic or the brains of your change be processor.

1:42:53 Here is where you handle what you're reacting to for the change.

1:42:58 Maybe it's writing to something downstream or maybe you're doing some analytics.

1:43:03 This is really where you write your change feed

1:43:07 logic and the Change feed processor is available in both.

1:43:10 Net and Java.

1:43:11 Now, as your changes are being processed,

1:43:14 the lease container helps to Maintain state and checkpoints for keeping track

1:43:20 of where you are in processing for all of your various partitions.

1:43:24 Let's take a look at some lease

1:43:25 management and really understand how our physical partitions

1:43:29 map to leases which map to the instances

1:43:32 we've already established as a distributed database.

1:43:35 Cosmos DB can scale out to multiple physical partitions

1:43:38 and each physical partition owns a range of partition key values.

1:43:43 That top row is our physical partitions.

1:43:47 In this example we have six physical partitions

1:43:49 and each of them have their own lease.

1:43:52 Leases are always one to one with partitions

1:43:54 and this is managing the state of where we are processing.

1:43:57 In terms of the modifications to that individual partition.

1:44:02 You'll notice they each have their own

1:44:04 continuation token which is what maintains the state.

1:44:08 Now your Change Feed processor is deployed across multiple

1:44:11 instances and each instance owns a number of leases.

1:44:15 These are evenly distributed across instances to ensure

1:44:19 that as you're processing you can keep up with these changes.

1:44:23 Assume you only had one instance that's handling all of your leases.

1:44:27 If that instance goes down, it might impact the resiliency of your application.

1:44:31 However, if you have more instances than the number of physical partitions,

1:44:34 you would have idle instances that aren't able to do any work.

1:44:38 The Change Feed processor automatically rebalances leases across

1:44:42 these instances to ensure the maximum efficiency of your processor.

1:44:46 Let's say one of your instances goes down.

1:44:49 The leases that are owned by that instance will

1:44:51 automatically be rebalanced across all of the other healthy options.

1:44:56 The next way to read change feed is through the Cosmos DB trigger.

1:45:00 This is an Azure functions trigger that is

1:45:04 a really simple way to read change feed.

1:45:06 It's built on Change Feed processor under the hood.

1:45:08 So all of the lease and distribution of partitions

1:45:10 that we just reviewed still applies in the Azure functions trigger.

1:45:15 However, the management is done for you.

1:45:18 Because it's built on Azure functions, it's serverless.

1:45:21 It has built in auto scaling to dynamically react to the number of instances

1:45:26 you really should be having without over

1:45:28 provisioning instances that are not actively being used.

1:45:32 This is a really easy way to build the change

1:45:35 feed and the logic of your Azure function is like

1:45:38 the delegate in the Change feed processor where you're just

1:45:41 responsible for the business logic of processing the changes themselves.

1:45:46 The last option is the Change Feed pull model.

1:45:49 This model gives you the most control over how to read the change feed,

1:45:53 but also means you need to have all the safeguards put in place yourself.

1:45:58 So there's no lease container for maintaining state.

1:46:01 You instead use continuation tokens where you're

1:46:04 responsible for storing and maintaining those continuation tokens.

1:46:08 You're responsible for any error handling.

1:46:10 There's no built in retries.

1:46:13 You need to handle all of that in your client code.

1:46:16 While it does give you full control, it's also more code to manage.

1:46:21 One additional benefit of the pull model say you

1:46:24 wanted to only process changes for a specific partition range.

1:46:27 This is possible in the pull model because again,

1:46:30 you have full control over what is actually being pulled.

1:46:34 Now that we've got our Change Feed app,

1:46:36 let's talk a little bit about monitoring and how we can debug

1:46:39 some common failure modes to ensure that our Change Feed app is resilient.

1:46:45 Some of the key metrics to monitor for Change Feed are the estimated lag.

1:46:50 This measures the gap between the latest change that actually occurred

1:46:54 in your container and the last processed item in your change feed.

1:46:58 This is going to be your most

1:46:59 important metric for monitoring your change feed health.

1:47:02 While this estimated lag gives you a number of outstanding changes,

1:47:06 it is intended to be an estimate and it's not

1:47:08 always the exact number of changes that are actually pending.

1:47:12 The key thing to measure here is not exactly what the number is,

1:47:15 but the slope of how the number changes over time.

1:47:18 Is your lag steadily increasing?

1:47:20 Is it decreasing?

1:47:21 Are you staying constant?

1:47:22 This will really help you understand if your Change Feed app is healthy.

1:47:27 Some other things you can take a look

1:47:28 at is the throughput of your change feed processing.

1:47:31 You can look at the number of items that are processed

1:47:33 per second and the RU consumption of your change feed processor.

1:47:38 It's important to look at our use not only of your source

1:47:41 container that you're monitoring the change

1:47:42 feed of, but also the lease container.

1:47:45 If you have a lot of lease rebalancing

1:47:47 and redistribution that can put strain on your lease container,

1:47:51 which typically is provisioned with a relatively low number of RUs.

1:47:54 Monitoring these things will ensure

1:47:56 that your change feed application is healthy.

1:47:59 Digging deeper into the leases, you can look at the owner distribution.

1:48:03 Are your leases evenly distributed across instances?

1:48:06 Is the last checkpoint increasing for each of them,

1:48:09 or is one of your leases stalled with processing?

1:48:13 For setting up the monitoring itself

1:48:15 you can use application insights or OpenTelemetry,

1:48:17 which is built directly into the SDK to make it really

1:48:21 easy for you to export this telemetry for your change feed handlers.

1:48:26 For alerting.

1:48:26 Some things that you might consider creating

1:48:28 alerts for is sustained lag growth or if

1:48:31 there's any partitions that have stalled

1:48:32 and are no longer making progress in processing.

1:48:36 We can look at a couple key failure

1:48:38 modes and Some resilience patterns to help combat these.

1:48:42 The first is as Changes arrive, they're processed in batches.

1:48:46 When everything is going smoothly in the happy path,

1:48:50 your process handler succeeds and your lease checkpoints are all updated.

1:48:55 However, because these changes are processed in a batch,

1:48:58 what if there's an error?

1:49:00 Unhandled exceptions can create issues in your processing.

1:49:05 While the change feed processor does have retry,

1:49:07 you don't want to infinitely retry these poison messages

1:49:10 that are not able to be handled after multiple attempts,

1:49:14 and instead consider writing them to something like a dead letter queue.

1:49:17 This will allow you to write the item

1:49:20 that is failing to take a look at it later,

1:49:23 instead of blocking the entire processing for that container for that partition,

1:49:27 because your partition would not be able to continue

1:49:29 making progress if you're getting a consistent error.

1:49:34 There may be other causes of silent stalls for a given partition,

1:49:39 and one of the common causes is outgoing calls.

1:49:42 So if you're making calls to an outgoing service and they're failing,

1:49:45 this can also cause errors in your processor that will stall your change feed.

1:49:51 You can implement the circuit breaker pattern to ensure

1:49:54 that you don't flood this downstream service and inhibit,

1:49:57 your ability to actually recover from these errors.

1:50:01 A lot of the key resilience patterns

1:50:03 for any application also apply to change feed,

1:50:05 and it's important to keep these in mind

1:50:07 for the most resilient application and processing.

1:50:11 Now, we also talked a little bit about RU throttling.

1:50:15 For consistent RU throttling,

1:50:16 you can consider implementing priority based execution

1:50:20 to ensure that your mainline transactions on your container,

1:50:25 maybe your writes and your reads,

1:50:27 perhaps they have a higher priority than your change feed processor.

1:50:31 With priority based execution, you can specify that in the client.

1:50:35 To ensure that the change feed is not

1:50:37 accidentally consuming extra RUs that you'd maybe prefer,

1:50:41 go to the rest of your application.

1:50:44 There's also throughput buckets that will

1:50:46 help you indicate the percentage of RUs that you want to distribute across

1:50:50 your various applications and including your change feed.

1:50:55 Now, while you may see ru, throttles across your entire container,

1:50:59 it's also important to consider hot partitions because the change feed

1:51:04 works at the partition level and your progress is per partition.

1:51:08 If you have a surge of writes that is generating a hot partition,

1:51:12 it may be difficult for your change feed to keep up.

1:51:15 Ensure that your leases are appropriately balanced across partitions,

1:51:19 across instances, so that if there is a hot partition,

1:51:23 it doesn't overload one of your instances

1:51:25 and result in higher lag across the board.

1:51:31 All right, let's take a look at a demo to see how the Change

1:51:35 Feed processor is actually configured and see

1:51:37 it in a real application application,

1:51:38 I'm going to pull up VS code with a simple app here.

1:51:42 I've got a social media app simulator which is really just two console apps.

1:51:48 My first console app is simulating users creating,

1:51:51 editing and deleting some of their posts.

1:51:54 And the program that we've got on screen here is

1:51:56 a second console app that has our Change Feed processor.

1:52:00 Now, because we know that we've got updates and deletes

1:52:02 and I want to react to all of them,

1:52:05 I'm going to use the Change Feed processor with all versions and deletes mode.

1:52:09 When I configure my Change Feed processor, I can choose a processor name.

1:52:14 This tells me the unique processor instance and will allow me

1:52:19 to coordinate multiple compute instances all to the same processor application.

1:52:23 This is really critical for least rebalancing

1:52:25 because if you're using different processor names,

1:52:28 the change feed will assume that it's four actual different,

1:52:33 change feed instances and different purposes.

1:52:37 So we've got the processor name which sort of ties all

1:52:39 of our instances together and we also have our instance name.

1:52:43 We'll take a look in the console app how the instance name allows

1:52:46 us to tie multiple different processes

1:52:48 of these together into that same processor.

1:52:52 I'm using the same lease container for all

1:52:54 of these, which is storing all those checkpoints.

1:52:57 And I'm also setting up a couple of notifications.

1:53:00 These lifecycle notifications are really important

1:53:03 for debugging the lease ownership in your application.

1:53:06 And you can get notifications for lease acquired and lease released,

1:53:11 as well as printing out some, error messages that may occur.

1:53:15 So this is the Change Feed processor itself.

1:53:18 But let's also take a look at the delegate.

1:53:20 Once we get a batch of changes,

1:53:23 this is the code that will execute and actually process those changes.

1:53:27 So first we'll simulate some extra processing with a short delay.

1:53:33 And of course, in your real application

1:53:34 this is whatever business logic you may have.

1:53:37 Then we can process our changes

1:53:40 slightly differently depending on the operation type.

1:53:42 The first thing we want to check for is any deletes.

1:53:45 The ID and partition key of deleted items

1:53:47 will come through in the metadata so we

1:53:50 can pull out those key pieces of information

1:53:52 to be used later for creates and replace operations.

1:53:56 We can directly get the ID and the user id,

1:53:59 which is our partition key from, the current aspect of this change.

1:54:06 Scrolling down a little bit further,

1:54:08 this change feed is writing to a notifications container

1:54:13 which then can power notifications for our social media app.

1:54:17 So now that we Looked at the code, let's see it running in action.

1:54:21 And I've got three separate terminals pulled up here.

1:54:24 The first thing I'm going to do is start running my social simulator.

1:54:27 This console app is going to be creating

1:54:30 items and posts from users in our application.

1:54:33 Then I'm going to run my notification processor.

1:54:36 Now notice I'm not submitting any arguments here,

1:54:39 it's just that basic, code that we talked about.

1:54:43 And it will have a default instance name which we'll see spin

1:54:47 up here in just a second as, the detailed stats come online.

1:54:52 Now in the meantime,

1:54:53 I'm going to start a stream of posts in my application and let's make sure

1:54:58 our processor is printing in verbose mode so

1:55:00 we can see all those creates coming in.

1:55:02 But we're using all versions and delete.

1:55:04 So let's simulate some delete events and some edit events and you

1:55:08 can see all of those are actually flowing through into our processor instance.

1:55:14 All right, let's turn it back into quiet mode so we don't have every single,

1:55:18 change, but just the batches of changes that are coming in.

1:55:21 And let's simulate a couple of viral bursts.

1:55:25 There's a lot of buzz on our social app and users are making

1:55:29 more posts than ever as we see these viral bursts start to flow through.

1:55:34 We see the lag is steadily increasing on our change feed processor instance.

1:55:40 So we can create a second instance to help us,

1:55:43 manage all of the changes that are coming through for all

1:55:46 of these partitions and balance it a little bit better across both of these.

1:55:51 So I'm going to print out the detailed stats for our second instance.

1:55:57 And you notice that this one is named Instance 2 as it starts up.

1:56:02 We start by acquiring Elise and we get an error on our initial prefacer.

1:56:07 Now this is really important because this is not actually a bad error.

1:56:12 This error is just telling us, hey, I lost a lease and it went to someone else.

1:56:16 This is the behavior that we actually

1:56:18 want change Feed processor is automatically rebalancing

1:56:21 leases for us so that we can

1:56:23 ensure they're evenly distributed across our processors.

1:56:26 If we print out the leases that are actually owned by each of these, we can see,

1:56:31 our first instance owns three of our leases and our second instance owns two.

1:56:36 This is great because this means that our processors

1:56:39 are working properly and our load is evenly distributed.

1:56:43 Now I'm going to go ahead and shut back down our second

1:56:46 instance and we should see these leases

1:56:49 redistribute back onto our initial processor.

1:56:52 We see those lease acquired messages now and when we print the leases

1:56:56 owned this time we see all five leases are back onto our main processor.

1:57:01 Let's go ahead and shut down this application

1:57:04 and we see how easily Change Feed allows us

1:57:08 to spin up new instances and dynamically react

1:57:11 to the volume of processes and changes coming into our container.

1:57:18 Flipping back into slides now that we've shown a basic Change Feed app,

1:57:22 let's take a look at some advanced patterns.

1:57:25 We can combine the event sourcing CQRs and materialized views

1:57:29 pattern and to create a really powerful application in Cosmos DB.

1:57:33 While our demo showed creates updates and deletes

1:57:37 inline to items with the event sourcing pattern,

1:57:41 every write, every create, edit,

1:57:43 delete actually becomes its own independent create event.

1:57:46 So it's a depend only event store.

1:57:49 We can write these commands into our event

1:57:52 store still stored in Cosmos DB as container and we can use Change Feed to read

1:57:57 this ordered log and populate some downstream systems.

1:58:01 These are also known as materialized views and it helps us craft a view

1:58:06 of our data that is better suited for the read patterns of my application.

1:58:13 In my app I've got a couple of read patterns that I want to support,

1:58:16 like show me my notifications.

1:58:18 This is that notifications container that we

1:58:20 were just populating in our example.

1:58:22 But let's say we also want to get posts by topic.

1:58:26 Or maybe we want to create a search index container.

1:58:29 We can use built in full text search with Cosmos DB

1:58:33 to create a second container that has full text policies on our content.

1:58:39 This allows our users to really easily search across those data patterns too.

1:58:45 With Materialized Views pattern.

1:58:48 Setting up all of these separate containers based

1:58:50 on the read pattern that we're trying to optimize really

1:58:53 gives us the best configuration for Cosmos to give

1:58:57 us these efficient reads without impacting our write path.

1:59:01 Our write path is still optimized for the writes.

1:59:05 Now the Materialized Views pattern is really common in many databases,

1:59:09 but I do want to highlight a Cosmos

1:59:11 DB feature that makes this pattern really easy.

1:59:14 We have a feature called Global Secondary Indexes which

1:59:17 are effectively managed materialized views directly into Cosmos DB.

1:59:22 So while I just showed you how you can

1:59:24 kind of build this yourself with the Change feed.

1:59:27 Reading the change feed of social events,

1:59:29 populating a new container with a new partition key,

1:59:32 you can add a global secondary index.

1:59:35 This will do the exact same thing for you,

1:59:37 but it will automatically maintain the secondary container without you

1:59:41 having to write the code to build and manage changepie yourself.

1:59:45 This is a really powerful container,

1:59:48 this is a really powerful feature that allows you

1:59:50 to build containers specialized to the read patterns that you have.

1:59:54 My container, I had five physical partitions in my source.

1:59:57 If I were to use my source container to serve this query by topic,

2:00:01 it would be really inefficient,

2:00:03 it would be slow because I'm checking every physical

2:00:05 partition and it also would cost a lot of RUs.

2:00:09 With GSI I'm able to automatically create this and I

2:00:13 don't have to manage the change feed myself.

2:00:16 So let's take a look at the Azure portal

2:00:18 and we can see how this is actually implemented.

2:00:22 I have my Change Feed application,

2:00:25 my Cosmos DB here that was powering that Change Feed app we

2:00:28 just looked at and you can see my source container is social events.

2:00:34 Now my source container is partitioned on user ID and we

2:00:37 see all of the posts that my users are making here.

2:00:41 But I added a GSI already.

2:00:43 Let's take a look at our GSI and we can see this has

2:00:47 a persistent copy of all of the items in our source container,

2:00:50 but now it's partitioned by topic

2:00:52 and this is automatically maintained by the Change feed.

2:00:56 We ensure any writes only go to social events.

2:00:59 We don't need to handle any dual write or error handling.

2:01:01 That's all given to us by the platform

2:01:03 and we're able to have this read optimized view.

2:01:07 Now let's take a look at a query.

2:01:10 I want to get the top 100 items where the topic is Cosmos Conf,

2:01:15 and this first query is against my social events container.

2:01:18 If I look at my query stats I can see this costs almost 27 RUs.

2:01:23 So that's quite a bit of RUs to get these topics.

2:01:27 But if I issue that exact same query against my global secondary index,

2:01:32 flip over to query stats.

2:01:34 I see this costs just about five and a half RUs.

2:01:36 So this gives us an roughly 80% savings.

2:01:40 All because we're using the GSI which

2:01:42 is partitioned and configured for our read pattern.

2:01:50 This demo really shows us how you can use Change Feed

2:01:55 and manage features to give you

2:01:57 the best configuration for your specific application.

2:02:01 All right, we covered a lot today, so the key takeaways are that Change Feed

2:02:06 really powers a lot of event driven applications.

2:02:09 There's many architecture patterns that help

2:02:11 you build these apps for your business.

2:02:13 Use case.

2:02:15 When you're using the Change Feed, consider the Change Feed mode that is best

2:02:19 suited for your application and also the consumption model.

2:02:22 Whether that's Change Feed, processor Azure functions or the pull model,

2:02:26 it's important to invest in monitoring and you can use

2:02:30 the estimated lag to ensure that your processors are Keeping up.

2:02:34 Keep resiliency best practices in mind in your handlers.

2:02:38 Ensure that they're idempotent as retries are happening and their resiliency

2:02:43 is all the best practices with circuit breaker retry timeouts, et cetera.

2:02:49 Advanced features like Global secondary Index really help

2:02:52 you simplify these patterns and let the platform

2:02:54 do the heavy lifting instead of you

2:02:56 needing to manage that Change Feed application yourself.

2:02:59 Now you can get started and learn even more about Change Feed at this link,

2:03:03 aka MsAzureCosmos DB ChangeBead.

2:03:07 Thank you so much and I hope you enjoy the rest of the conference.

2:03:13 Okay.

2:03:14 Hello everybody.

2:03:15 So, this is MultiCloudDB.

2:03:17 Write once, run anywhere.

2:03:19 I'm Theo van Kraay, I'm a PM in the Cosmos DB team.

2:03:23 I work on SDKs, connectors,

2:03:26 developer experience and lots of fun stuff like that.

2:03:29 But I'm going to be doing something very different this time.

2:03:32 Probably the first time we've done anything like this, on Cosmos DB Conf.

2:03:37 I'm going to be talking about something that we're calling MultiCloudDB.

2:03:41 This is a unified data access layer for best of breed cloud databases.

2:03:47 So for the first time ever we're talking about databases other than Cosmos DB.

2:03:52 But before I go into that, before I talk about what that does,

2:03:55 I want to set some context here.

2:03:58 So why teams still want cloud native databases?

2:04:03 Of course, databases like Azure, Cosmos DB, Amazon,

2:04:07 DynamoDB, Google Spanner, are still very highly sought after.

2:04:11 They offer high availability and elasticity built

2:04:14 from the ground up in those platforms.

2:04:16 These platforms are built to exploit cloud core,

2:04:22 properties in those environments.

2:04:24 They have strong operational guarantees and SLAs and manage scaling.

2:04:28 It's simply not possible to get this same kind of performance

2:04:32 if you're installing a traditional database software and putting that onto

2:04:37 VMs for obvious reasons and of course you have a fast

2:04:39 path to production without owning the database or the control planes.

2:04:44 Everything is very easy, it's very elastic,

2:04:46 it's very scalable, it's very highly available.

2:04:50 The benefits there are pretty obvious.

2:04:52 But teams tend to want managed database upside,

2:04:55 without the irreversible coupling that you tend to get.

2:04:59 There's a tension here.

2:05:01 The deeper that you adopt native SDKs of these types,

2:05:04 of databases and query models, the harder it becomes to move anywhere else.

2:05:13 So this becomes something that is concerning for customers,

2:05:17 if they want to exit the architecture or move into another cloud

2:05:22 and Portability concerns show up pretty much before any line of code is written.

2:05:27 And so sadly for us working on Cosmos

2:05:29 DB&M for our counterparts in Amazon and Google,

2:05:33 often these very good database services get excluded

2:05:37 right off the bat because there's no portability,

2:05:40 they only run in Azure or Amazon or Google etc.

2:05:45 And so the cloud native benefits are obviously significant,

2:05:48 everybody knows that, but so is architecture level lock in and risk.

2:05:52 And this is very real and we are obviously recognizing this.

2:05:56 So the customer reality is that customers do want

2:06:00 the benefits of cloud native databases as we've said,

2:06:02 but they also need a story for portability,

2:06:05 procurement and regional strategy and so on.

2:06:08 This is the gap that we're trying to fill with MultiCloudDB,

2:06:11 and it's designed to close.

2:06:13 So portability is no longer optional for teams

2:06:17 who are building these types of applications,

2:06:20 operating across regions, customers and different clouds and so on.

2:06:23 We see two different types of multi cloud

2:06:27 portability pressure if we can put it that way.

2:06:29 The one is cross cloud deployment.

2:06:31 So this is where you have the same solution delivered into multiple clouds,

2:06:35 typically white labeled products,

2:06:37 sovereign deployments or maybe partner managed environments,

2:06:40 where the application needs to behave consistently even when

2:06:44 the backing database changes by the customer or region, et cetera.

2:06:49 And then the other one maybe is more familiar is where you

2:06:53 want to deploy into one cloud but you want a credible exit strategy.

2:06:57 In other words you want to avoid cloud vendor locking at least.

2:07:02 And portability obviously matters there as well.

2:07:05 Even if the team never actually switches cloud providers.

2:07:11 In both cases the requirement is the same.

2:07:13 Application code doesn't want to have to be

2:07:16 rewritten if you're moving into a different cloud.

2:07:19 And again sadly for us working Cosmos DB this means a great database product

2:07:23 is often excluded right out of the bat because it only runs in Azure.

2:07:32 So there are some abstractions out there in the Java world you have things

2:07:37 like Spring Data and Hibernate of course

2:07:39 and the Net world you have Entity Framework,

2:07:41 Core and Python you have things like django etc.

2:07:46 That offer something close to this.

2:07:47 So you'll get something like a programming to an interface model.

2:07:51 In Spring Data you have repository abstractions and object mapping and so

2:07:55 on and you get something like a productive lowest common denominator.

2:08:00 The problem with this is that they're

2:08:04 not based strictly on portability contracts.

2:08:06 This is a side effect of developing an abstraction that is Meant to simplify

2:08:14 development on many different databases rather

2:08:17 than it being the core design goal.

2:08:20 So what you get over time is divergence still appearing.

2:08:24 You still get query features drifting by backend annotations and mappings code,

2:08:29 level changes that you need to apply even though

2:08:32 more or less things tend to be very similar.

2:08:35 So portability is sort of possible in part but it's not guaranteed by design.

2:08:39 And in practice this shows up when you

2:08:42 want to migrate from one cloud to another.

2:08:43 You still have in some cases almost as many changes if not quite as many.

2:08:49 And divergent still accumulates because as I

2:08:51 said this is not the core design goal.

2:08:55 It's a side effect of some principles to create an abstraction that you

2:08:59 get some level of portability but it's

2:09:01 not really completely portable in that sense.

2:09:06 So why MultiCloudDB and what we've built

2:09:08 here and what we are building is different.

2:09:11 We have a strict portability design goal from the ground up.

2:09:15 This is not a side effect capabilities explicitly

2:09:18 surface what is and what is not portable.

2:09:21 The promise that we're making is write once,

2:09:24 run anywhere at semantics MultiCloudDB

2:09:28 treats portability as a product guarantee,

2:09:31 not a best effort convenience that you get

2:09:34 as a side effect of some goals of abstraction and whatnot.

2:09:39 So again design goal strict portability same code

2:09:43 and query code should run unchanged across Cosmos DB,

2:09:47 DynamoDB and Google Spanner should be portable by default.

2:09:50 Transparent limits.

2:09:52 We will be providing some native escape hatches

2:09:54 but we don't expect customers to be using this.

2:09:58 The whole point of this abstraction is that you shouldn't be using those because

2:10:03 that will obviously break portability and that's

2:10:05 what you want if you're in this space.

2:10:08 So simple architecture view here you have a Java application

2:10:11 that's going to be connecting to multi CloudDB client and then

2:10:13 you have a service loaded discovery that's going to load

2:10:16 the appropriate module that is being driven by config files only.

2:10:21 And this is set very similar to what you get out of things like Spring,

2:10:24 and Hibernate and so on.

2:10:26 But again as we've said this is portability from the ground up and strict

2:10:30 guarantees around not only the feature set

2:10:34 but the behavior of those features as well.

2:10:37 And it follows from that we need a portable query dsl.

2:10:40 So you can see on the left there this is MultiCloudDB query dsl the same

2:10:45 regardless of the platform that's running on it

2:10:47 would be different on the right hand side,

2:10:49 syntax differences between those different databases and so on.

2:10:53 It also follows that we have a fairly conservative feature roadmap.

2:10:58 Bottom line is example a feature will not

2:11:03 be supported if all providers cannot support it.

2:11:06 So if it can't be supported in every

2:11:08 available provider then it's simply not shipped.

2:11:13 So you might be wondering at this point why

2:11:15 would we be crazy enough to do something like this?

2:11:20 And the answer is of course we've done this before.

2:11:22 Those of you who are Familiar with Cosmos DB,

2:11:24 Azure Cosmos DB was the first database platform

2:11:27 to build broad compatibility APIs at this level.

2:11:31 Those of you who've used MongoDB API, Cassandra API,

2:11:34 Table API, Gremlin API and so on, these were compatibility APIs,

2:11:40 APIs sitting on top of Cosmos DB,

2:11:42 providing the surface area of a different database along on the same platform.

2:11:47 This gives us years of real world experiences handling compatibility drift,

2:11:52 behavior gaps, edge case semantics, et cetera.

2:11:55 We've learned that how portability fails in practice

2:11:58 where feature gaps create friction and so on.

2:12:01 How to set clear contracts and manage the relevant trade offs et cetera.

2:12:07 Why this matters.

2:12:09 Well this is not our first rodeo as my American

2:12:12 colleagues would say we're applying hardened compatibility discipline

2:12:16 to multi cloud DB from day one and we

2:12:20 actually think this is the easier problem of the two.

2:12:24 Compatibility APIs are hard.

2:12:26 Fitting a database engine to a programmability surface

2:12:30 area that we don't control is well difficult.

2:12:34 Let's say surface area keeps expanding,

2:12:37 behavior contracts keep shifting, goalposts move continuously.

2:12:41 It's very difficult.

2:12:43 Now I'm not going to say that multi clouddb is easy

2:12:45 but certainly from our perspective given

2:12:47 our experience it's a more tractable problem.

2:12:51 We can constrain the programmability layer that we control across

2:12:56 a focused set of databases and therefore we define the contract,

2:13:02 the portability of the contract.

2:13:03 We scope that to known database targets and we

2:13:06 can enforce limits explicitly so we have more control.

2:13:09 Even though there are challenges, will be challenges around supportability.

2:13:14 It's certainly a more tractable problem than the one that we

2:13:17 have already solved in the past with Cosmos DB nonetheless.

2:13:23 So of course where this fits best,

2:13:27 of course if your most important things that you want to solve

2:13:31 that you want the benefits of cloud native databases, the elasticity,

2:13:35 the high availability guarantees,

2:13:38 the SLAs and so on the managed service aspect but you also

2:13:42 want Portability then this is going to be a great fit for you.

2:13:46 Of course there are trade offs to accept as there are everywhere.

2:13:49 Strict portability means some trade offs.

2:13:52 Of course you are going to get access to more granular features if you're

2:13:56 using the native SDKs for Cosmos DB and DynamoDB and Google Spanner and so on.

2:14:03 So bottom line you want to be choosing

2:14:06 this when portability and the benefits of a cloud

2:14:09 native database are really the higher order

2:14:11 bit for your architecture and for your strategy.

2:14:15 So then again ultimately the key takeaways,

2:14:18 managed cloud databases are still very,

2:14:20 very attractive to many customers for some very good high availability,

2:14:25 elasticity, performance maintenance, operational guarantees.

2:14:28 The list goes on.

2:14:29 The blocker isn't really the database capability, far from it.

2:14:34 It's of course the application level coupling to one provider's SDK.

2:14:38 It's lock in.

2:14:39 We don't like to say the words lock

2:14:41 in but that's really what we're talking about here.

2:14:43 And MultiCloudDB is designed to blow that open by having

2:14:50 a strict portability design from the ground up so teams can

2:14:54 keep an exit strategy and deploying cross cloud scenarios without giving

2:14:59 up the massive benefits of modern managed cloud databases like Azure,

2:15:04 Cosmos DB and our counterparts in Amazon and Google and so on.

2:15:10 So before I do a demo I just want

2:15:11 to mention we do have some documentation and you'll see there,

2:15:14 it says at the bottom there in small print.

2:15:16 The SDK is currently available as a public preview.

2:15:19 It is not yet fully ready for production use.

2:15:22 Expect breaking changes, incomplete features, limited support during this phase.

2:15:28 But we definitely will encourage you to try this out,

2:15:31 raise issues in GitHub, ask questions, make feature requests, et cetera.

2:15:36 We want to know how you want to use this, what this is solving for you.

2:15:39 We're definitely committed to it.

2:15:42 So yeah, check out the documentation.

2:15:43 There's a nice getting started guide, a developer guide as well.

2:15:47 It's fully comprehensive with everything that we have so far

2:15:50 and obviously we're going to be

2:15:51 developing on this, information about compatibility,

2:15:53 a couple of samples that you can get

2:15:55 your teeth into, an API reference and so on.

2:16:00 All right, so without further to do, let me set up my demo here.

2:16:05 Okay, so here we are.

2:16:06 In VS code I have an application called Risk Platform App.

2:16:11 It's a multi tenant app that is giving a dashboard

2:16:14 of different applications that are providing risk management services and so on.

2:16:21 And this is built on Multi Cloud DB and you

2:16:24 can See the multi cloud client is being used here,

2:16:28 and that is going to do a bunch of things and one

2:16:31 of the things it's going to do is load a config file.

2:16:34 And again I mentioned earlier,

2:16:35 the configs are the only thing that are different per provider.

2:16:39 So you've got one for Cosmos DB here

2:16:41 and one for DynamoDB and everything else is identical,

2:16:45 the code is the same regardless of which provider you're pointing to.

2:16:48 You don't need to change it.

2:16:49 You can treat it as basically the same database

2:16:52 and we guarantee that the behaviors and the functionality

2:16:55 that is available to you should be the same

2:16:58 and performance differences within some

2:17:00 acceptable limits of a statistical difference.

2:17:04 So let me just start up my Cosmos DB,

2:17:09 version here with the Cosmos DB properties that I have.

2:17:12 So let's go ahead and start this one up.

2:17:17 Okay, so that's started up.

2:17:18 So let me go into this app here and we can see I've just put

2:17:23 a little moniker there just based on the config

2:17:25 file to denote which database it's pointed to.

2:17:27 But everything else should be identical if we were going to run this in Amazon.

2:17:33 And so I can see here I've got different portfolios,

2:17:36 I click on these, I'm going to see

2:17:39 a different list of positions on those portfolios.

2:17:41 So just to prove it let me go back here and let me cancel

2:17:45 this app and let me start it up

2:17:49 again but use the Dynamodb properties files instead.

2:17:57 Okay, so that started up, now you can see there provider, DynamoDB.

2:18:01 Instead of just clicking on the link I'm just

2:18:03 gonna go straight back to my original link here

2:18:05 and I should able to refresh or hit that URL

2:18:08 and you see it's just now pointing to Dynamodb.

2:18:11 But everything else is exactly the same.

2:18:14 The data is the same and all the behaviors should be identical.

2:18:20 And if we were to look in the portal for Cosmos DB,

2:18:23 now of course this is where things may look different.

2:18:25 So you'll have databases and containers in Cosmos

2:18:28 DB for those of you familiar with Cosmos DB.

2:18:31 In Dynamo of course you have tables but you don't have databases.

2:18:35 So we prefix this with the database

2:18:37 name in order to replicate the resource hierarchy.

2:18:40 And so that's from the SDK perspective that's what it's abstracting.

2:18:43 But of course the data is ultimately identical

2:18:45 and most importantly of course how the SDK interacts

2:18:48 with the data and the results that come

2:18:50 back and the behaviors should be identical as well.

2:18:53 And ultimately that's what you will care about

2:18:55 if you want to maintain a single playbook,

2:18:57 a single set of application code for your solutions

2:19:00 that are being deployed into different cloud environments,

2:19:02 or if you want to make sure that you

2:19:05 have an easy migration and exit strategy from one,

2:19:08 cloud environment to the other.

2:19:12 So that's it.

2:19:13 That's everything for MultiCloudDB again, check out the documentation,

2:19:20 and the links that we gave here earlier on.

2:19:23 Thanks for watching.

2:19:30 Hey everybody.

2:19:30 My name is Andrew Ruffin.

2:19:31 I'm a senior Product Manager here at AMD and I

2:19:34 focus on launching and growing new products and services on Azure.

2:19:36 With AMD under the hood, you're here at Cosmos DB conference most likely because

2:19:41 you care about building systems that actually work at scale.

2:19:43 They're globally distributed, they're highly available,

2:19:46 and they're predictable under real workloads.

2:19:48 Azure Cosmos DB gives you all of that as a managed service,

2:19:51 and that abstraction is incredibly powerful, especially as a managed service.

2:19:55 This all comes by design.

2:19:57 They come from real engineering decisions, joint enablement,

2:20:00 and the deployment of actual infrastructure across

2:20:02 the global scale that only Azure can provide.

2:20:05 The hardware choice isn't your concern,

2:20:06 but the physics absolutely stick around and they matter.

2:20:10 And that's where AMD comes into the picture.

2:20:12 Today, Azure Cosmos DB runs on modern AMD EPYC processors across the globe.

2:20:17 We launched AMD EPYC in the cloud on Azure nearly 10 years ago,

2:20:21 and we've continuously executed and generated new products and services

2:20:25 that give the features that you need generation over generation.

2:20:29 You'll hear more details on how Azure

2:20:31 Cosmos DB takes advantage of AMD infrastructure,

2:20:33 including the latest 5th gen processors later on.

2:20:37 While you, as an Azure Cosmos DB user, don't need to spend any cycles

2:20:40 worrying about the hardware or the infrastructure.

2:20:42 You get the benefits in terms of cost

2:20:44 savings enabled by efficient and sustainable performance across Azure,

2:20:48 you'll find benefits when you're running on top of modern EPYC processors.

2:20:52 The newest V7 generation, for example, provides up to 35% more performance

2:20:56 in performance per dollar over the prior generation.

2:20:59 With these newest generations integrated into Azure Cosmos DB,

2:21:02 you get more performance per request unit

2:21:04 and a better experience to the many dynamic scaling

2:21:07 and elasticity features that the team continues to roll

2:21:09 out and you hear more about later today.

2:21:12 This is all enabled by the close partnership

2:21:13 we at AMD have with Microsoft as a whole,

2:21:15 and it creates a feedback loop between hardware and software development.

2:21:19 This results in optimized performance per dollar

2:21:21 per watt for the most advanced workloads,

2:21:23 like the ones you run every day on Azure Cosmos DB.

2:21:26 So thank you to the Azure Cosmos DB

2:21:28 team and all the organizers of the conference.

2:21:30 We appreciate the opportunity to help put

2:21:32 this all together and make this possible.

2:21:33 Enjoy the show.

2:21:36 Welcome back and a quick thank you to Andrew Ruffin and our partners

2:21:40 at AMD for being part of the Azure Cosmos DB conf.

2:21:43 Yeah, absolutely.

2:21:45 They were a key part of how we were able to extend this show to five hours,

2:21:50 more sessions, more speakers and more depth.

2:21:53 Quick reminder, you can explore the full agenda of all those talks at AKA Ms.

2:21:59 Azurecosmos dbconf.

2:22:00 All of the talks.

2:22:02 All of them.

2:22:03 All of them.

2:22:03 They're going to be live.

2:22:04 Tell them about it.

2:22:05 Yeah, there's going to be $0.21 sessions total and every

2:22:08 session will go live before the end of today's show.

2:22:11 You can find all of today's links up in the YouTube description, a lot of links.

2:22:16 So you're going to be looking past the comments.

2:22:19 So, we're going to talk a little bit about agents AI,

2:22:22 but before we do that we've got some comments

2:22:24 from you that have to do with agents in AI.

2:22:27 So let's take a look.

2:22:28 So, one of our viewers, a very, very engaged viewer, Chenola,

2:22:34 has said I asked the chat and I'd love you to answer those questions too,

2:22:39 what your AI agent development, environment may look like.

2:22:44 They said we are using GitHub Copilot at the enterprise level.

2:22:48 We've developed our agents to run widely with Copilot CLI,

2:22:53 using it for code test DevOps.

2:22:56 That's huge.

2:22:57 It's an end to end kind of experience.

2:22:59 And then Sandra says, I am a DB admin who usually uses the Azure

2:23:05 Portal for DB management and Bicep templates for deployment.

2:23:09 So they do have a copilot with PowerShell, for doing that automation.

2:23:13 So.

2:23:13 Hey, Patty, I hear you also got

2:23:15 another question about something related to AI coding.

2:23:18 Yeah, also from St.

2:23:20 Andrew.

2:23:21 First of all, thank you so much for the great comment on the keynotes.

2:23:24 Now the question is, about the agent skills.

2:23:27 Does it only work with the extension?

2:23:29 No, you can use NPX to install it.

2:23:34 Installation and usage instructions should be in the documentation.

2:23:40 Yeah, but what do we have, Jay?

2:23:42 Absolutely.

2:23:43 And up next, when you're building with AI agents,

2:23:46 token usage isn't just a billing detail, it's a design construction.

2:23:50 And so every extra turn and chunk of context adds up.

2:23:55 So you need an approach that stays efficient as your agent scales.

2:23:59 And I know I may get access to lots and lots of tokens,

2:24:03 but not everyone else does.

2:24:05 I know efficiency really is key and that's where memory becomes critical.

2:24:11 Farah Abdou will show you how an agent memory

2:24:13 layer on Azure Cosmos DB helps you reuse Cosmos context,

2:24:16 reduce tokens and keep response quality high while improving

2:24:20 latency and keeping costs as predictable as you scale.

2:24:23 Absolutely.

2:24:24 And I love this session.

2:24:26 Farah is so great.

2:24:27 This is cutting AI agent costs with Azure Cosmos DB, the agent memory fabric.

2:24:33 Here's Farah.

2:24:36 Hello everyone, my name is Farah Abdou.

2:24:39 I'm a lead machine learning engineer and AI researcher.

2:24:43 And today I will walk you through how

2:24:45 to cut AI agent costs with Azure Cosmos DB.

2:24:50 This is a war story about building an AI system

2:24:54 that costs way too much money and how I fixed it.

2:25:00 So for the agenda today you are going to talk about the problem.

2:25:04 Why is the multi agent AI based model the solution?

2:25:08 We have one database, three features,

2:25:11 then a live demo and the real production numbers.

2:25:17 So global AI spending in 2026 will be $2.5 trillion.

2:25:25 The agent market grows from 7.84 billion to 52.62 per year.

2:25:34 This is between 2025, 2030, a compound of annual growth,

2:25:40 rate of 46.3% and 75% of large enterprises will be running AI agents by 2026.

2:25:51 And inference demand will grow about a, thousand times by 2027.

2:25:57 And this will keep that host law is not an optional thing.

2:26:04 It's how we can stay in the business.

2:26:12 So this is the architecture that everyone is building right now.

2:26:16 Store data systems, a cache, a relational database,

2:26:19 a vector database and event pass.

2:26:22 We have four terms for this, four filler points,

2:26:25 we have dual coordination between them.

2:26:28 And this is exactly what I built first.

2:26:30 It worked, but it didn't work after this.

2:26:39 So what about the production?

2:26:43 So in the demo, first of all we have three agents,

2:26:48 we have 100 requests, it's about six donors.

2:26:53 Then we come to production.

2:26:54 In production we have now three agents,

2:26:57 still our agents, but now we have 10,000 requests.

2:27:03 This will cost us $18,000 a month.

2:27:06 Nothing changed about the design, we just got real traffic.

2:27:15 So we have five filler modes.

2:27:17 I have seen with my own eyes, number one is about the cost exclusion.

2:27:23 We have from 5 to $15 in a demo becomes 18,000 to $90,000 a month in production.

2:27:33 Number two is the reliability collapse.

2:27:36 So 95% per agent multiplied across three agents equals for example like 85.7%.

2:27:45 This is 14:30 every day at 10,000, huge number.

2:27:55 So number three is about the latency cascade.

2:27:59 For example agent one let's say take three seconds.

2:28:02 Agent B take about four seconds.

2:28:05 Agent C take about five seconds.

2:28:09 There are 12 seconds total plus 200 to 500 millisecond coordination between

2:28:15 the handoff and for the state corruption

2:28:20 like parel rights with no concurrency control.

2:28:24 It means silent data and 5 is the context degradation.

2:28:32 Agency gets only 60% of agents a original entity after 47 steps.

2:28:45 So what is the reliability mass that kills you?

2:28:49 One agent at 95%, 500 filler a day at 10,000 rubles.

2:28:59 Three agents at 85.7%.

2:29:03 1,430 filler per day.

2:29:08 We have five agents at 77% 2,300 Philip and they're gonna

2:29:16 say 40% or more of agent project will fail by 2027.

2:29:22 So look at this math.

2:29:23 I think huge numbers the little Pill now

2:29:30 that nobody plants are Winterbeat analyzed 100,000 production queries.

2:29:38 18% were exact duplicates.

2:29:42 47% were semantically similar.

2:29:45 Same meaning, different words.

2:29:48 Only 35% were genuine new questions.

2:29:51 When I run the same analysis on my own system,

2:29:55 about 65% of my LLM Spent was answering question I already answered.

2:30:05 So the breaking point is $47,000 a month.

2:30:11 40% of the sprint time debugging agent fillers.

2:30:15 Multi agent debugging takes three to five

2:30:19 times longer than the single agent debugging.

2:30:22 And only 11% of the organizations have AI agents in production at all.

2:30:29 We were in that 11% but it didn't feel like success.

2:30:34 Something now had to change and this is why we are here today.

2:30:41 We have one database, three capabilities.

2:30:44 The agent memory subgroup which is built entirely

2:30:49 ON Azure Cosmos DB for NoSQL this replaces four systems.

2:30:54 Cut the cost 73%, cut the latency 65% and let

2:31:00 me show you now how So why Azure Cosmos?

2:31:07 Because we have four properties that make

2:31:11 it uniquely suited for the multi agent footprint.

2:31:14 First, the native vector search using DiskANN sub 20 milliseconds

2:31:20 at 10 million vectors under 100 milliseconds at one period.

2:31:26 The second is a change feed like

2:31:29 the built in event streaming no message queue needs.

2:31:33 And the third is the ETag optimistic concurrency at the document

2:31:39 level no distributed op manager is here and the global

2:31:44 distribution so multi region 99.999% SLA single digit multi second point

2:31:54 triads all in one SDK one string connection all in one.

2:32:04 Now we know that a traditional cache matches exact strings 18% hit rate.

2:32:15 A semantic cache matches meaning using the vector similarity 65% hit age.

2:32:25 The extra 47% is the money you are leaving with a table evidence the query

2:32:32 on the screen does it in one SQL call vector distance with a threshold of 0.80.

2:32:43 If the incoming question scores at or above that returns a cache answer.

2:32:48 No LLM.

2:32:53 Now for the embedding pipeline we have four components.

2:32:56 An embedding model which will be a text

2:33:00 embedding.3 small by the Azure OpenAI 1536 dimensions.

2:33:07 Then we have a Cosmos DB container with a distance in vector index DTL

2:33:14 item from 32nd to 7 days partition key which will fill the query hash.

2:33:20 And we have the vector index configuration which where we set the indexing

2:33:26 search list size between 10 and um.500 to tune the accuracy versus build speed.

2:33:33 And then the lookup query which will be the select top

2:33:37 one with the vector distance greater than or equal to 0.80.

2:33:42 Below that call the LLM And store the new entry.

2:33:50 So now let's have a look how this works.

2:33:54 Like.

2:33:59 So we have here, this is the entire cache lookup one SQL query.

2:34:13 We have the vector distance built into the Cosmos DB.

2:34:19 No extra library, no separate vector database.

2:34:24 Then I have some agents to try on.

2:34:28 This is the first one with three questions and then we have a next

2:34:32 run with another three question to see the cache hit and the cache.

2:34:39 So if I run the demo, we have here three pilots on the screen.

2:34:47 Semantic cache change feed, optimistic concurrency, all in one database.

2:34:52 The first agent as you can see here, like for example it have a support agent

2:35:00 with what is the refund policy for electronics?

2:35:03 It have a cache miss.

2:35:05 It doesn't find this before.

2:35:07 So now we have the answer.

2:35:09 The second agent, the return agent.

2:35:12 How can I return an electronic product?

2:35:14 The third agent called the product itself or anti on laptops.

2:35:20 So now we have missed right now what if I just Change the question again?

2:35:29 So I will just comment this uncomment the second

2:35:34 one or the second run it has similar questions,

2:35:38 similar meanings but not exact same writing.

2:35:43 So here for example what is the electronics return undefined policy

2:35:47 Then join item I put Then the warranty comes with that.

2:35:52 So let's run this one more time.

2:36:03 So as you can see we have here a cache

2:36:06 hit similar to 0.89 which is more than that 0.80.

2:36:13 We have got it in the slide.

2:36:16 The second one is also a cache hit similar to 0.91.

2:36:21 The third is also cache hit with 0.92 so

2:36:25 it finds all of this info on the database.

2:36:28 It doesn't need to call the LLM again and cost me money.

2:36:33 So how much it saved you?

2:36:35 It saved me.

2:36:36 Now just 327.

2:36:39 Okay, let's go back to our slides.

2:36:46 So what you just saw was three agents, three different questions, zero impulse,

2:36:52 all answers compact in about one second I

2:36:58 say what is electronics return and refund policy?

2:37:01 And I found it matches the refund policy for electronics.

2:37:04 For example at 0.89 similarity a traditional

2:37:08 cache would have made a semantic cache.

2:37:12 Catch it every time.

2:37:16 So the production results and this is the result

2:37:20 from a real production deployment which is constant with published benchmarks.

2:37:28 Before $47,000 a month,

2:37:32 803.50millisecond average latency with 10% cash hit rate.

2:37:39 After that I have $12,700 a month.

2:37:45 These 100millisecond average three months,

2:37:49 300millisecond average latency and 65% cache hit rate and 73% cost reduction,

2:38:01 65% latency improvements, one architecture.

2:38:06 Change it all.

2:38:10 So what about the disk?

2:38:11 And at scale, let's see.

2:38:14 So under 20 milliseconds at 10 million vector was greater than 90% recall.

2:38:24 We have under 100 millisecond at 1 billion vectors.

2:38:30 This is still greater than 90% recall.

2:38:34 70iu per query let down a complex SQL query and when

2:38:40 the index grow 100 times the latency increases less than 2 times.

2:38:50 And so the change feed we have two ways,