Beyond Vector Search: What RAG Actually Needs | Yohan Lasorsa | Azure Cosmos DB Conf 2026

0:08 Hey folks, my name is Yohan Lasorsa and I'm

0:11 a Principal Developer Advocate in the data team at Microsoft.

0:14 Welcome to this session where we'll talk

0:17 about rag pipelines optimization among other things.

0:20 Let's talk about agents.

0:22 There are many potential agents that you

0:24 may be building for your applications and most of those agents need the ability

0:30 to find relevant information to complete their task.

0:33 For example, let's consider the agent

0:36 used by Contoso Burgers fictional restaurant.

0:39 We have customers that talk to the agent

0:42 when they want to buy something or consult

0:45 the status of their order and that agent

0:49 needs to find products or specific order,

0:51 most likely from a database, which in our case is Cosmos DB.

0:57 The data types that agent can search for can be diverse,

1:04 relational, structured, audio, images, video.

1:07 and others.

1:08 For this session today, we'll specifically focus on structured and text data

1:13 that may come from a database or documents.

1:17 In an agentic scenario like this, it means that the agent first has

1:22 to decide what tools to use

1:24 for a given request to get the contextual information.

1:27 And for each tool, we have to provide the best

1:31 way to search for the information that we need.

1:35 Let's do a quick demo of our agent that we build for this use case.

1:41 So here we have our web application for the agent.

1:45 Let's, for example, ask, what's your best spicy burger?

1:53 So you can see that the agent is thinking, retrieving some information in there.

1:58 Let's just wait a bit for the results.

2:03 So we got our answer in there.

2:06 So yeah, this looks like a good spicy option,

2:10 but what's more interesting is how it came to that result.

2:14 So let's have a quick look uh in there to see.

2:18 Yeah, we see that there are a few intermediate

2:21 steps that the agents use to get to that answer.

2:24 First, it got the list of burgers by calling a tool, then the list of toppings.

2:32 Many agents use the retrived information to do RAG,

2:36 retrival augmented generation,

2:37 in which the agent sends the information to the LLM to answer a question.

2:43 So RAG was one of the first practical use cases for LLMs.

2:48 And it's still just as relevant today in the era of agents.

2:52 In this example, RAG flow,

2:54 the user asks about the best spicy burger on the menu.

2:58 So what we did.

3:00 We go get the burger option in our database and those results

3:05 are sent to an LLM along with the original user question.

3:10 And then the LLM answers the question, citing the relevant results.

3:15 So Rack uh is a way to ground your agent's answers in a context,

3:20 in our case, the Contoso Burger restaurant.

3:23 And it's often paired with complex search pipelines,

3:26 but sometimes it doesn't need to be.

3:29 Back to our user query for spicy burgers,

3:32 it's a restaurant and the menu is limited.

3:35 In fact, there's only 10 burgers.

3:38 So we don't need a search engine for that.

3:40 We just call the web API to fetch all the burgers

3:43 on the menu and we feed it directly to the LLM.

3:46 So that's basically mostly what we did,

3:48 except uh instead of just having the LLM call the API,

3:53 we use the NMCP server to act as the middleman.

3:56 So this works.

3:58 because the data size in there is very well delimited and small.

4:02 We only have 10 burgers on the menu, so it can work because it fits in the LLM

4:08 context with ease and without using much tokens.

4:11 So, no brainer here.

4:12 We don't need anything more complex.

4:15 But now, our restaurant has recently

4:18 made a partnership with Contenso World Beers.

4:21 And we want to be able to recommend beers if the user asks for it.

4:27 So for example, we have this other user

4:29 that wants white beer to go with his spicy burger.

4:33 But this time, the catalog is more than a thousand beers.

4:37 So we need a good search pipeline.

4:39 And what we want is only send the relevant results to the LLM.

4:46 So sure, maybe some of you may say

4:50 that nowadays we have LLMs that can fit like...

4:53 million tokens also in their context window.

4:57 So it may still fit in this specific case, the whole catalog inside the context.

5:02 But even if it fits,

5:04 you would have an increased latency and huge cost to the large amount of tokens.

5:09 So it's really not efficient.

5:11 And most importantly, it doesn't scale.

5:13 If instead of 1,000 billion, we would have like 10,000 or 100,000,

5:18 it wouldn't work at all anyway.

5:24 So when users ask things to agents, there's a huge diversity in their request.

5:30 We have some queries that are short,

5:32 some that are long, some are general, some are very specific.

5:36 Some are in the language of the data and some are in different language,

5:40 the native language of the speaker.

5:42 So for example, these are four requests that users could make to the Contenso

5:47 burgers agents when asking for beer recommendation to go with their menu.

5:52 So the first user that we have just want French beer,

5:57 but a very precise alcohol by volume degree.

6:00 The second user wants a beer with no alcohol,

6:04 but that pairs well with a spicy food.

6:08 The third user asks something more vague and taste-oriented.

6:11 And the final user asks for exactly the same thing, but this time in French.

6:16 We want our agents to find great results for all of those queries.

6:21 So there are two main search strategies that you've probably heard.

6:26 Keyword search, which has existed for a while now and vector search,

6:31 which got really big the last few years thanks to better embedding the models.

6:35 Some of the queries by user can be handled better by keyword search,

6:40 while many will be handled better by vector search.

6:43 So the question is, which strategy do we use?

6:48 And we should actually use both strategies.

6:51 The optimal search strategy is a full hybrid search

6:55 stack where we first do parallel vector and keyword searches,

6:59 then merge the results together,

7:01 and finally apply a reranking step to ensure the best results are at the top.

7:06 So let's have a look at each of these steps in this stack to understand

7:11 fully how it works and why it's an important part of an optimal search strategy.

7:16 Let's start with keyword search.

7:19 So the Google classic keyword search.

7:21 The idea here is that we search for the terms in the query

7:25 and find the results with have those terms in the highest frequency.

7:29 So the best algorithm for that is BM25,

7:33 a probabilistic ranking algorithm that takes into account the relative

7:37 frequency of a term proportional to the length of a document.

7:42 So say otherwise, if the user searches for lights,

7:47 and the document says light, light,

7:49 light all over the place and other documents don't,

7:52 then that's probably a good result.

7:55 So let's try keyword search in our actual database.

8:00 Let's move on to the demo in there.

8:04 First, let's have a look at what we have in our database.

8:07 So here I'm using the Cosmos DB extension.

8:11 So you can see the four database that we have.

8:15 Let's have a look specifically at the...

8:17 beer database and let's have a look at just a random

8:21 item so we can see what's in the beer's data.

8:26 So you have the name, the ID, the style, the brewery,

8:32 the country, the alcohol by volume,

8:35 description and uh flavors and pairing notes.

8:39 So this is basically all the beers that we have here in our database.

8:44 You may also have noticed that we have dedicated containers named beer vectors.

8:48 So we also have a quick look in there.

8:50 Let's take this one, for example.

8:52 So you can see that we have the actual vector data,

8:57 all the numbers that represents the embedding uh of the item.

9:02 This one actually uh related that's stored in the other container.

9:07 And we have basically the text that was used to generate the embedding.

9:11 So it's just...

9:12 taking all the structure data and put in text form.

9:17 So going back to uh testing our keyword search.

9:22 So here I'm using this query, for example, to test,

9:28 uh to search for, let's say, a 5% from France.

9:34 And we'll see how it goes against our database.

9:36 So we can see the SQL query in there.

9:39 And we're using the full text score function to basically rank our results.

9:46 So let's just run it.

9:52 So let's take a look at the results.

9:54 So we wanted 5% beer from France.

9:56 And the first result, actually pretty good.

10:00 France, almost 5%.

10:02 Second one, same thing.

10:04 So yeah, top results looks good in there.

10:06 Keyboard search did its job.

10:10 But yeah, if we look below the two first results,

10:13 yeah, we have beer from Japan, Netherlands, and Japan.

10:17 So yeah.

10:18 After that, and this one, it's 0% alcohol by volume,

10:23 so not a great result in there.

10:26 yeah, keyword search work quite OK for the two top results,

10:32 but not great otherwise.

10:36 Let's try another request.

10:38 So this time, searching from something with no alcohol

10:43 that pairs with spicy food for my pregnant wife.

10:49 And yeah, this time, results are not so great using keyword search.

10:55 For example, if we look just as the main request, something with no alcohol,

11:01 we can see that pretty much all the results uh have some alcohol in there.

11:07 yeah, probably that's in that case because we didn't have the right keyword.

11:15 we didn't get any relevant results that match what we initially wanted.

11:21 So that's why keyword search is a bit hit

11:24 and miss when it comes to broader queries like this one.

11:29 Let's try another one.

11:32 We want something just light and citrusy.

11:39 So yeah, first result, it's quite light, no alcohol.

11:45 Let's look at the flavor notes.

11:49 Yeah, clean citrus finish.

11:51 So yeah, could be considered a good result.

11:54 Second one, less light, but still OK.

11:58 uh Let's look, uh yeah, gentle citrusy finish.

12:04 So again, quite good results.

12:07 Yeah, generally if we look at the other,

12:10 it's not so bad except maybe like the last one, not so light.

12:14 And yeah, not so much lemon in there too.

12:19 So let's finally try our final uh one,

12:23 which is basically the same light citrusy beer request, but expressed in French.

12:35 And yeah, this time not so much light beer and basically the results

12:41 are pretty much all irrelevant uh because our data is stored in English.

12:47 uh Making a request in French means that we don't have like the right keywords.

12:53 So that's definitely something to consider.

12:56 uh If you want your agents to be able to handle queries in multiple language,

13:01 uh you need to consider that when you're deciding

13:05 on which search strategy to use for your application.

13:11 So that's why developers were so excited when Vector Search came out,

13:16 since it can handle many of the queries that keyword search struggles with.

13:21 So this time we're using an embedding model like the Azure

13:26 OpenAI Text Embedding 3 models to create vectors from our contents.

13:30 So when a user query comes in, we embed that using

13:35 the same embedding model and use the vector similarity metric.

13:39 to find the vectors closest to that query vector.

13:43 And to make that vector uh search efficient across millions of vectors,

13:49 we can use like an approximation algorithm, like uh HNSW or disk ANN.

13:57 So let's try a vector search with our BR database.

14:01 Again, just switching from different scripts.

14:07 So we can see we have basically the same setup as before,

14:12 except this time uh we are making a vector search.

14:16 yeah, we have a first step uh transforming

14:19 using uh the OpenAI Text Embedding 3 model,

14:23 transforming the user query into a vector.

14:26 And then using that vector to basically compute the vector's

14:31 distance comparing and picking up the top five results.

14:36 So let's try again with our different queries.

14:39 So first one, 5% beer from France.

14:44 And yeah, not so great with that one.

14:49 The keyword search actually handled it better

14:52 because the first two results were from Belgium,

14:55 so not exactly France, even if they speak French there.

14:59 6% is not 5%.

15:01 This one is a bit closest.

15:03 Our first actual good result is in third place.

15:07 Yeah, after that, it's not so bad,

15:09 but the two top results not great for that one.

15:12 Keyword search, not that better.

15:14 But let's try the one that keyword search didn't handle very well,

15:20 like the alcohol free request query.

15:26 So yeah, this time, at least the top result is 0%.

15:31 And if we take a look, all the results are non-alcoholic beers.

15:37 So at least uh what we wanted was respected in there.

15:43 So does it pair well with something spicy?

15:50 Smoked salmon, maybe not the best one.

15:53 Second one is better.

15:55 Spicy food, yeah, tacos.

16:00 at least it's all better results than what we got uh and more closer

16:04 to the intent of the request compared to what we got with keyword search.

16:10 So let's try the third one that's more general request.

16:19 So I wanted something light and citrusy.

16:21 So first result, two top results not so light.

16:26 uh Citrusy, yeah.

16:29 Grapefruit, uh orange.

16:32 So yeah, can be similar to citrus taste.

16:36 Same for the second one.

16:38 Third one, yeah, quite light since it's alcohol free.

16:43 uh With, OK, something lemon.

16:46 uh So citrusy taste.

16:49 something light also for the true last results and tangerine,

16:54 grapefruits, juicy finish.

16:56 So yeah, two top results again, not the best one,

17:00 but that's because when you're doing vector search, results can be a bit noisy.

17:05 And sometimes, yeah,

17:07 you get mixed results with some good results and some not so great.

17:14 So let's try the last one that didn't work at all using our keyword search.

17:19 So light citrusy beer, but in French.

17:21 And this time, yeah, it's looking good.

17:24 First results is actually better, like light, fresh lemon.

17:28 Yeah, okay.

17:31 Can be light, subtle lemon peel leaf.

17:34 So yeah, good result this time.

17:36 It matched what we wanted,

17:38 despite our request being in French and our data being in English.

17:44 So it worked better in that case.

17:49 But yeah, you've seen that for some requests like the 5% one,

17:54 it doesn't work so well because vector embeddings don't

17:57 distinguish well between specific numbers or specific brand names.

18:01 It's the kind of thing that keyword search is great at.

18:05 Vector search is better to focus on the concepts like closing,

18:12 like the semantic matches, like, OK, I'll call fruit.

18:17 It's the same as booze-free.

18:19 uh Citrusy is basically something a bit bitter and acid.

18:25 So any fruit that match this taste might be matched using vector search.

18:34 That's why we should always use both keyword search and vector search.

18:39 And then we need to find a meaningful way to measure results.

18:42 So the most popular algorithm for result merging is called RRF,

18:48 reciprocal rank fusion.

18:49 So it sounds fancy, but it's really just a formula that computes

18:54 a score for each results based off its relative rank in each result set.

18:59 So that way, if a results uh show up high in both results,

19:04 it will be ranked highly in the final merge results.

19:07 So if a result is high on one, but low on the other,

19:11 it will end up more like in the middle.

19:14 So let's go back to our database searching using RRF.

19:22 Okay, so the setup is pretty similar as before.

19:26 You can see that we have in our SQL query,

19:29 we're saying that we want the result order by rank rrf.

19:33 So this is uh native cosmos.

19:36 We're using uh mixing the full text score, sorry.

19:41 the keyword search and the vector search in there.

19:44 We're also doing the two separate requests to get the individual rankings,

19:48 just to get an idea.

19:50 why we got this particular score at the end.

19:54 So let's first try this uh more broad request,

19:58 the one for the light citrusy beer.

20:05 First result, OK, somewhat light beer.

20:09 Let's have a look.

20:10 Gentle citrusy finish.

20:12 So good results.

20:13 You can see that it was ranked 2

20:15 in the keyword search and 14 on the vector search.

20:18 Let's have a look at the second one.

20:20 It's actually one that was better ranked

20:22 on the vector search and bit less on keyword search.

20:25 So light because no alcohol.

20:28 And lime zest.

20:31 So yeah, still good light and citrusy beer.

20:34 Let's have a look at other results.

20:37 So all light results.

20:40 So the one that were from the keyword

20:43 search were basically uh not there this time.

20:45 Lime, orange zest, again, switch orange peel.

20:51 So basically all kind of good results mixing because

20:55 the search works quite OK for both keyword and vector search.

21:00 Let's try a more difficult request that was working well for vector search,

21:04 but not so well for keyword search.

21:06 The one requesting uh no alcoholic or spicy food.

21:13 So the first results, as you can see,

21:16 it's quite in the middle for both keyword and vector search.

21:20 And yeah, it's still a non-alcoholic beer, even though it's not fully 0%.

21:26 And let's look at the pairings.

21:29 So pairs well with couscous.

21:31 So yeah, couscous is quite spicy.

21:34 So first result is quite good.

21:37 But if we look at the other results, we still don't have.

21:43 good, at least what we wanted regarding non-alcoholic beer.

21:49 And yeah, that's because uh in the end, RRF isn't the perfect solution either,

21:55 because both vector search and keyword search can yield sometimes

22:00 noisy results like this, especially keyword search in this particular case.

22:06 uh So yeah, if you have average results, In both search,

22:12 you can still surface the best result at the top with hybrid search.

22:16 So it works relatively well in most use case,

22:20 but still isn't perfect as you can see in there,

22:23 especially with more complex case.

22:28 That brings us to the final step, re-ranking.

22:31 We take the merge results and send them through

22:34 a re-ranking model whose job is to look at each result,

22:37 look at the user query and give a score

22:40 indicating how well a result matches the query.

22:43 So we typically use a special kind

22:45 of model for this called a cross encoder model,

22:48 but LLMs can also be used with the right system prompt.

22:53 A cross-encoder model is the best option.

22:55 It's actually been trained for this specific task

22:58 based off human scoring search results and saying,

23:01 yeah, this result is great for this query and this one isn't great.

23:05 So let's try this out with our beer database.

23:12 I will just do the most difficult one that wasn't working

23:16 quite so well with hybrid search after the first top result.

23:26 So here you can see that the first result,

23:30 and if actually we look at all the results,

23:34 we get non-alcoholic beers for all the results.

23:37 So uh the good thing about using a rerunker is that it

23:42 will also filter out all the results that you don't want.

23:46 And in this case, the alcoholic results.

23:49 And let's look at the pairing.

23:52 So pairs with spice.

23:54 ChickenP wraps, so yeah, good pairing.

23:56 uh Same thing for the second one.

24:00 And you can see that this way actually good results for vector search.

24:05 This one was ranked three using RF.

24:11 Not so well using keyword search.

24:14 And yeah, we still got our first top results from RF.

24:20 But this time you can see that it's only ranked third.

24:23 basically, yeah, the ranking you can see some some results move down some moved

24:28 up uh and all the one that we didn't want were basically filtered out.

24:33 So it's a really more reliable way to search for things.

24:41 Let's look at what we did in there.

24:44 So we did basically the hybrid search,

24:47 but what we added at the end, we're using an LLM.

24:52 In this case, I'm using GPT-5 mini with basic prompt saying,

24:57 okay, you'll be a recommendation expert.

25:00 Here's the user query is the list of candidates and just give rerun,

25:05 reorder the beers to get the best result at the top.

25:11 That's how I got it.

25:12 So you can still do that with LLM just as a way

25:16 to, if you already have an LLM in your pipeline,

25:19 that's a good way to get quick results.

25:27 Now you've seen an hybrid search flow on Azure Cosmos DB.

25:30 And what's really exciting is that we can do native hybrid search today.

25:35 The vector search, keyword search, and RRF are all natively supported.

25:40 And even the final rerunker step,

25:43 I've just shown you that I'm currently using an LLM

25:47 to do that, but it will soon be also natively supported.

25:51 Currently it's behind a private preview

25:54 with SDK support for Python and .NET today.

25:58 And we also have upcoming support for JavaScript and Java SDK.

26:02 You can just sign up following the link

26:05 in this slide uh if you want to get access, if you're interested in trying it.

26:10 I'll just show you a quick glimpse of what it looks like.

26:14 So as I can say, it's only available on Python and .NET for now,

26:18 so I'm using the Python SDK.

26:20 I'm using the exact same query and basically

26:24 the same setup as before with the same request,

26:28 except that this time for the rerunking,

26:31 I'm directly using the SDK rerunker and it's using a dedicated model,

26:36 cross-encoder model to do that rerunking step.

26:40 So let's run it and have a quick look at the result.

26:58 So yeah, results were a bit different, but quite similar in the end.

27:03 We again have had the top result,

27:07 a non-alcoholic beer from India, that match with Spice Chickpea.

27:13 Again, just have a look, we all have non-alcoholic beer.

27:16 So at least this was working just as what

27:20 we had when we are tried with the LLMs.

27:23 The good thing is that with a native rerun care,

27:27 The request will be faster and more reliable in the sense that the results will

27:33 be more stable if you try to run the same request a few different times.

27:43 So the reason that I'm talking about hybrid search today

27:46 and why it's well integrated across uh all our Azure databases,

27:50 including Cosmos DB, is that it really improves the relevance of the results.

27:54 So this table is from research done by the Azure AI Search team,

27:59 compiling search strategies for different kinds of queries.

28:02 You can see that keyword search and vector search

28:06 differ in kind of queries that they handle well, as we've seen in our example.

28:11 But as long as we use a full hybrid flow with a reranking

28:16 step that we see great results all across all kinds of queries.

28:23 And we've seen in all how to implement a full

28:26 hybrid search pipeline from scratch using Cosmos DB SDK,

28:29 but to make it even easier,

28:31 we also have worked to have a complete hybrid search integration in long chain,

28:36 both in JavaScript and Python.

28:38 And I'm personally very interested in open source.

28:42 I worked on the specific JavaScript integration, so really try it.

28:47 I've been a long time Longchain.js

28:48 contributor since the beginning of this library

28:51 and this is a framework that we really love here at Microsoft.

28:55 Before wrapping this talk, I just want to show you what the final

28:59 beer recommendation pipelines looks like for Burger Agent.

29:04 So we had our initial request for a spicy uh burger pick.

29:12 I'll just ask, recommend me a good, let's say, what do you want?

29:21 A good Belgian beer to go with it.

29:29 And while it's thinking, let's have a quick look at how it's implemented.

29:35 So this is our recommendation pipelines.

29:38 So uh all implemented in Longchain.js.

29:41 Currently, there's a slight limitation.

29:43 If you're doing keyword search, is that you can only use five keywords at most.

29:50 So our...

29:51 For our first step, we are basically using an LLM

29:54 to extract the five most relevant keywords from a query,

29:57 if you have more than five keywords.

29:59 Otherwise, we just take the query as it is.

30:02 So this is a limitation that we have currently,

30:04 but oh that will be gone in just a few months.

30:07 I know the product team is working on that.

30:09 ah Then we do the hybrid search with Longchain.js.

30:15 So no complex stuff to set up.

30:18 We can do it in one go, just...

30:20 pass it the query.

30:21 don't even have to compute the embedding ourselves.

30:24 Longchain.js, once you uh tell it which embedding model that you want to use,

30:29 it will do that part itself.

30:32 It's all integrated.

30:34 The number of results, the kind of search that you want.

30:37 So we can try to redo all the demo

30:39 ID just by switching the search type in there.

30:42 And because this one will be vectorized, we still need to send in the keyword

30:48 that we want to use for the full text search.

30:52 And we get the results.

30:54 And for now, because uh all I'm using is JavaScript in there,

30:58 I'm using an LLM-based rerunker,

31:00 so the same one uh using GPT-5 mini as I used in the previous demo.

31:06 Once it will be available publicly in the JavaScript SDK, I will switch to that.

31:12 And once I have all my rerun results, I have just a final step to map

31:18 that to the original entries that we had in the database,

31:22 the JSON one, not the vector ones.

31:25 And let's look at what it looks like in our user agents.

31:32 So yeah, with the spicy jalapeno burger, I wanted a Belgian beer,

31:36 so at least I got two uh nice beers.

31:39 And yeah, that's matched well with my burger.

31:44 So this is how basically you can get some good beer to go with your burgers.

31:54 So thanks for joining this session.

31:55 You'll find here uh all the links and QR code for the burger demo application.

32:00 I've shown you have the full source

32:03 code and a reality documentation also for Azure

32:06 Cosmos DB hybrid search if you're interested

32:09 into implementing that in your own application.