Is RAG Still Needed? Choosing the Best Approach for LLMs

0:00 There's a fundamental truth about LLMs, large language models.

0:05 They are frozen in time.

0:09 They know everything about our world up

0:12 until their training cutoff date and absolutely

0:16 nothing about what happened 5 minutes ago.

0:18 Nor do they know anything about your private data,

0:21 your internal wikis, your proprietary codebase.

0:23 And if we do want an LLM to know any of that stuff,

0:28 well, we have to solve the problem of context injection.

0:32 How do we get the right data into the

0:36 model at the right time?

0:37 And there have been two very different ways to handle this.

0:42 Now, the first is really what we can think of as the engineering approach.

0:49 It's RAG, retrieval augmented generation.

0:52 So here we've got an LLM and we've also got an input prompt from the user.

0:59 Now ahead of time

1:01 we take some of the documents that we want to give to this LLM.

1:06 So these are documents that could be

1:09 PDFs or code files or entire books and we chunk them.

1:14 We break them up into smaller chunks and

1:19 we pass them through to an embedding model and

1:23 the embedding model takes those chunks and it

1:27 turns them into vectors and those vectors are

1:32 then stored in a dedicated vector database.

1:36 Now when a user asks a question,

1:41 it performs a semantic search to retrieve the most relevant chunks

1:47 and then inject them into the context window.

1:51 So now the context window has the user prompt,

1:54 but it also has all of these chunks that we

1:59 have taken from the vector database and together this

2:03 forms the context window.

2:04 Now this works but it does rely on something.

2:08 It relies on the

2:10 hope that your retrieval logic actually found

2:12 the right information in the vector database.

2:15 Now the the second approach is really a bit more

2:19 of a brute force approach and that one is called

2:23 long context.

2:24 Now this is really the model native solution

2:28 because you skip the database here and

2:31 you skip the embedding model.

2:33 All you do is you take your documents and you just well you put them

2:38 straight into the context window and then you

2:41 let the model's attention mechanism actually do the

2:44 heavy lifting of finding the answer.

2:46 Now for a long time this kind of brute force method wasn't

2:50 really much of an option because initially context windows were tiny.

2:56 Early LLMs had context windows

2:58 that could maybe store what like 4K of tokens.

3:02 You couldn't fit a novel in there, let alone a

3:06 corporate knowledge base.

3:07 You basically had to use RAG.

3:10 But today's models have much larger context

3:14 windows.

3:14 Some of them have, you know, a million tokens plus.

3:19 And to put that into perspective,

3:21 a million tokens is roughly 700,000 words.

3:24 and you could fit the entire Lord of the Rings series into

3:29 the prompt and still have room for The Hobbit.

3:32 So, this massive jump in capacity forces us to ask a

3:36 difficult question about our architecture.

3:38 Because if we can simply command A, command C, command V,

3:43 all of our documentation into the models context window,

3:46 do we really need the overhead of

3:49 embedding models and vector data stores?

3:52 Is RAG becoming an unnecessary complexity layer?

3:56 Well,

3:56 if we accept that we can fit whatever data we need into the context window,

4:02 then the argument for doing so basically boils down to one word, simplicity.

4:07 And let me give you three reasons why

4:11 stuffing the context window directly may indeed be the way to go.

4:17 And reason number one is collapsing

4:20 the infrastructure.

4:20 A production RAG system.

4:22 Well, it is quite heavy.

4:25 You need a a chunking strategy

4:27 which is like fixed size maybe or sliding window or recursive.

4:31 You decide.

4:31 You're going to need

4:33 a embedding model to encode the data.

4:35 You need a a vector database to store it.

4:39 You're going to need a reranker to sort the results.

4:42 you need to keep all the vectors in sync with your source

4:45 data.

4:45 It's basically a lot of moving parts, a lot of places for things to break.

4:51 And long context offers what we might call well

4:55 just simply the uh the no stack stack.

4:59 You remove the database, you

5:01 remove the embeddings, you remove the retrieval logic.

5:04 The architecture simplifies down to getting

5:06 the data and just well sending it to the model.

5:11 So that's reason number one.

5:13 Reason number two is the

5:15 retrieval lottery.

5:16 Now, RAG introduces a critical point of failure here, the retrieval step itself,

5:23 because when a user asks a question,

5:26 RAG looks at mathematical representations of the data, which are

5:30 stored in vectors.

5:31 And vectors are basically just like a really long series of numbers in

5:37 an array.

5:37 And it tries to find the closest match.

5:41 That's semantic search.

5:42 But semantic search is probabilistic and for all manner of reasons,

5:46 the retrieval might fail to find the

5:49 relevant document.

5:50 And we actually have a name for this.

5:53 It's called silent failure.

5:54 The answer, well, it existed in the data,

5:57 but the LLM never saw it because the retrieval step didn't return

6:01 the right results.

6:02 With long context, there is no retrieval step.

6:05 The model gets to see everything.

6:08 Now, reason number three that is well,

6:10 I think we're going to call this the whole book problem.

6:15 A RAG is fundamentally designed to retrieve what exists.

6:18 It relies on finding a semantic

6:21 match between your query and a specific snippet of text in your database.

6:25 But what if the answer

6:27 lies in what's not in the database?

6:30 So, so let's say you have a set of product requirements stored

6:35 as a document and you've also got a set of release

6:39 notes stored as a document and then we ask which

6:42 security requirements were omitted from the final release.

6:46 Now using RAG when you query for omitted

6:50 security requirements the vector search looks for

6:54 chunks discussing well security and requirements.

6:57 It retrieves snippets from the requirements doc.

7:00 It retrieves snippets from the release notes,

7:02 but it cannot retrieve the gap between them.

7:05 And because RAG only shows the model a few isolated

7:09 snapshots,

7:09 the model never sees the full picture required to spot the missing pieces.

7:14 The model really needs both of these

7:17 documents in full to perform the comparison,

7:20 which is exactly what

7:22 long context does by dumping the whole book,

7:25 the full requirements doc and the full release notes

7:28 into the context window.

7:30 So, is RAG dead?

7:31 Is the vector database destined for the museum of things

7:35 we needed in 2024?

7:37 Well, not quite because while long context wins on simplicity, RAG still has a

7:43 place.

7:44 And I got another three reasons to support that.

7:48 So, reason number one is the rereading text.

7:53 Now, long context creates a massive compute inefficiency.

7:56 So, if we take a manual, let's say

7:59 this is like a a 500 page manual, and we've got to turn this into tokens.

8:06 Well, that's something like

8:08 250k of tokens.

8:09 And we need to do that every time we make

8:13 a user query and we put this document in

8:17 the prompt.

8:17 You're basically requiring the model to process that manual every time.

8:22 Now, RAG also has to process that manual,

8:25 but it only pays that processing cost once at indexing time.

8:29 Now, prompt caching that can partially offset some of this for static data,

8:34 but for dynamic data streams where

8:36 content changes frequently, you are stuck paying the full tax on every request.

8:43 Reason number two is the needle in the haystack problem.

8:48 Now, there's a an intuitive assumption that if data

8:52 is in the context window,

8:54 the model's probably going to use it, but research suggests otherwise.

9:00 Because as we start with a context window and then it

9:03 grows and it continues to grow and now we're at

9:06 like 500,000 tokens, well,

9:08 the model's attention mechanism can get a bit diluted.

9:12 If you ask a

9:14 specific question about a single paragraph that's buried in,

9:16 let's say, the middle of a 2,000 page

9:19 document, well, the model often fails to retrieve it

9:23 or it hallucinates details from the surrounding

9:26 text.

9:26 But with RAG, we're giving the model less noise.

9:31 So by retrieving, say only the top five

9:35 relevant chunks,

9:36 RAG has removed the haystack and presents the model with just the needles.

9:42 It forces the model to focus on the signal and not the noise.

9:47 And then reason number three,

9:49 well that is the infinite data set.

9:52 Now a context window of millions of tokens sounds great but in

9:57 the scheme of enterprise data that's really just a drop in the bucket.

10:03 I mean an enterprise data lake

10:05 that's probably measured in terabytes or or maybe even petabytes.

10:10 So if you want an infinite data

10:13 set that stores everything,

10:15 you really do need to have a retrieval layer to filter information down

10:19 to something that fits into the LLM context window.

10:23 So where does this leave us?

10:25 Well, if your problem involves a bounded data set

10:28 and requires complex global reasoning like analyzing

10:30 a specific legal contract or summarizing a book,

10:33 I think long context is the way to go.

10:37 It simplifies the stack and it improves the reasoning.

10:40 But if you're navigating the infinite data set of

10:44 enterprise knowledge,

10:44 the vector database remains the only viable warehouse for your data.

10:50 But how about you?

10:52 Are you team long context, team RAG, maybe a bit of both?

10:59 Let me know in the comments.