Vector Search with LLMs - Computerphile

0:00 I thought we'd take a look at vector search, right?

0:02 Which is one of the techniques used in modern chat systems

0:06 to find data and use that data when answering questions, right?

0:10 So, we know we talked about retrieval augmented generation in a previous video.

0:13 In that video, to try and get the language model to answer

0:18 a better question about I can't remember what it was I was asking about.

0:20 I think it was a a football stadium.

0:22 Let me tell you about the amazing Olympia Stadion in Berlin, Germany, right?

0:25 I just jammed the whole Wikipedia article

0:28 for the football stadium into the prompt.

0:29 Now, that's okay, but what happens if

0:33 your uh you don't have a Wikipedia article?

0:35 Or what happens if your company records

0:38 are thousands or tens of thousands of documents?

0:42 How do we find a good document to help answer a question?

0:45 If I say, "Where in our back catalog of documents is the the thing about how

0:50 me Shawn promised to pay me for doing this?" Um how can I find that, right?

0:58 The process we use is something called vector search.

1:00 And it's actually quite cool because it's it's similar in in in sort

1:03 of in principle to the video we did on Face ID, right?

1:05 Uh okay, so that's where you had to unlock your face with your phone.

1:08 Exactly, right?

1:09 And it's painful.

1:10 If you want to unlock a face with your phone, you won't let me forget that one.

1:13 I'm sorry, it's just that Um it in that what happens is you train

1:18 a a neural network to look at a picture of someone's face or depth

1:21 map or a lidar image of someone's face and put their face into some

1:26 embedded space where they are near or far away from other people's spaces.

1:30 And you can log them in if they're near

1:32 to what you think they're meant to look like, right?

1:34 This is exactly the same,

1:35 but it's now for sentences and paragraphs or words rather than for faces, right?

1:41 So, let's imagine that I ask a question.

1:44 So, I ask a question like, "Why is the sky blue?" Question mark.

1:49 Now, everyone knows of course it's because of Rayleigh scattering, right?

1:51 And this is also quite a common example we used in AI.

1:54 Um now, let's put aside the fact that most big models

1:57 could probably answer this because there's

1:58 sufficient information in the training data.

2:00 And let's pretend this is a question where we want to seek the answer

2:03 of this from some data set of paragraphs of text that we've got.

2:07 Let's say we've downloaded a huge encyclopedia, we've got all these texts.

2:11 How do we find the bit of text that answers

2:13 that question so that we can paraphrase it and respond?

2:16 So, what we're going to do is we're going to embed this sentence,

2:20 the tokens in the sentence,

2:21 using a language model into an embedded representation.

2:24 So, let's just draw a representation in two dimensions because that's easier.

2:28 So, this is dimension one.

2:30 And this is dimension two, and they're completely meaningless, right?

2:32 And the sentence, "Why is the sky blue?" is over here.

2:37 And maybe you ask a different question like,

2:38 "What is an atom?" I might have to ask one

2:41 of the people from Sixty Symbols cuz I'm not actually sure.

2:44 Um the what is an atom?

2:46 And this will come out somewhere else, right?

2:48 And the idea is that the network is it's been

2:50 trained on enough text that it's contrastively learning to say,

2:54 "Look, that sentence is not like that sentence,

2:56 right?" in this space in the embedded space.

2:57 Now, the problem with two dimensions is

2:59 that you quickly run out of space, right?

3:01 So, you need to have many more.

3:02 But the key to this is any paragraph that said something like,

3:05 "Because of Rayleigh scattering." I think I spelled it right.

3:09 Apologies if not.

3:10 The hope is that these two sentences are semantically similar, right?

3:14 And these two are not.

3:16 And so, when we embed this bit of text, we come out somewhere over here, right?

3:21 Which says, "Okay, we don't know that these are

3:23 the same sentence or that they say exactly the same thing,

3:26 but semantically the content of this sentence

3:29 is similar." That should have been purple.

3:31 How embarrassing.

3:33 I didn't fix it.

3:34 Uh this purple embedding is close to this green one.

3:37 So, now let's imagine you were building a system where you've got

3:39 hundreds and hundreds of paragraphs or thousands

3:41 or millions of paragraphs of text.

3:43 Some of them might be useful, some of them might not be useful.

3:46 What do we do?

3:47 Well, what we do is we store all

3:49 of these embeddings and we can retrieve similar-looking documents.

3:54 So, you ask a question,

3:56 "Why is the sky blue?" It embeds that into a numerical space and finds,

4:01 let's say, the 10 nearest points to "Why is the sky blue?" Right?

4:06 Now, some of them might answer the question, some of them might not,

4:09 but we then use those to look up the original text and we

4:12 put that into the prompt of our language model as well as well.

4:15 So, instead of just dumping a Wikipedia page

4:17 in like you don't know what you're doing,

4:19 uh you smartly find relevant pieces of text.

4:24 And this is an unbelievably powerful tool.

4:27 It's it's it's also very very common, right?

4:29 If you're inserting things into a large language model,

4:32 vector search is a huge hugely useful part of that process.

4:34 It's not practical to stick millions of documents into a large language model.

4:39 And if you do that, you might not find the right bit of text anyway.

4:42 Let's let's make the problem a little bit easier.

4:45 Okay.

4:45 So, how does this process work?

4:46 Well, we train this model using something like contrastive learning, right?

4:51 Where we basically give it lots of examples of text that is similar and lots

4:54 of examples of text that's different and teach

4:57 it to put them in different places, right?

4:59 This is the same way that CLIP embeddings

5:01 work when we did our video on CLIP embeddings, but that's for images and text.

5:05 This is just for text.

5:06 You can't control what you're going to find on the internet, right?

5:07 So, Absolutely.

5:08 This is called vector search, right?

5:10 So, you might have a database of many many and the text,

5:19 text one, text two, dot dot dot dot, right?

5:22 It's a huge lookup table.

5:24 So, you you find your position, you find these embeddings that are close to it,

5:29 and then you get the text, right?

5:31 Um and then you've just got to create this database.

5:33 And to do that, you just find a load of text,

5:35 you embed it once, and you store those embeddings, right?

5:37 So, it takes up quite a lot of memory, but these embeddings are not that big.

5:41 128, 256, 500 dimensions, right?

5:44 Just 500 numbers in a row.

5:46 Now, this is quite a nice way of doing it.

5:48 I know I've been grumpy about large language models

5:49 and and AI hype and things in some of my videos, right?

5:52 But actually, this is one of the things

5:53 that I think is really impressive about large language models.

5:55 If you never have to get it exactly right, right?

5:58 Which is a useful thing in this case.

5:59 So, for example, if I ask,

6:02 "Why is the sky colorful?" the answer is probably similar, right?

6:06 And so, but it might move it slightly,

6:09 but you might still get interesting bits of text.

6:11 You Or "Why is the sky blue?" but I spell blue incorrectly,

6:15 it will be close enough, right?

6:17 And the large language model embed The embedding will be in the right place.

6:20 So, you can make typographical errors or grammatical or or use poor grammar when

6:25 you talk to a large language model

6:26 because the way that it interprets these tokens,

6:29 that kind of is ignored, right?

6:30 Or at least overcome.

6:31 And that's I think quite a useful feature.

6:33 It's one of the things that people like about these um these AI systems is

6:38 that they offer this kind of natural language

6:40 way of interfacing with something like this database.

6:43 If if I ask you a question, but I foul up my sentence,

6:46 you'll probably get the gist of what I'm saying, right?

6:48 Maybe I try and ask ask something in French.

6:50 My French is not that great.

6:52 Someone who's French will probably get what I'm saying because I get

6:54 enough of the words correct even if it's in the wrong wrong order.

6:57 So, looking at that on two dimensions, that's pretty straightforward.

7:00 We can see where those X's are, but even if it was just three dimensions,

7:03 one of those X's might be 100 miles away in the other direction.

7:07 Yeah.

7:07 How do you cope with that?

7:08 Yeah, um okay, good.

7:09 So, there are a few ways you can measure similarity in an embedded space, right?

7:13 The the common one is is something we call cosine distance, right?

7:16 And it's the angle between two vectors.

7:18 So, if we imagine this is the this is the origin, right?

7:22 Not not.

7:23 This here is a vector.

7:26 It has a D1 and a D2 or an X and a Y.

7:28 This is another vector.

7:30 All right, that and I nearly didn't have to bend it to All right,

7:33 this is the angle between those vectors.

7:36 Whereas this vector here, you can see the angle is much smaller.

7:39 So, in two dimensions,

7:40 the cosine distance will just be what is the angle between these two vectors?

7:43 What is the angle between these two vectors, right?

7:45 And we normalize it typically between, let's say,

7:47 not and one or minus one and one.

7:49 Now, in three dimensions, it's actually exactly the same.

7:52 Do you think I can draw that?

7:53 I mean, we can give it a go.

7:54 So, in three dimensions, you'd have something like I'm I'm regretting doing

7:59 this already and I haven't even started yet.

8:01 So, you got you got this, and then you've got this coming out here.

8:04 Um and then you've got a vector in 3D space like this one,

8:09 and you've got another vector in 3D space like this one,

8:12 which for the sake of this diagram is more angled that way.

8:16 It's still the angle between them, right?

8:19 And which is going to be between not and 180.

8:22 In four dimensions, I'm afraid I can't draw it on the page.

8:24 In five dimensions, I can't draw it on the page,

8:26 but the calculation is exactly the same.

8:28 So, why would we do that and not just measure this distance here, right?

8:33 Because we've got essentially 500 or or or three numbers.

8:38 They are a point in 3D space.

8:39 They don't have to be a vector.

8:41 Why would we measure this angle?

8:43 Well, the answer is that it then basically

8:45 degenerates everything everything is of length one, right?

8:48 They're all on a a big circle like this, right?

8:50 In 500 dimensions.

8:52 If you can visualize a 500-dimensional circle.

8:54 I try sometimes, right?

8:56 It might be that for just for reasons of network output,

8:59 this vector is actually really really long, right?

9:02 It's the same angle as it was before,

9:04 but what we're saying is the length of the vector is unimportant.

9:06 The magnitude of this vector is unimportant.

9:09 We only care about how the direction they're facing.

9:12 And this essentially it reduces a bit of sort

9:15 of error and noise in your process, right?

9:17 You could do it this way, but you'll find that maybe a sentence is just

9:21 like a bigger version of what it should be,

9:23 and it ends up putting it somewhere else, which is similar to other sentences.

9:26 Is Is it possible though that that you've

9:28 got two things that are nowhere near each other,

9:30 but they're on the along the same vector line or more than two things?

9:33 That's what cosine measures, right?

9:34 So, cosine is measuring whether two things are essentially

9:37 along the same line regardless of how long they are.

9:39 So, if you had a bunch of different lines on here,

9:43 they would all have a cosine similarity of one or, you know, no angle.

9:47 So, they are similar Yeah, they are similar, right?

9:49 And that's kind of the hope.

9:51 Um these would all be measured as quite different

9:54 if we were doing it based on just distance Yeah.

9:56 in through in, you know this dimensional space.

9:59 All right, so I've coded this up and when I

10:00 say coded this up I've used libraries to code up.

10:03 Just a very simple example to show you how this works.

10:05 Okay, so I've imported some libraries.

10:07 Um I'm going to create my embedding network, right?

10:09 So the embedding network is not a network that predicts

10:12 the next word in a sentence or predicts what image is interesting.

10:15 It just turns your sentence, however long it is,

10:19 or your paragraph into a numerical representation.

10:23 Uh so I'm going to import that.

10:25 Um I'm using hugging face to do

10:26 this and the model I'm using is MPNet-base-v2, apparently, right?

10:29 It's a standard one, right?

10:31 So it's downloading all the model and so sorting that all out.

10:34 So there's obviously there's a there's kind of like

10:35 a a process of going through and turning it into a vector.

10:40 Yeah.

10:40 In actual fact, the process is not dissimilar

10:42 to what a normal large language model does, right?

10:43 It's still a transformer network.

10:45 It represents the sentence as tokens.

10:47 It goes through all these attention layers.

10:48 It's just been trained to output lots of numbers

10:50 instead of what the next word will be.

10:53 Um so I've got three embeddings here that I'm going to create.

10:56 E1 is why is the sky blue?

10:58 E2 is the sky is blue due to a process called Rayleigh scattering, right?

11:02 Which I would say is similar in theme to the previous sentence.

11:06 Um and E3, which is my wildcard, is bicycles typically have two wheels,

11:10 otherwise they're called a tricycle or a car.

11:13 And I'm sure the people in the comments will point out all the other

11:15 non-two-wheeled vehicles that um I decided that wasn't

11:21 the most important part of this video.

11:23 So I'm going to embed this.

11:23 So each of these will go through the network and produce E1,

11:27 E2, E3, which are embedded variables.

11:29 So these just think of these as positions

11:31 on our in our dimensional space that we can measure the distance between.

11:35 If we add another um code block to this, I can

11:38 say for example E1 and you can see here it is.

11:43 These are the embeddings for E1 and they're quite long.

11:45 They're just meaningless numbers.

11:47 They aren't useful to humans.

11:48 They're useful because the distances between or the angles

11:51 between this embedding and others are useful,

11:54 not the numbers are useful in other ways.

11:56 So if we scroll down,

11:58 I'm going to calculate the cosine distance between E1 and E2,

12:02 E1 and E3, E2 and E3.

12:04 And as far as I know this returns a number between zero and one,

12:07 where zero is they are the same vector and one is they are very,

12:10 very angled very, very far apart, right?

12:13 So let's run this.

12:14 Um so I'm doing E1 to E2, E1 to E3, and E2 to E3.

12:18 And remember E1 and E2 are our Rayleigh scattering related sentences.

12:25 And there we go.

12:26 So the distance between E1 and E2 is 0.2 and the distance

12:30 between E1 and E3 or E3 and E2 is 0.94, right?

12:35 All right, so almost about as different as you can get, really.

12:38 So imagine you had a bunch of bunch of interesting facts, right?

12:42 Like your bicycle wheel fact, right?

12:43 Which is very interesting,

12:45 in your database with these embeddings and you looked you said we need

12:49 to answer a question on Rayleigh scattering or why is the sky blue,

12:52 what text do we use to do that?

12:54 This will find the nearest text, which is the thing about Rayleigh scattering,

12:57 not the thing about bicycles, right?

12:59 And that's how you try and get the correct answer.

13:02 There's one thing I'll point out is that we live in Nottingham, very famous

13:06 [snorts] bicycle factory called the Raleigh Bicycle Factory was based here.

13:10 Maybe that's why this is 0.94 and not and not one.

13:13 I don't know.

13:14 Um but my my spelling of both could be wrong.

13:17 So let's improve [snorts] my retrieval augmented

13:19 generation process with a slightly more complicated example.

13:22 Here we're going to download and load

13:24 up the NIST recommendations for key management document.

13:28 Now, the NIST recommendations for key management is

13:30 a great document if you're interested in key management,

13:32 that is cryptographic keys, right?

13:34 How long should your key be?

13:35 What algorithm should you use?

13:36 Where should you store it?

13:37 How long should you store it for?

13:39 How many times should you copy it?

13:40 Should you encrypt your key with another key, right?

13:42 These are all questions answered in this document and it's 170 pages long.

13:47 So let's suppose you had a cryptographic key management question

13:51 and you didn't want to read the whole document, right?

13:53 Now you could use the index.

13:55 But but let's suppose you can't use the index

13:56 for for for reasons of it makes my example better.

14:00 Um what we're going to do is we're going to load up this PDF

14:04 and then we're going to look at here's an example page from my PDF.

14:07 This is actually just I think

14:08 the the boilerplate at the beginning of the document.

14:11 We're going to use something called a text splitter to chunk

14:14 the whole PDF up into groups of about 800 words or 800 tokens.

14:19 And we're going to have a bit of an overlap between them.

14:20 You might you don't want your really important fact to be split halfway,

14:25 so you just don't you just do a bit of overlap.

14:26 You could fiddle about with this and change it, right?

14:28 So I'm going to chunk up my deck.

14:30 And so I can see, for example, that split 171 is something about

14:35 exchanging information between internet identities, right?

14:38 I don't, you know, who knows.

14:40 The next thing for me to do is to embed every single one of these chunks

14:43 into the embedded space and stick them

14:45 in a a database where I can retrieve them.

14:47 I'm using a database called Chroma for this.

14:49 There are various that you could use.

14:50 So all I'm doing here is I'm passing the embedding

14:54 function by our network that we used earlier into Chroma DB.

14:58 And then I add all the documents.

15:00 And this takes a little while on the GPU.

15:02 It's now actually processing all of those chunks.

15:04 There is only about 800 to 1,000 of them.

15:06 It's not too many.

15:07 And so now I have a query.

15:08 What it Now I'm not going to ask about why is

15:10 the sky blue because I don't think it's in the document.

15:12 Uh so I'm going to ask what is the current

15:14 recommended crypto period for a symmetric key wrapping key, right?

15:17 Now I don't know if there is such a recommendation, right?

15:19 But it seems like an interesting question one could ask of this document.

15:22 So I'm going to query it.

15:24 I use the vector DB.

15:25 I'm going to do a similarity search.

15:27 That will be using cosine almost certainly with my query and I'm

15:31 going to return 10 potential power the closest 10 points, right?

15:35 And the text for those.

15:36 So I run that.

15:37 Doesn't take very long.

15:38 The look up these these comparisons are points are very, very fast.

15:42 And then [snorts] we look at the results for doing it.

15:44 So the Ah, here we go.

15:47 So the first text says the crypto period recommended use

15:50 of period for symmetric key wrapping key is that it's used

15:53 to wrap very large numbers of keys in a short period

15:55 of time is on the order of a day or week.

15:56 So I don't know whether that's I mean it's probably right, right?

15:58 It's in the NIST document,

15:59 but it does seem like a relevant paragraph that it's dug out, right?

16:03 It hasn't returned anything on bike manufacture.

16:05 Um so you could actually build this into a kind

16:08 of hands-off query and then respond thing.

16:12 So I'm going to create a a Mistral 7B instruct model, right?

16:16 So I'm I'm basically just going to have a little little large language model,

16:19 which usually is quite a small model.

16:21 It's not great at answering questions compared to some of the bigger ones.

16:25 [snorts] Um I'm going to build a small pipeline where I automatically query,

16:29 retrieve the data, and then answer based on that data, right?

16:33 So I've created a prompt template that says you answer

16:35 me a question using only the information provided in the context.

16:38 Do not use outside knowledge.

16:40 And then this context is going to be populated with my query, right?

16:45 It's loading up the the model.

16:46 It's good thing I chose a 7 billion one and not the 170 billion one.

16:50 Where is that running?

16:51 On the cloud?

16:51 This is running on the on the cloud, yeah.

16:53 You could run this.

16:54 It's quite a small model.

16:54 You could run it on your own graphics card at home.

16:57 Okay, this seems to have done something.

16:59 So create my plot I'm going to create my pipeline, create my prompt template.

17:04 Um this is mostly using hugging face and LangChain, all right?

17:07 And then I've got my rag query.

17:09 So there's also a rag query here that says why.

17:12 Uh [laughter] so I'm going to delete that.

17:15 Is it asking you why you're doing this?

17:16 Yeah.

17:17 Um so the rag query says it does it first of all

17:20 in it does a vector similarity search a vector search with the query,

17:25 returns 10 answers, and then sticks them all into the context,

17:28 and then asks the language model to answer

17:30 the question based on that context, right?

17:32 So that's fairly straightforward.

17:33 So what is the current recommended crypto

17:35 period for a symmetric key wrapping key?

17:37 Run that.

17:39 Language model responds, hopefully in a minute.

17:41 The context suggests the recommended crypto period for a key depends

17:44 on the number of keys being wrapped in the duration of use.

17:46 For wrapping large numbers of keys, order of a day or a week.

17:49 Relatively small number of keys, it could be up to two years, right?

17:53 So we'll let the cryptographers

17:55 in the chat decide whether that's actually reasonable.

17:57 That doesn't sound too bad.

17:59 But the point is I haven't had to look through the document.

18:01 It's all happened automatically based on my query, which is quite cool.

18:04 Now what happens if you talk you ask it nonsense, right?

18:08 Well, in the prompt I've said don't answer

18:10 the question if it's not in the context.

18:12 So if why is the sky blue?

18:16 blue {question mark} and I run that, my hope is

18:19 it will say I don't know or it will say ah,

18:22 the provided context does not contain information about

18:24 the color of the sky specifically what is blue.

18:25 The contract focuses on cryptography,

18:27 network protocols, and key management, right?

18:29 So that you know, you have to think about your prompt,

18:33 otherwise it will start having a guess.

18:34 That I was going to say that's possibly my favorite thing

18:37 about that the fact that it knows when it doesn't know.

18:40 That's an interesting point, right?

18:41 A lot of people ask me when we talk about AI,

18:43 why don't they just say when they don't know?

18:45 A lot of the time you're not

18:46 using retrieval augmented generation or vector search.

18:48 You haven't got this context of truth.

18:51 If every time you ask a question,

18:53 it sourced a data source and answered based on that, it could

18:57 easily say the data source is conflicting with what you're asking, right?

19:00 Or I don't know cuz it's not in the data.

19:02 But what they usually do is rely on their own network weights

19:04 and training data to answer the question and they have a go.

19:07 Um it's easier to get a language model to say no when you're saying

19:12 compare this with this, as opposed to just

19:14 answer this with no comparison, you see?

19:16 So it's about how do you develop these systems?

19:19 What do you want them to do when someone says something, right?

19:22 And and it isn't an easy problem to solve.

19:24 If you train your network to say I don't know to a lot of stuff,

19:27 it will start saying it to loads of stuff even if it did know,

19:30 and then no one wants to use your system either, right?

19:33 So it's not an absolutely obvious system.

19:36 But for building a system,

19:37 let's suppose you want to build a system that's more robust.

19:39 So maybe you're actually looking to put all

19:40 your company's data records in here and you're

19:43 looking to find the record that that is

19:45 is is is you know linked with this theme.

19:48 This is more likely to give you a response that's correct,

19:51 but it's also more likely to say there is no such data, right?

19:54 As opposed to, yeah,

19:55 here's the data and then literally make something up, right?

19:57 Which of course is never a good thing.

20:06 Add up the bytes and check the sum.

20:07 That's great, right?

20:08 It's great except that it's quite common to get two bit errors,

20:13 one that adds maybe 16, you know,