Search What You See: The Tech Behind The Magic | Made by Google Podcast S9E4
0:00 Welcome to the Made by Google Podcast,
0:03 where we meet the people who work on the Google products you love.
0:06 Here's your host, Rachid Finge- Circle.
0:09 to Search is getting a huge upgrade,
0:10 making it even more powerful to find out what
0:13 you're looking at on your Pixel 10 or Galaxy S26.
0:17 And if you're into fashion, it has a mind blowing trick up its sleeve
0:21 that will help you find a beautiful new look.
0:24 Let's learn more today from Director of Product Management, Harsh Kharbanda.
0:29 This is the Made By Google Podcast,- Harsh.
0:32 Welcome to the Made By Google Podcast.
0:34 Uh, we're talking circle to search today.
0:36 Do you remember the very, very,
0:38 very first time you came across Circle to Search?
0:41 So, I've worked on Lens since Google Lens,
0:43 since the beginning of Google Lens, um, since 2018.
0:46 And, um, at some point during as Lens,
0:50 as Lens was growing up, we, like, our main focus was the camera.
0:54 And you know, like how users can take a picture
0:56 of things in front of you and then get,
0:58 get question, uh, uh, their questions answered.
1:00 And at some point we realized a lot of the value is, uh,
1:04 for people to be able to search things that they see on their computer,
1:07 on their phone screens.
1:09 Uh, and so because what we started notice
1:11 was users take a lot of picture screenshots,
1:14 um, and then they just upload that to Google Lens.
1:18 Um, and so, uh, a part of our team started working with Android
1:21 and trying to figure out how to make this whole slo a lot, a lot more seamless.
1:26 And that's where the Circle Gesture came around.
1:28 And the first time I used it, it just felt so intuitive, um,
1:32 that, that I was like, oh yeah, I'm going to use this like 10 times a day.
1:36 Uh, 'cause there's so many different questions, uh,
1:38 on my phone screen that I come across every single day.
1:41 And then, yeah, from then on I was just hooked.
1:43 So as the Lens and the Circle to Search products merged over time,
1:47 I ended up owning more and more Circle to Search, uh, product as well.
1:49 And yeah.
1:50 And here we are.
1:52 So- I guess in order to understand where we are today,
1:54 it might be useful to back up a little.
1:56 Of course, the search engine started with typing in queries in the search box.
2:00 Then at some point we got voice search,
2:02 and then we got something as you mentioned already called Google Lens.
2:06 So why did we come up with Google Lens back, uh, a few years ago, I guess?
2:11 Uh, great question.
2:12 So the concept of a visual search has been around for a very long time.
2:16 If you remember, I think Google Goggles was something that we- Oh, right.
2:20 Google worked on,- Goggles, worked on in 2012 or 2014.
2:22 I, I forget exactly when, but Google invested in this, this concept
2:26 of being able to search for things visually,
2:28 very long time, uh, very early on, but the technology was just not there.
2:34 Um, you know, early on Google Lens could, could only do QR codes,
2:38 maybe some objects, um, and some translation and so forth.
2:42 Um, and so the, the key thing with Lens was,
2:45 uh, you know, like we knew that there is,
2:48 there is demand because we see that there's a lot of questions that are very
2:53 hard to articulate just verbally and for users
2:56 to actually express via text or voice.
2:58 Like how do you describe a very unique
3:01 dress that you come across on social media?
3:03 There's so many different words.
3:05 And even if you use those words,
3:07 you're still not capturing the essence of that dress.
3:10 Um, or like what people use Lens a lot for like plants.
3:13 How do you describe a plant that is in your house that somebody gifted to you
3:17 and you don't know the name of, and it's dying in a very specific way.
3:21 And it has these like spots on it, how, like, it's just really,
3:25 really hard and you can give clothes where it'll take you a long time and then,
3:29 you know, you lose a bunch of fidelity in doing that.
3:32 And so the impetus of like, Hey,
3:34 users have so many questions that are hard to describe,
3:36 was something that Google saw very early on.
3:40 So we had Lens Visual Search, as you call it as well.
3:43 And then at some point came circle to search.
3:46 So for, for people who have never used Circle to search,
3:49 how is that different from just using the camera to search something?
3:53 When we started Lens,
3:55 a lot of the focus was on things that you see in front of you, right?
3:59 Um, but what we quickly realized was,
4:02 you are in stimulus in, in your daily life is not that much.
4:06 Uh, and so you're seeing your same plan, your same dog,
4:09 your, you know, same office location and, and work desk and so forth.
4:12 And so you don't have that many opportunities to ask new questions.
4:16 And phones have gotten better and better.
4:18 Pixel is an amazing product, and people just spend a lot,
4:21 many number of hours on social media
4:24 and on different parts of, of, of their phone.
4:26 And as they scroll through and as they browse their phones,
4:30 they come across the stimulus where they have these questions.
4:33 And so the whole, the goal was how to bridge the gap between,
4:36 okay, we have visual search,
4:38 we're really focused on things in front of you, and it doesn't really,
4:41 you know, like it's not a thing that people do all the time.
4:44 How do we bring it closer to the device?
4:45 How do we bring it where the user's questions are?
4:48 And that's what led to Search.
4:51 And for those who never had the chance to use Circle to Search,
4:54 can you explain to me as a user, like how, how do I use it and, and what,
4:59 what is maybe your favorite example of a situation where it's super useful?
5:04 Uh, sure.
5:05 So, um, it's very simple, uh, on any, on your new Pixel device, uh,
5:10 you just have to, uh, long press the nav handle, um, on any app that you're on.
5:15 That's that bar at the bottom, right?
5:16 Correct.
5:17 Yes.
5:17 Uh, the bar at the bottom.
5:18 Uh, and that just takes, uh,
5:20 the freezes your screen, and then you can tap anywhere.
5:24 Or the best part is you can draw a circle
5:26 around exactly what you want, and then you get, uh, you know, like, uh,
5:30 we will show you and most likely an AI answer of what you're looking at.
5:34 And then you can ask questions like very specific questions about the thing,
5:37 uh, things that, that you, that you're looking at as well.
5:40 Um, one of my favorite examples, so, uh,
5:43 I get a lot of spam from my mom on WhatsApp,
5:48 uh, like, she sends me a ton of, uh,
5:51 like messages that are definitely fake, at least in my opinion.
5:55 Uh, or like videos of like, I'm like, this, this cannot be true.
5:59 Um, and I use Circle to Search a lot to verify a lot of this information.
6:03 So all I do is just long press, uh, you know, the, the nav handle,
6:06 and then I circle and it just tells me, Hey, this thing is not true.
6:10 Here's the reasons why.
6:11 And it actually searches the web figures out like, what,
6:14 what's, what's, uh, actually true, what's not where the nuance is.
6:18 And often I would screenshot that and send it back to her.
6:21 Although, you know, that has gotten me in trouble many times.
6:24 But, you know, that's one of my favorite personal use cases of Circle to Search.
6:28 And I guess what I use it a lot for, if, if there's something on my screen, uh,
6:32 in a, in, in a language that I don't,
6:35 that I cannot read, I can also circle text Right.
6:38 And get a translation.
6:39 Yeah,- Exactly.
6:41 And that, that happens to me too,
6:41 like in Family Chats sometimes somebody would forward a message
6:44 that's like in a, in a different Indian Language,
6:46 uh, dialect, uh, that I don't speak,
6:49 or it's, I, I, I might even speak, but it's hard for me to read,
6:52 and I would just circle it and it can read it out to me, it contrast it for me.
6:55 Uh, so it's, it's a pretty nifty, uh,
6:57 feature that I, a lot of our users use a lot.
7:01 Harsh.
7:01 I'm curious, like when did, because that's of course, you know, we're,
7:04 we're going to talk about the new
7:05 update for Circle To Search that recently launched.
7:08 And I guess at some point you probably had the insight,
7:11 and I'm not sure if you expected it,
7:13 that people are using Circle to Search in sort of a fashion sense as well.
7:18 Yeah, actually.
7:19 So, uh, this was very early on in Circle to Search.
7:22 We realized that this would be, this would be a thing.
7:24 So one thing we learned from Google Lens was,
7:28 uh, especially during COVID users, uh,
7:30 spent a lot of time on their phones and a lot of time on social media,
7:35 and they would come across, uh, videos or, you know,
7:39 like, uh, social media posts of influencers,
7:41 and they be like, oh, I love that jacket,
7:44 but obviously I'm not gonna buy a $5,000 jacket.
7:46 And so they would take screenshots and then upload
7:49 it and bring it to Google Lens and upload it,
7:51 uh, to Google Lens to find something similar
7:53 that is in their price range and so forth.
7:55 Um, and so with Circular Search,
7:57 we knew very early on that if we reduce the friction of, you know,
8:01 the user not having to take screenshot and then, you know,
8:04 closing the app, then opening Google Lens and uploading the screenshot,
8:08 if they can just, you know,
8:09 on the screen without leaving Instagram or TikTok or whatever you're watching,
8:13 and just, you know, ask the question right then and there,
8:16 it would actually lead to growth.
8:17 And so we did see this very early on.
8:20 Um, you know, users as they were like when we launched Circular Search,
8:23 they, they used it very naturally for shopping.
8:26 In fact, younger users use circular research for shopping a lot more,
8:30 um, because they're all,
8:32 all obviously budget conscious and, and they have a sense
8:35 of style and aesthetic that they want to like recreate.
8:38 Um, and so, uh, younger users, female users is very,
8:41 like this, this is a population that ends up using shop,
8:44 like visual shopping using Circle to Search a lot.
8:48 And, um, what we started to see is a lot of times they're not actually
8:53 circling or wanting a single jacket or like
8:57 a single product that they're looking at, right?
8:59 Um, they actually wanted the whole vibe, the whole feel of the outfit.
9:03 Um, an example is like, oh, I saw Taylor Swift,
9:06 uh, in this like, awesome look, uh, that is going viral.
9:10 Uh, and, uh, it's like she posted her Instagram,
9:13 or I saw through her Instagram account,
9:15 and now I want to like, find every part of that look and of course,
9:18 recreate that look, but in my budget.
9:20 Um, and so Circle to Search didn't really work well for that.
9:23 It worked really well for single objects, and you could circle one by one,
9:26 like the jacket, the top, the jeans, the shoes, the, the bag.
9:30 And some of these influencers have sunglasses.
9:32 So there's like, you know, five- A lot to circle, right?
9:35 Yeah.
9:35 Right.
9:36 Uh, and so, um, and so now with this update,
9:39 what we enabled was the user to just
9:41 circle the whole thing and then find the look.
9:45 And with the late, the, with Gemini 3 and all the latest, uh,
9:49 model updates that we've made,
9:51 we are able to really look at the image in its whole, uh,
9:55 and break it down and think through what are
9:57 the different parts of the image that are really interesting,
9:59 uh, and break it down, uh, uh,
10:01 for the user of all the different parts of the image.
10:04 And then the best part,
10:05 as we do the same visual search for each part of the image,
10:08 so we find the sunglasses, the jacket, the jeans, et cetera,
10:12 uh, using visual search, and then we find you the exact product,
10:16 if possible, and many, many similar products as well,
10:20 um, uh, for you to like, be able to, uh,
10:22 you know, browse and, and, and find something that's
10:24 in your range and really recreate the whole look.
10:27 And then the cherry on top, once you find the product that you like,
10:30 you can click on it and you will see an option to try that thing on you.
10:34 So when you, when you click the try on new option, you will see the jacket, uh,
10:38 that you found that is maybe in your budget, on you,
10:41 on your on and on, what it looks like on you.
10:44 And so that whole like, you know,
10:46 end to end journey or something that when we tested with users,
10:49 they really, really loved it.
10:50 Like, oh my God, Taylor Swift was wearing that, and I,
10:53 I found something for like less than
10:55 a hundred bucks for that, and it looks great.
10:57 I'm gonna buy that.
10:58 Right?
10:58 Um, and, and it really delighted users.
11:01 That's amazing.
11:01 Now, if you back up a little bit, you mentioned,
11:03 so Circle to Search can now, uh, detect multiple objects.
11:07 Yep.
11:07 Uh, and we've been speaking about fashion,
11:09 but what other scenarios are you thinking about where it
11:12 might actually be useful to detect multiple things in one image?
11:15 Yeah, yeah.
11:16 And get a, a, a result from search?
11:18 Yeah.
11:18 There's, there's many, uh, so like, uh,
11:22 one obvious one related to products, uh, is one,
11:25 one the PMs who, uh, one of the PMs, uh, in the team who works on this, uh,
11:29 came up with, uh, she's really into skincare and, you know, she came across,
11:34 uh, like this, like skincare routine on Instagram,
11:36 which had like 14 different products.
11:38 And it literally lists out 14 different,
11:40 like, or images of 14 different products.
11:42 And so, you know, she basically just circled it and it's like, Hey,
11:45 can you find me all the products and give me reviews for everything?
11:48 And like, uh, you know, like rank it by price and it's able to search
11:53 for every single thing and then just do that for you.
11:56 Uh, another one that, uh, you know, uh,
11:58 I came across, uh, was like the, in, I was,
12:01 I was on social media and I came across the setup of plants, um,
12:04 and I was like, oh, these plants look great,
12:06 but I don't know what the names are.
12:07 And they'll, you know, uh, like I can grow in them in my house.
12:12 So I just circle it and ask for like, Hey,
12:13 can you name all the plants their growing conditions, what, what it takes?
12:17 Will they thrive in these conditions, et cetera.
12:19 I was able to search for everything, uh, for, for those as well.
12:22 Um, another one was like, uh, I saw this post,
12:25 uh, where like there were a group of like five actors, uh, holding their, uh,
12:29 awards at a, I think it was Golden Globe or one of those awards ceremonies.
12:33 Mm-hmm.
12:34 I was like, I know a couple of them, but I don't know the others,
12:36 and I don't know what they won the awards for.
12:38 And so I just circled all of them and I'm like, Hey,
12:40 can you just tell me in a, in a nice table what,
12:43 what they, who they are and what they won these awards for, and like,
12:47 what should I watch a movie?
12:48 And then it just like constructed a great, uh,
12:51 result for, uh, result for me there as well.
12:53 So lots of different use cases where the user really is asking
12:57 about the whole thing and not just a part of the thing.
13:00 Uh, and now, uh, we are able to do that for users, um,
13:04 but like find the, finding the look is
13:06 one of the most canonical journeys that like,
13:08 kind of gets to, you know, like it really makes the, uh,
13:12 people feel, aha, I get get it,
13:14 but then they end up using it for many different things,- You know,
13:17 Harsh, what's better than talking about the try on tool,
13:20 I guess is maybe showing it to us.
13:21 How about that?
13:23 Yeah.
13:23 Let me just quickly record my screen, uh, so I can show you guys.
13:28 Um, but let's, let's take this example.
13:31 Uh, it's like a example I came across on social media.
13:35 It's a golf look.
13:36 It's, it's a, it's, it's a look
13:37 that I wouldn't completely ever conceive of, like getting,
13:40 but there are parts of it that are very interesting to me.
13:43 So I will, I, so all I need to do is invoke circle to search, which I just did.
13:47 And then, uh, I circle the whole look, and when I circle the whole look,
13:51 I get this a I overview response that allows me to find the look.
13:55 And when I tap on find the look, um, I'm,
13:59 as you can see now, it's actually able to deconstruct that entire look.
14:03 So look at the shirt, look at the shots,
14:05 look at the, uh, you know, uh, the shirt.
14:09 And so now I have, okay, this is what the cap is,
14:12 this is what the shirt is and so forth,
14:14 and I can click on and it found the exact shirt so I can open the shirt,
14:19 um, you know, in, in the viewer and see it closely.
14:22 Um, and then what I'm able to do is, uh, I'm able to, uh, try it on as well.
14:27 So let me take, uh, an example here.
14:30 And now I see a try it on button and when I tap on, try it on, uh,
14:34 it is able to instantly put that shirt on me.
14:36 Uh, and so probably not the thing that I want to wear, uh,
14:40 but at least tells me, uh, you know, like not something for me.
14:44 Um, so that's like the whole journey of like, Hey,
14:46 I found this look, uh, online looks interesting, uh, let me search for it.
14:51 And then, you know, you search for it in two taps,
14:53 we found you something very similar and now you
14:55 try it on and you're like, uh, probably not.
14:57 And then you can move on or like buy- It, you know?
15:01 I'm so curious what it was like to build this feature and then see it progress,
15:05 because of course you need to do a lot of testing.
15:07 Can you tell me a little bit how that works?
15:09 I mean, I presume you've done dozens and dozens and dozens of tryouts,
15:13 uh, while building the product,
15:15 but probably you need more than just the people on the team who are testing it.
15:19 Uh, yeah, like I think the, so, so to, to be able to like do,
15:24 like, find the look, well, uh, there's like a couple of things.
15:27 So, uh, first there's like,
15:30 we need to get a very diverse set of images that have
15:35 all the different types of looks that people might wanna search for, right?
15:40 So, uh, you know, like female influencers, male influencers, younger,
15:44 older influencers wearing different types of outfits,
15:47 jackets, skirts, dress, and so books.
15:50 And so first we need to like just get a very
15:52 broad set of like all the different images and then, uh,
15:55 we have to like build a model to really,
15:58 and make sure that all the different objects are like,
16:01 you know, parts of the look are being deconstructed properly.
16:05 So, uh, early on we saw like, you know, the model will just not,
16:09 won't do a good job act actually searching shoes because shoes
16:12 are generally a very small part of, uh, the whole look.
16:15 I see.
16:15 Um, and so we had to really tune it and say like,
16:18 Hey, there's shoes all there too.
16:19 Like, you have to look at the shoes or handbags.
16:21 They wouldn't, wouldn't pick up handbags or sunglasses.
16:22 They would pick up 'cause there's a very prominent,
16:24 prominent, but it won't pick up these other things.
16:26 And so, uh, that, that was like the second part of like,
16:29 can, can we make sure that it's actually deconstructing that look reliably?
16:34 And then the hardest part, uh, is actually finding, uh, you know,
16:38 visual like basically the results via visual search that are very,
16:42 very similar, you know,
16:44 because what users expect is that exact product or the exact,
16:47 the very similar product.
16:49 And doing that is very hard, especially, um,
16:51 consider a look where you have a shirt and a jacket and a and and you know,
16:56 like sunglasses and, and skirt and so forth.
16:58 And there shirt is partially occluded by the jacket, like we can't even see it.
17:02 And so we get very few visual
17:04 cues to actually re retrieve similar looking shirts.
17:08 And so we really have to do a good
17:10 job of deconstructing that look and finding the, the shirt,
17:13 uh, and then like retrieving other visually similar shirts.
17:16 And then even when we do retrieve visually similar products,
17:21 then users want it to be in their locale that's in stock, that's it.
17:26 That, that, that they can quickly browse prices for, uh,
17:29 from a, from a retailer and merchant that they trust.
17:31 Um, and so there's a bunch of ranking
17:33 and quality problems on top of that to make sure
17:35 that all of these like line like pages
17:37 that we are showing to users are, are high quality.
17:40 And then, so that was the last piece of the puzzle.
17:43 And so like, it took us really breaking down
17:45 the problem into all these different pieces and nailing each
17:47 of ev each and every one of them
17:49 to, to get to the point where the product feels good.
17:52 By the way, if, if someone who is listening
17:54 isn't as much into fashion but is into interior design.
17:58 Yeah, I noticed that the updated circle
18:00 to search is also really useful for that.
18:02 Yeah,- For sure.
18:03 The exact same logic applies to interior design.
18:05 So you come across, you know, like, uh,
18:08 a living room setup that you really love, uh,
18:10 you love the couch, you love, uh, you know,
18:12 the coffee table, uh, the accent chair and so forth.
18:15 Now you just have a circle and say, find the look.
18:17 Now again, we like bake down each of those different parts
18:19 of the living room and search for you something very visually similar,
18:23 but then again, like something that's in stock,
18:26 it near you from a retailer that you trust.
18:29 Um, so you can actually go and, you know,
18:31 actually look at it or consider buying it- Harsh.
18:34 What would be your top tip for anyone interested
18:36 now in circle to search to, to try it?
18:38 Like what's one thing they definitely should
18:40 do with the updated circle to search,-
18:43 Push the product on the kind of things you want to use it for.
18:46 Uh, we, when we started it was really, you know, like, oh,
18:50 you can search parts of the image or parts of something that you're looking at.
18:53 But more and more where we are going is, uh, it's not about a single thing,
18:58 it's actually about the full context of your journey.
19:01 Uh, so this update we are starting to get into, okay,
19:04 you can, uh, ask a question about the full image,
19:07 but then soon enough we are also working on things where we can, you can,
19:10 we allow users to ask a question about the full content that they're looking at.
19:15 So with your permission, you can have us, uh, look at the full PDF,
19:18 let's say if you are on a PDF or the full webpage that you are
19:21 on, or the full video that you're looking
19:23 at, and we can really search through that entire thing.
19:25 So push the product, give us feedback,
19:27 and we continue to strive to make sure that it works best,
19:30 uh, for all your needs.
19:32 Harsh.
19:32 Thank you so much for telling us about the updated circle
19:34 to search now available on select devices like our Pixel 10 series.
19:38 Um, go and try it out.
19:40 Enjoy it and thank you so much and please come back next time.
19:43 Great, thank you Rachid.
19:45 Thank you for listening to the Made by Google podcast.
19:48 Don't miss out on new episodes.
19:50 Subscribe now wherever you get your podcasts to be the first to listen.