Evaluating and improving Replit Agent at scale

0:21 All right.

0:22 Hi everyone.

0:23 I'm Mikael Cata president and head of AI at Replit.

0:27 And today I'm going to be talking about our both

0:29 evaluating and improving on a daily basis Replit agent at scale.

0:34 And as you know, Replit is a back-coding platform for knowledge workers.

0:38 We're one of the top players in the space and we've been

0:41 literally battling with this problem for the last you know almost 2 years.

0:45 The key difference between I would say back-coding has a very broad definition.

0:50 It ranges all the way from being used by software developers.

0:54 But in our specific case is even more of an extreme definition where you start

0:58 from just a natural language specification of what

1:01 the user wants and literally nothing else.

1:04 So the user expects to go from a prompt to a working application,

1:08 but they don't tell us what kind of framework they want to use.

1:11 They don't write tests.

1:13 They just expect things to work after you know our the agent run.

1:18 So it means that a lot of the evaluations and a lot of the ways

1:22 in which we've been building agents in the past

1:24 especially for software developers has to fundamentally change.

1:29 Now, how do we track internally the fact that the agent gets better

1:34 on a daily basis and is actually aligned with all with what our users want?

1:38 This is especially challenging because if you think about it,

1:41 models are changing almost constantly and if anything the pace

1:45 of evolution is even faster than it was a year ago.

1:47 So we are always sit on shifting grounds.

1:51 Models change fast.

1:52 We have to change our prompts, system prompts, user prompts very often,

1:57 you know, to comply to the new features that we're building in product.

2:00 Of course, the tools definitions are changing as well.

2:03 We're adding more of them, we're simplifying, and we're shipping constantly.

2:07 Every single day we have multiple releases of wrap agent,

2:10 which we put in front of, you know, millions of users.

2:15 So, my argument today for this talk will be

2:17 that we have to fundamentally rethink how we do evaluations.

2:21 On the left side, you can see how evals look like for basically,

2:26 you know, since since we've been working on AI.

2:28 Um they you run your evaluation, you they produce a single score,

2:34 and that score is produced by a human.

2:36 And what the human decides based on the score is,

2:39 my agent harnesses better than it was before or it regress,

2:42 or my model is better than before,

2:44 or we should be doing, you know, more post training.

2:46 That's how how we've been doing this, you know, for a for a long time.

2:49 I argue that we should move more

2:51 and more towards a continuous setting of evaluations,

2:54 where especially when you actually have something runs in production,

2:57 and in our case it generates millions of traces every day,

3:01 there is a lot of hot fat to be extracted from the traces themselves.

3:05 So, what we do is we have a combination of fine evals,

3:08 so very similar to sweet bench and other

3:11 benchmarks that are very famous in the space,

3:13 as well as online evaluations based on the on the system usage.

3:17 What do we want our evals to capture?

3:19 Fundamentally, first of all, they need to optimize what our users care about,

3:24 you know, rather than just telling us if the agent is editing code correctly.

3:29 They should also highlight immediately what is breaking,

3:32 and they should inform us where what we should

3:35 be shipping next in terms of improvement for our agent.

3:39 What goes into this box, this black box?

3:42 So, we're building a new system

3:44 that allows us to do these constant improvements.

3:47 The system looks in the following way.

3:49 We have two pillars.

3:51 On one end, we have the standard old-school benchmarks.

3:55 They are usually employed before shipping a new version of the agent.

4:00 Think of them sort of as a boolean flag.

4:02 It's a gatekeeper before deciding if you

4:04 should be launching a new version or not.

4:06 So, if there is a major regression on an offline benchmark,

4:09 of course you stop your release.

4:11 If you see no changes or positive changes, you know that you can go on.

4:15 But the part that is more intriguing and is what is

4:18 helping us to make constant progress is the fact that we do

4:21 a lot of baby testing and we also cluster all the traces

4:26 that we obtain in production and we try to gain insights from them.

4:29 So, in this case, the online pillar actually works after you ship a version

4:34 of the agent and it forces you to react as fast as possible.

4:39 And the loop here is, once I have these two pillars in place,

4:42 you reflect on the results that both of them are giving you,

4:45 you likely change the code once again, run your eval, rinse and repeat.

4:49 And this is the cycle that we go

4:51 we are going through every single day at Replit.

4:55 I mentioned before SweetBench and I want to give

4:58 you a quick understanding why we work on something,

5:02 you know, more powerful than SweetBench or at least

5:04 like more tailored towards our use case.

5:07 Um something as SweetBench, HumanEval, they played a key role in our space,

5:12 you know, to allow us to air climb them and make models more powerful.

5:16 But eventually, they all follow the same protocol.

5:18 Uh you make sure that the code that is generated complies with the user prompt,

5:22 then you apply the patch on your repository, you run the test.

5:26 If the tests are passing, you know,

5:28 you you have a higher score on the benchmark.

5:30 This does not reflect what happens in by coding.

5:33 As I was mentioning before, users are not writing the tests.

5:35 They often start from a completely empty code base.

5:38 So, there is not a scenario where you can just apply patches.

5:41 You're building things from the ground up.

5:43 So, what we need to capture in instead is does the app do what the user asked?

5:48 So, there is a functional correctness gap between

5:52 what SweetBench provides and what we were working on.

5:55 And today on stage, I'm launching ByBench.

6:00 It's a new public benchmark for back-coding end-to-end

6:04 that we worked on at Raptic for several months.

6:07 I'm going to give you all the details after the URL and so forth,

6:09 but let's start from the setup of the benchmark.

6:13 The input is just a PRD.

6:15 It's literally like a long prompt that describes how to build an application.

6:19 Rather than crafting them from synthetic data,

6:22 we pick 20 real-world traces from, you know, usage of Raptic, just plain

6:28 English specifications without any implementation constraints.

6:32 Then, we have an harness that builds the application end-to-end.

6:37 So, it goes from an empty repo into something functional.

6:40 And then, the key insight is rather than stopping the benchmark

6:44 here and then having humans evaluating the output of those PRDs,

6:48 we built automated evaluators.

6:50 This is what allows you to go from a benchmark

6:53 that you run maybe on a weekly basis

6:55 into something that you can run every single time

6:57 you literally have a new PR merge in your repository.

7:00 So, of course, as you can imagine,

7:01 we have AI running all the evaluations on our behalf.

7:06 The the setup is the following.

7:09 There is one core, and the protocol is you have an you have a input,

7:14 you have an implementation strategy, and then you have a set of our core,

7:18 sorry, our wired evaluations that you run on your app.

7:21 And we came up, you know, with five different pairings, but in reality,

7:26 there are a lot more pairings that you could come up with yourself.

7:29 The most basic pairing is the input is the PRD,

7:32 and we build the application single shot zero to one.

7:36 Then, maybe in order, you know, to go from uh less complicated to more advanced,

7:42 we have something called by bond ref

7:43 or like the reference implementation where we

7:46 start from something is already working and we build a feature on top of that.

7:50 Let's make it more advanced.

7:51 We have by bond vibe where we start from an agent

7:55 on MVP and then we build a new feature on top of that.

7:59 Maybe this is something that in the X lingo you would call slop on slop.

8:03 So if we have something that's been built

8:04 by the agent not verified and then you had more agent

8:08 code written on top of that and then you

8:10 run the evaluations to see if it's working or not.

8:12 You can keep making this arbitrarily more complex.

8:16 Uh for example on in agent four that we launched a couple of months ago,

8:19 we completely hide the complexity of doing task decomposition and running

8:24 agents in parallel and then merging all the patches together.

8:27 So we can also take as an input the PRD, we do the task decomposition,

8:32 we run all of that in parallel and then we run the valves after that.

8:36 As you can imagine the parallel plus merge scenario is far more uh you know,

8:40 challenging for a standard coding agent.

8:43 I give you here some more ideas we describing in the paper,

8:45 you know, but you can even start from a buggy application,

8:48 build a feature on top of that and see how much your agent struggles,

8:51 you know, to actually make it make it work correctly.

8:55 So the the key question or the complexity

8:57 here is how do you actually do the grading?

9:00 And that's what I told you before, you know,

9:01 was where we spent most of our effort.

9:04 It turns out that when we're working on our previous version of the agent,

9:07 we put quite a lot of effort in building an automated,

9:10 you know, app testing feature which literally

9:13 doesn't know anything about the app per se.

9:15 Cuz if you think about it, when it comes to web coding,

9:18 you we don't give any level of uh you know, guardrails to our users.

9:23 They can decide the language they want to use.

9:25 We can decide a different framework.

9:27 So the evaluator has to be totally

9:29 agnostic on how the implementation looks like.

9:32 And what our evaluator evaluator agent does is it reads the code base,

9:37 it then opens a browser and points it to the application

9:40 that our agent has built and then step-by-step goes through our testing plan.

9:46 And even the testing plan is expressed

9:47 in natural language and the actions are like open the admin dashboard and log

9:52 in with a certain account and click on this toggle.

9:55 And if any of those steps fails, then you know,

9:57 we collect all of them together and we generate a score.

10:00 So, we go through all these test plan and then

10:03 we decide ultimately if the score is good or not.

10:06 So, this is the key complexity.

10:08 When it comes to SweetBench, we always had a fixed surface.

10:12 We know the repositories, we know the task hardness,

10:15 we know exactly how to make this work.

10:17 In ByBench, the surface is completely green field and that's

10:20 why it took us several months of work to get there.

10:24 Now, as I mentioned, we're announcing it today.

10:28 Here's the QR code.

10:29 You can find the entire benchmark available open source on bybench.ai.

10:34 I'll let you to dig into the paper.

10:36 You know, we it's going to be presented a couple

10:38 of weeks from today at the conference on AI and energetic systems.

10:42 Peter, the you know,

10:43 lead of the project and first author of the paper is sitting here.

10:46 So, after my talk, please come and have

10:48 a chat with him if you want more details.

10:50 I'm just going to give us sneak peek of, you know,

10:52 the most notable results there.

10:54 First of all, we are noticing almost a 2x

10:57 gap between the frontier models and the open weights models.

11:01 And the reason why I want you to pay attention there is how I think it because

11:06 we know that every single model player as every

11:09 single model builder is hit climbing specific benchmarks.

11:12 That's why we release this open source.

11:14 I want the entire community to embrace it across open

11:17 weights and closed weights and make sure that you know,

11:20 we make progress overall on by coding as well.

11:23 The other one, possibly not very surprising is most

11:27 models do a worse job when extending their own code.

11:30 So, the slop on slop or by bomb vibe scenario that I

11:33 was describing to you before is by far the most challenging.

11:37 And that's the reason why I always advocate to have

11:39 a testing step in between every time you create a new feature,

11:43 otherwise you keep building on shaking foundations and eventually

11:46 your vibe coding application is going to fail.

11:50 I'm going to give you back the QR code later on.

11:52 But, you know, let let's move to the to the next step.

11:55 We talk about offline eval.

11:57 I talked to you about vibe bench,

11:59 but as you recall there were like two main pillars that we care about.

12:03 And the reason why we also care about the online one is because there

12:07 is a great difference in terms of volume of signal that we can collect.

12:12 So, on the offline or right now we're working roughly 20 applications.

12:16 The reason, by the way, why we made it open is because we

12:19 are more than welcoming contributions from the community.

12:22 So, more applications, harder prompts,

12:25 we can keep making this benchmark arbitrary harder so that we

12:29 make our traffic's life harder to actually come with over time.

12:32 But, when it comes to Rabbit Engine instead, you know,

12:35 we're we're collecting millions of sessions every day and they're very

12:39 valuable because they capture what our users actually do on the platform.

12:43 And they're completely unscripted, the agent is always running.

12:46 So, how do we distill useful information out of them?

12:50 Well, it turns out that we run a lot of AB tests.

12:54 It's basically our way to keep ourselves honest because

12:58 vibe bench only tells us part of the story.

13:01 And I invite every agent builder out there to start

13:03 to invest on their AB testing infrastructure as soon as possible.

13:07 It's the best way in which you can make steady progress.

13:10 Now, we ask a wide variety of questions for ourselves.

13:14 So, we we built a lot of different instrumentation

13:16 and metric in our agent so that for instance,

13:19 we can find out is there a normal span

13:21 ongoing or is the agent running for longer than expected?

13:24 Or we constantly keep track of the user sentiment.

13:28 It's it's a very easy thing of to collect

13:30 because every time you receive a prompt,

13:31 you can do sentiment analysis on it and then

13:34 receive the user is getting frustrated or not.

13:37 And last but not least, this is very idiosyncratic to Rapid.

13:41 Our users can also publish their application on our product.

13:44 So, it's a very strong positive signal

13:47 if they decide that whatever they built is

13:49 worth to share with their colleagues or to put

13:51 it in public in front of everyone.

13:53 So, all these signals clustered together gives us something like this.

13:58 If you have ever done an AB test in your life, I'm sure this looks familiar.

14:02 And it might look familiar especially because

14:05 not everything is either green or red.

14:08 This is the, you know, hard harsh truth about running an AB test.

14:12 They never give you a crystal clear signal of what you should be doing.

14:15 So, in this case for example,

14:18 uh the average run duration of our agent, you know,

14:20 went up by 7% but conversely is 8% cheaper and we

14:25 saw the fluctuation in terms of positive and negative sentiment.

14:29 What shall we do in this case?

14:31 This is where human taste and product philosophy still plays a key role.

14:36 You will never get or very rarely do we get like

14:38 a crystal clear configuration of the results of your AB test.

14:43 And to make our life a bit easier on how to generate these AB test candidates,

14:49 what we do is we cluster all the traces that we receive every single

14:53 day and night as you can imagine and we try to identify first of all,

14:58 which cluster are capturing normal nominal behavior of the agent.

15:03 And as you can imagine, there are plenty of them because the vast

15:06 majority of times the application is actually

15:08 successful but of course there is a long

15:10 tail of problem that it happens to surface.

15:13 In case we find a problematic cluster, then what do we do?

15:16 We embed all the failure summaries.

15:19 We we cluster them by type so that we find out

15:23 the different failures that are happening at a certain point in time.

15:26 Then we pass them through an LLM to classify exactly what is going on.

15:30 And the fact that we do this not based on regex on logs

15:34 or like very deterministic techniques is

15:37 what actually makes a difference because you

15:39 will be able to cluster things that are semantically close to each other

15:43 even though the agent doesn't give you always the same type of output.

15:47 And you know, once once we find these uh cluster configurations,

15:52 we are forced to retrain them every single night because

15:55 we are running several versions of the engine in parallel.

15:58 We ship many versions every single day,

16:00 so we can't live with a fixed cluster configuration.

16:05 And aside, you know, from also helping us to identify the failure modes,

16:09 another interesting feature is the fact that once we retrain those clusters,

16:14 we can go back and see if after we believe we fixed a problem,

16:18 a certain cluster is actually disappeared.

16:20 So, say you have a certain tool failure

16:22 that is happening only a limited amount of time.

16:25 So, not something that will show up in your DataDog dashboard, for example,

16:29 not a failure that happens 50% of the times,

16:32 but maybe it was happening 1% of the times in certain specific conditions.

16:37 And from here, you will start to realize, okay,

16:39 I shipped my new PR and that cluster is disappeared.

16:43 So, you start to have some evidence that you mitigated the problem.

16:48 The The technology that we built internally was

16:53 called Telescope and it's a fairly simple loop,

16:56 I hope, after I explain everything.

16:58 Uh first of all, we start by discovering the problems.

17:02 And this is based on the trace clustering that I was talking about before.

17:06 Then, based on that, we create code changes.

17:09 So, it's completely automated.

17:11 We basically cut PRs with a coding agent

17:14 based on the information we got from the trace,

17:16 based on information we collected from the logs,

17:18 from all the different different dashboards that we had.

17:21 Then we evaluate if the change we generated,

17:25 first of all, is is is a breaking change.

17:27 So, we rerun by bench.

17:29 By bench is kind of like a litmus test.

17:31 If the score drops by 10 points, we decide that the change is bad.

17:35 If it's a controversial change that we believe could

17:38 affect also negatively our agent, we run an AB test.

17:42 So, we put it in front of the users

17:44 and we try to find out what are the trade-offs.

17:46 If it's a clear sure shot, then we just ship it in production.

17:51 And few times when we believe that the hypothesis

17:55 was correct and maybe the PR was not perfect, we keep iterating on the problem.

18:00 Maybe maybe we run another AB test until we are

18:03 in a good configuration and then we actually ship it.

18:06 If you think about it,

18:06 this is very similar to the work you do as an AI engineer all the time,

18:11 but 90% of it is now aided by an agent in the loop,

18:15 you know, rather than you having to sweat

18:16 about every single step of the process.

18:20 Let me give you all an example of, you know,

18:22 what we actually experienced, you know, once in production.

18:25 Um Rabbit Agent immediately starts an execution

18:29 environment the first time you submit a prompt.

18:31 And of course, to set up everything,

18:33 especially with the level of complexity we built,

18:35 requires, you know, quite a few seconds.

18:37 We had a degradation in the long tail where our, you know,

18:40 setup time was longer than expected and our agent was

18:44 ready to roll before the environment was fully set up.

18:48 As you know, agents have become really eager to fix problems.

18:51 So, Rabbit Agent went on a tangent and started to try to fix the environment,

18:55 you know, on the spot.

18:56 Um it turns out that by the nature of agents not being deterministic,

19:02 every single debugging session looked a bit different.

19:05 So, if it tried to find out this long tail problem just by grabbing the logs,

19:09 it would have almost not appeared.

19:11 But the moment we cluster all the traces, you know,

19:14 with basically with we we through the semantic layer,

19:17 then we started to figure out that this problem was happening quite often,

19:21 and we immediately able to create a patch and fix the issue.

19:25 In this case, we didn't have to run

19:27 an AB test because it was a pure regression case.

19:30 But in case it was a problem that required testing,

19:32 of course, we would have put an additional step in between.

19:37 And even though I'm a big fan of trying to optimize

19:41 as much as possible the life of an AI engineer with agents,

19:45 I do think that there is a lot of intellectual work to be

19:49 done by humans still in the kind of loop that I presented to you.

19:53 For example, whenever we find one

19:55 of these clusters that I was talking about before,

19:58 as a team, we always focus on formulating

20:00 an hypothesis on what could be possibly go wrong there.

20:04 Cuz the truth is when running in production,

20:06 you're going to get a really a large

20:08 amount of candidates of what you could be fixing.

20:10 So, you need to prioritize where you put effort, where you put resources.

20:14 So, if the hypothesis is intriguing enough, then we move forward,

20:18 and we don't just let the agent write a PR without any kind of supervision.

20:23 We actually try to feed some of our understanding of what is wrong.

20:26 So, we give it more guidance.

20:28 And ultimately, if you think about it,

20:31 every single time we decide of working on a PR and doing an AB test,

20:35 we're sort of like shaping the hill that we try to optimize.

20:39 Cuz if we only care about, say, making the product more affordable,

20:43 we're going to go and try to fix all the anomalous spend problems,

20:46 and then we're going to do our best to optimize that as much as possible.

20:50 So, all these choices really determine the product philosophy

20:54 of what you want to put in front of our users.

20:57 And then, last but not least,

20:58 as I was showing you before on the AB test dashboard,

21:01 if there is not a clear result, ultimately,

21:05 if you decide to launch or not, is still a choice that a human does.

21:09 Often times, that's that's on me in the rabbit.

21:12 And again, that determines really what's

21:14 going to be the ultimate product experience.

21:18 So, my In closing, what I want you to have as a takeaway

21:21 today is don't think of evaluation just as this last check before shipping.

21:26 It shouldn't be just a boolean flag,

21:28 but rather think of this as an engine that allows

21:30 you to ship a better agent every single day.

21:34 Thanks, everyone.

21:35 And I'm going to invite Hannah on stage so we can have a chat

21:38 about how we've been working together in the last few months to go.

21:40 Thank you.

21:43 [applause] Awesome.

21:47 So, my name is Hannah.

21:48 I'm part of the Applied AI team at Anthropic,

21:51 and I work with Michele on the Replit

21:53 agent and on everything that they build with Claude.

21:56 So, I want to ask you a few questions about what you built here.

21:59 My first question is Vibez is something that we've seen a couple times.

22:03 You've used it with us to give

22:04 us feedback on research models that we're testing.

22:07 But, what was was your vision from the beginning to make something

22:10 that you were going to open source and make available to the community?

22:13 I think a lot of people may be trying to build evals

22:16 like this for themselves where they see holes in the public benchmarks,

22:19 but I think this is kind of a unique approach,

22:21 and I'm just wondering if you could talk about that a little bit.

22:24 Yeah, I've been asked this often,

22:26 why don't you keep this as a private appeal of your company?

22:29 Fundamentally, I believe that we should try to give

22:32 back as much as possible to the community.

22:34 And we all have to benefit from creating public evals.

22:37 It helps you to make better models.

22:39 It helps us to make a better agent.

22:41 It helps everyone to create better products.

22:43 And I I don't believe in competing on evaluations.

22:46 I come from a research background where everything should be open.

22:50 So, we'll we'll keep doing this for as long as we can in Replit.

22:53 And And again, I I really invite people to collaborate on Vibez.

22:58 I I tried to instill this idea in the community

23:00 several times when I was giving talks in the past

23:02 and at a certain point I realized we have to build

23:05 it ourselves rather than just trying to invite others to do it.

23:08 Mhm.

23:08 Well, I'm really excited to see what people do with that.

23:11 Um I also want to ask you about the second pillar, telescope.

23:15 This is um you know, a pretty sophisticated system that you've built.

23:19 It sounds incredibly useful.

23:21 It sounds like something you've tuned quite a lot over time.

23:24 For people in the room who might be trying to build

23:27 something similar like this, what lessons did you learn along the way?

23:31 What kind of tips could you give them

23:33 about how they can build something useful like this?

23:38 So, first of all, if you try to do this even, I don't know,

23:41 6 months ago and you got discouraged because

23:43 you didn't get like a return on your investment,

23:46 uh definitely try it try it again.

23:48 The same inflection point that you experienced with coding agents back

23:52 say in November with Opus 4.5 and and similar frontier technology,

23:57 uh it also reflects to what I just talked about now.

24:01 So, the the fact that long context and models

24:04 that are really capable of reasoning on a lot of content

24:07 means that you can practically inject an entire trace

24:10 of your agent into Opus and get fairly sophisticated feedback about it.

24:15 Um so, you at the same time you should really be investing in collecting

24:21 all the signal that you can from your agent and as well as you know,

24:24 the environment where you're executing it.

24:26 So, I know that I I had to run fast during the talk, you know,

24:29 but I couldn't go in depth on all

24:30 the different signals that we bring into telescope.

24:33 It is not purely what's coming from the trace,

24:36 it's also the feedback that we collect in product.

24:39 Like we have a feedback form,

24:40 so every time the agent doesn't behave correctly and the user complains,

24:44 we have the user point of view, we have the trace of the agent,

24:48 we have everything we instrument in the platform in DataDog.

24:52 Unsurprisingly, all of that together, put, you know, in context,

24:55 helps even more to pinpoint where where the issues.

24:58 Yeah, I think that idea of building the clusters,

25:01 like you were saying that a single trace,

25:03 it's very hard to debug what might be going wrong there,

25:05 but when you are able to group them all together and get

25:07 these other signals in, really reveals

25:10 a lot more information and something actionable.

25:12 Makes a ton of sense.

25:13 Yeah, it does and I think it also makes the life of an agent builder

25:17 less overwhelming because it really is when you start to get a lot of usage,

25:22 the amount of feedback you receive is actually overwhelming.

25:25 And it's a good It's a good signal.

25:26 It means that people care about what you're doing,

25:28 but at the same time it's hard to prioritize

25:31 what's actually important versus what doesn't move the needle.

25:34 What I showed today is also a way for us to try to understand, okay,

25:38 this cluster keeps showing on a daily basis and it's

25:41 fairly large and it has a lot of volume,

25:43 so we should actually fix that first of all and then there is the long tail

25:46 of problems to fix will always be there

25:48 with agents given the fact that they're non-deterministic.

25:51 Yeah, that really makes me think about the third thing I wanted to ask you,

25:54 which is what you said about taste.

25:56 Uh, I think this is really interesting point um that the taste of the AI

26:00 engineer is very important and could you

26:02 talk about how your team develops that taste?

26:06 Have you always had this taste or have you improved your taste over time?

26:10 Like what's the process of building that and developing

26:12 that for people who might be trying to build their own taste?

26:16 I think we all brewed it over time because you know,

26:19 when we launched these, you know, was more than a year and a half ago,

26:22 agents were way less powerful than they are today.

26:25 Uh, maybe not much taste to be applied.

26:27 It was more like a survival game just to make sure it was barely functional.

26:31 And now that they're becoming so powerful and you

26:34 have such a wide variety of choices you can make,

26:37 um then you you start to develop a taste

26:39 that should really be in tune with your actual user base.

26:43 Like if if you're building an agent for software developers,

26:46 maybe 80% of the choices that we make every day

26:49 in the team will be almost the polar opposite of what we do.

26:53 And you always want to keep in mind

26:55 that it's being used by personal likeness different from you.

26:58 Especially in our case,

26:59 like everyone at Rabbit is very technical and we are giving the product

27:02 to knowledge workers who have never written a single line of code.

27:06 So, it's a it's a very good trade-off to to learn.

27:09 Awesome.

27:09 Thank you for sharing that.

27:11 I'm sure people are going to have lots of questions for you and for Peter,

27:14 who is one of the authors of Vibrent.

27:15 So, everyone please come find them and thank you so much for coming for

27:19 Thank you.

27:20 Have a good one.