Gemini 3.1 Pro and the Downfall of Benchmarks: Welcome to the Vibe Era of AI

0:00 The latest, and some would say greatest,

0:03 AI model has just been released, Gemini 3.1 Pro.

0:07 And in the 24 hours since release, as well as a short period of early access,

0:13 I have tested it hundreds of times, and of course read its model card.

0:16 But here's the thing, for the average user,

0:18 I want to get beyond the headline scores,

0:21 and try to give you a sense of why every new hot take you see on X,

0:25 or YouTube, or TikTok, or podcast seems to contradict the last one you saw.

0:31 Because there's actually a technical reason

0:33 for the confusion over which model is best overall.

0:37 But I will say that there's one private benchmark, my own,

0:40 that has recently seen a model pass

0:42 a threshold that I think is worth talking about.

0:45 First 30 seconds of context,

0:46 because you may well know that the pre-training stage

0:50 of growing or training LLMs involves training them on internet-scale data.

0:54 But that actually now only accounts for 20%

0:57 of the compute that is spent on training LLMs.

1:00 So, it's the post-training stage, as I wrote about in my newsletter,

1:04 where those generalist base models are

1:06 honed against internal benchmarks on specific domains.

1:09 This includes using industry-sourced data to get

1:11 particularly good at perhaps your domain.

1:14 Here's the catch, just a year ago, that wasn't the case.

1:17 Dario Amodei, CEO of Anthropic, said back then,

1:20 "The amount being spent on the second stage,

1:23 RL stage, is small for all players." Why did I give you that context though?

1:27 Well, because if one of these labs have data relevant to your domain,

1:30 and post-train their models to optimize for high scores in that area,

1:34 then your experience of that model might

1:37 be quite different to what other benchmarks say.

1:39 In the older paradigm, if a model was clearly better in one domain,

1:44 it was much more likely that they'd be better in many or all domains.

1:48 That just isn't the case anymore.

1:49 In fact, my second point in this newsletter was precisely an example of that.

1:54 Many of you would have heard

1:55 about the intense discussion surrounding Claude Code,

1:58 and all sorts of Claude-powered agents that are now sweeping the web.

2:01 So, we're seeing exponential improvement, right, across the board?

2:03 Well, let's take one chess puzzle benchmark made by Epoch AI,

2:07 and more on them later.

2:09 Five months ago, Claude Sonnet 4.5,

2:11 which is their smaller model compared to Opus, scored 12%.

2:15 But just last week, Claude Opus 4.6, five months further on, scored just 10%.

2:21 That's not to knock Claude Opus 4.6,

2:24 I use it all the time, it's an incredible model at coding.

2:26 And of course, if the AI labs

2:28 wanted to improve this performance, they easily could.

2:30 I think GPT 5.2 on extra high gets around 50%.

2:34 You could say chess is a fairly pure

2:36 measure of a general kind of forward-thinking reasoning prowess.

2:39 Back in the generalist era of AI, you would definitely expect chess performance

2:44 to translate to all sorts of other domains.

2:46 We just not in that paradigm anymore,

2:48 it's going to depend on the domain you're in.

2:50 None of this is to say that Gemini 3.1 Pro isn't an incredible model, it is.

2:55 In almost any domain you care to measure,

2:57 it will be competitive with the best other models,

3:00 like Claude Opus 4.6, or GPT 5.3.

3:03 But you would be understandably slightly confused to see

3:06 it being better in all sorts of coding benchmarks,

3:09 measures of scientific reasoning,

3:11 and academic reasoning, like GPQA Diamond and Humanity's Last Exam respectively,

3:15 as well as general pattern recognition, ARC-AGI-2, and I'll come back to that.

3:20 But yet, in a head-to-head on GDP val,

3:22 which is a broad measure of expert tasks that human

3:25 professionals do that I've covered many times on the channel before,

3:27 it falls seemingly quite far behind Claude Opus 4.6, and even GPT 5.2.

3:32 Now, yes, one big explanation of that is

3:34 the domain specialization I talked about earlier.

3:37 But there are three or four fascinating bits of context

3:40 that I want you to be aware of in addition.

3:42 First, let's zoom in to ARC-AGI-2,

3:45 in which its score of 77.1% puts it way ahead of Claude Opus 4.6,

3:50 which is the more expensive model, which got around 69%,

3:53 I start with this one because Demis Hassabis, CEO of Google DeepMind,

3:56 featured it prominently in his Twitter post

3:59 announcing the launch of Gemini 3.1 Pro.

4:01 And on puzzles that shouldn't be in its training data,

4:04 the Gemini 3 series outperforms all other models on a cost-efficient basis.

4:09 But the first additional caveat comes from Melanie Mitchell,

4:11 a famous AI researcher and professor.

4:14 She pointed out that if they change the encoding

4:16 from numbers to other symbols, accuracy goes down.

4:19 Digging deeper, the group found that the numbers

4:22 representing colors in the input can be

4:24 used by LLMs to find unintended arithmetic

4:27 patterns that can lead to accidental correct solutions.

4:31 I wouldn't call that the models cheating,

4:32 they're using any shortcuts they can find to get the correct solution.

4:35 Fair's fair.

4:36 But it does remind us that even within a benchmark,

4:39 how you set up the question matters.

4:41 Okay, well, let's say you don't care about ARC-AGI-2,

4:43 or Simple Bench, or any other benchmark, just coding performance.

4:46 Well, the creator of ARC series, the ARC-AGI tests, François Chollet,

4:50 has this to say, "Sufficiently advanced

4:52 genetic coding is essentially machine learning.

4:55 A goal is given to the agent or agent swarm,

4:58 and then the coding agents iterate until the goal is reached.

5:01 As in other areas of machine learning, the result is a black box model.

5:04 You have a code base that performs the task,

5:07 but you don't necessarily inspect the internal logic.

5:10 Just like how Gemini 3.1 may have found

5:12 spurious patterns in ARC-AGI, in your code base,

5:15 Claude or Codex may overfit to the spec,

5:18 or may drift from your original concept.

5:20 So, the fallibilities presented in this video are relevant

5:23 to you even if you only care about coding,

5:26 or letting your open Claude agents code for you.

5:28 Gemini 3.1 Pro indeed hit a record Elo in live Code Bench Pro,

5:33 which involves competitive coding problems.

5:35 That's great, but you can turn that optimization dial a little too far.

5:39 Let me show you what happened last night

5:40 when I used Gemini 3.1 Pro inside Cursor.

5:43 How could we reconcile these pages of paplum with the record-breaking Elo?

5:48 Well, again, that's the theme of the video.

5:50 If I sound unduly skeptical of Gemini 3.1 Pro, by the way,

5:53 let me try to balance that out with heaping on some praise.

5:57 On my private Simple Bench, a test of, you could say,

5:59 trick questions or common sense reasoning,

6:01 it beat its own previous record from Gemini 3 Pro, and got 79.6%.

6:06 That essentially brings it within the margin

6:08 of error for the human average baseline,

6:11 at least among the nine participants that we used.

6:13 And I do want to spend just 60 seconds

6:15 on marking the threshold that I think this represents.

6:18 All the time on podcasts and in articles,

6:21 you hear about AI models being compared to professionals and experts,

6:25 and phrases like superintelligence being

6:26 bandied around and recursive self-improvement.

6:29 But what about comparing models to the average human?

6:32 Now, sure, of course you can find audio or visual puzzles

6:35 that they will still fail at that the average human wouldn't.

6:37 But in English, in text alone,

6:40 I think it's worth marking the moment wherein I don't

6:44 think you can write a test at which the average human,

6:47 the average man or woman on the street,

6:49 would clearly outperform frontier models.

6:52 I'm not talking about exploiting tokenization bugs,

6:54 like how many Rs in strawberry.

6:56 I'm talking about a fair text-based test in English with a non-specialist human.

7:01 Let me know if you [clears throat] disagree,

7:02 but I think the passing of that threshold is a moment worth marking.

7:07 I will note that even with Simple Bench,

7:09 we get a reminder of the caveat I was just describing.

7:12 Models are brilliant at shortcuts.

7:15 And I had noticed, as long ago as I think at least 12 months,

7:19 that because Simple Bench was a set of multiple-choice questions,

7:22 sometimes one of the answers being, for example, zero,

7:25 would flag to the model that, "Wait,

7:27 this might be a trick question." For example,

7:30 if we go to try yourself, even in question one about frying eggs in a pan,

7:36 the fact that there may be zero ice cubes left in the pan,

7:40 even as just one of the options,

7:42 may alert the model to think, "Hang on, how could there be zero?

7:45 How would that be possible?" So,

7:46 what happens if you take away the multiple-choice questions,

7:49 get the models to answer in an open-ended fashion,

7:52 and then get a blind grader model

7:54 to compare their answers to the hidden correct answer?

7:57 Well, you still get some pretty impressive scores, but just not quite as high.

8:01 Call it a 15 to 20 percentage point drop.

8:04 That's, by the way, a double reminder.

8:06 Yes, models are taking shortcuts.

8:08 Yes, if you ask the same question in a different way,

8:10 performance may well be different.

8:12 But it's not like performance dropped to zero.

8:14 Frontier models are genuinely getting better,

8:16 even in domains they didn't directly train on.

8:19 Time for the next big caveat before we return to the glory of the exponential.

8:23 Let's take a look at the brand new Gemini

8:25 3.1 Pro and Anthropic's Claude Sonnet 4.6 from this week.

8:30 How do they do in terms of hallucinations, or factual accuracy?

8:33 You'll notice that model providers don't often

8:35 want to talk about or measure hallucinations anymore,

8:38 because that was predicted to be a solved problem by now.

8:41 And on this release chart from Google,

8:43 there wasn't a direct measure of hallucinations.

8:46 But in fairness, they did cite

8:48 this benchmark from Artificial Analysis, AA Omniscience.

8:52 And on first glance, Gemini 3.1 Pro seems to shellac the other models.

8:57 Gemini's top score of positive 30 compares

8:59 to Claude Opus 4.6 getting positive 11,

9:02 and Claude Sonnet 4.6 getting negative four.

9:05 And that's even accounting for penalizing hallucinations,

9:08 as well as rewarding correct answers.

9:10 However, if we zoom in on just incorrect answers,

9:14 and whether the models hallucinated an incorrect answer or explanation,

9:17 versus refused to answer or admitted to not knowing the answer,

9:21 Gemini 3.1 does well at 50% of its incorrect answers being hallucinations,

9:26 but Claude Sonnet 4.6 is down at 38%, which is better.

9:31 Interestingly, GLM-5, a Chinese model, is even better at 34%.

9:35 So, hallucinations is definitely not a solved problem,

9:38 and just because a model is optimized or better at its best,

9:42 does not preclude the possibility that it's worse at its worst.

9:46 What's that saying?

9:47 If you can't take me in my bad moments, you don't deserve me in my good moments.

9:50 Well, for all models, you're going to have to handle that kind of trade-off.

9:53 One quick note on the model card for Gemini 3.1,

9:57 it's only nine pages and has ever these model

10:00 or system or safety reports will serve the purpose

10:02 of de-hyping when the release post by the CEO

10:06 or release video serves the purpose of hyping.

10:09 For example, let's focus on Gemini 3.1 in the cyber domain.

10:12 Well, if you are an ultra subscriber, you can use deep think mode and Google's

10:16 model card says this, "Accounting for inference costs,

10:19 the model with the deep think

10:21 performs considerably worse than without deep think,

10:24 even at high levels of influence, results for the model with deep think do

10:27 not suggest higher capability than without deep think." Okay,

10:31 that's deep think mode, which we might discuss another time,

10:33 but what about just 3.1 Pro?

10:35 Well, back to that specializing in individual domains,

10:37 they found out that in one test of machine learning and R&D,

10:41 optimizing the LLM boundary involving fine-tuning,

10:44 that 3.1 Pro could indeed reduce the runtime

10:47 of a fine-tuning script from 300 seconds to 47 seconds,

10:51 better even than the human reference solution of 94 seconds.

10:54 Whereas you might have read that as meaning that it's

10:58 now going to accelerate its own self-improvement through machine learning R&D,

11:02 you might now more so interpret that as being,

11:05 "Oh, they added in some new fine-tuning data, data about fine-tuning,

11:09 or an internal benchmark measuring fine-tuning

11:11 performance." But enough with the caveats.

11:13 What do all of these models from the last few weeks,

11:15 including Gemini 3.1, show about what we're about to unleash on the world?

11:19 Because many of the exponentials you do see are real and meaningful.

11:23 But first, the sponsors of today's video,

11:26 Epoch AI, because they recently featured,

11:28 just yesterday, one exponential that you may not have heard of.

11:32 That's that Anthropic's annualized revenue is 10x'ing

11:35 per year until the tail end of 2025,

11:38 whereas OpenAI's is 3.4x'ing per year, albeit starting from a bigger base.

11:43 It's a big if, but if those trends continue,

11:45 by mid-2026, we could see Anthropic out-earning OpenAI.

11:50 And as you might have guessed, Epoch AI's research is one of the main ways

11:54 that I stay on top of AI research and developments.

11:57 I've been covering them for years,

11:59 even before they were a sponsor, and their newsletter is also incredible.

12:02 If you want to learn what powers some of these exponentials,

12:05 of course, you'll have to focus on their frontier data center analysis.

12:09 I had to actually double-check with them that it was free.

12:12 I just couldn't really believe it, but it is.

12:14 Check out the unique link in the description.

12:17 Back to the central question

12:18 of whether benchmark performance measures general intelligence.

12:21 Because I have given you lots of counterarguments,

12:24 but Dario Amodei, the CEO of Anthropic,

12:27 did raise a point the other day that really

12:29 gave an insight into the bet that Anthropic is making.

12:32 He was asked, "Why do you need all

12:33 these RL environments specializing in Slack, for example,

12:36 or browser use?" Surely all of that's redundant

12:39 if models are going to keep getting generally smarter.

12:41 Amodei said this, "Yes, we're trying to get a whole bunch of data,

12:45 not because we want to cover a specific document or specific skill,

12:49 but because we want to generalize.

12:51 For me, this is critical." Because what he is

12:54 almost saying is that if you specialize in enough specialisms,

12:58 you'll generalize to all specialisms.

13:00 This is why later in the same interview,

13:02 he said that we can get most of the way there to a AGI

13:07 or superintelligence or country of geniuses

13:09 in a data center without continual learning,

13:11 without learning on the job, without you teaching the model about your domain.

13:15 How could you get to superintelligence without that data?

13:18 Again, in my words, I think he

13:19 thinks that if you specialize in enough specialisms,

13:22 there are only so many patterns to be deduced from human training data.

13:26 Yes, they're going to work on continual learning in case that's not the case,

13:30 but if it is the case, Anthropic might not need the data from your domain.

13:34 Or maybe he says later, models will almost get there,

13:37 but just need a bit more context about your domain in the context window,

13:41 in the prompt you give it.

13:42 This is why he says, "One idea they've got is just to make the context longer.

13:46 There's nothing preventing longer context from working.

13:48 You just have to train at longer context and then

13:51 learn to serve them at inference." In other words,

13:53 there might be a little bit of nuance

13:54 in your domain that this generalist model doesn't know.

13:57 Even after specializing in all the specialisms,

13:59 it might only generalize so much and need a bit more context from your domain.

14:03 But Claude 4.6 can now absorb 750,000 words in its context window.

14:08 In short order, that might be a couple million words.

14:10 Maybe that's enough specific context from your domain

14:13 for the model to do the rest.

14:15 It can learn the patterns it needs in context, in context learning,

14:18 and finish off what it needs to do

14:20 in marketing or software engineering or back office automation

14:23 or data analysis or finance and accounting

14:25 as this chart of its activities and tool calls show.

14:29 Amodei brought the conversation back to coding.

14:31 He said, "I don't think people would say that learning on the job,

14:33 continual learning, is what is preventing

14:35 the coding agents from doing everything end-to-end.

14:38 They keep getting better.

14:39 No, they didn't train on your code base,

14:41 but they were better than you at improving it anyway," he's saying.

14:45 That question then of to what extent do you

14:48 need to train on all the different domains and subdomains

14:51 versus generalized patterns across them will be one [clears

14:54 throat] of the central questions of 2026 and 2027.

14:57 If you want a deeper analysis on some of Amodei's

15:00 points and some context behind the famous meter time horizon exponential,

15:04 then do check out this recent post on my Patreon.

15:07 Just one more word then on your hunt

15:09 for that one true benchmark that measures general intelligence.

15:12 The giveaway perhaps as to whether Amodei's bet is correct or not.

15:16 Well, who would be the people most incentivized to have such a benchmark?

15:20 The labs themselves.

15:21 Because then they could do reinforcement

15:23 learning with verifiable rewards on that benchmark.

15:25 If there was a pure general intelligence benchmark,

15:28 they could optimize against it and have the most generally intelligent model.

15:31 Many of these benchmarks come from small teams with sub-million dollar budgets.

15:35 But expecting those small teams to craft a benchmark that objectively

15:38 captures real-world performance without overestimating it is expecting a lot.

15:43 As one researcher at Meta said,

15:44 that essentially means making more realistic reinforcement learning

15:47 with a verifiable reward settings than labs, which is hard.

15:51 That's why so many of the benchmarks now are written by the labs themselves.

15:55 They're the only ones with the heft and budget to craft such benchmarks,

15:58 although obviously that makes them somewhat biased.

16:01 Now, this probably deserves its own video,

16:02 but there is of course one truly objective benchmark,

16:04 which is forecasting the future.

16:06 And Metaculus have noted that the predictive

16:08 performance of models is rising significantly.

16:11 It's almost at the level of an average human forecaster from Metaculus,

16:15 not quite at pro level, but getting there.

16:17 But I did have a quick side note on that.

16:19 You've probably noticed how the world is going,

16:21 unfortunately, wild for Polymarket and predictive markets.

16:25 I get the basic idea, but of course,

16:27 for many people, it's just akin to gambling.

16:29 But here's the question, what happens when there's so many open claw

16:33 agents out there that a model can just make

16:36 money by changing something and making a prediction

16:40 on it at the same time, gaming the system?

16:43 How long until we get the first unfiltered open claw model that takes

16:46 action in the real world to make it money on a prediction market,

16:51 money which is perhaps fed back to the human controlling the model?

16:53 In other words, even forecasting the future,

16:56 which seems the purest benchmark of them all,

16:58 is still slightly in the future susceptible to gaming.

17:02 And finally, for this video, of course,

17:04 there are benchmarks that are nothing to do with those we've discussed so far.

17:07 What about speed as a benchmark?

17:09 Check this out.

17:10 My question is, "Tell me about Simple Bench,

17:12 the LLM benchmark." And I'm recording live, watch what happens.

17:16 Boom.

17:17 Full answer.

17:18 Unbelievable tokens per second.

17:21 This is a model made for a chip, but that's a discussion for another day.

17:24 I just think this portends a future where

17:27 entire apps are created in a single millisecond.

17:30 Then there's the benchmarks or questions that you may have that no one else has,

17:33 and that's what I use LLM Council.ai for.

17:36 That's my own site, of course, and it's free currently to compare the responses

17:39 of say Gemini 3.1 Pro and GPT 5.2.

17:43 Lastly, what about bloody realism as a benchmark?

17:45 Of course, it would have been hard for you to have

17:47 avoided the talk of C Dance 2.0 from China's ByteDance.

17:51 But I did think it was worth ending the video with maybe

17:53 45 seconds where you can see how it is a step up, I think, from V03.1 or Sora 2.

17:58 If you're watching the video, you're seeing a video from C Dance 2.0,

18:02 but even if you're listening, here's a comparison between,

18:05 again, C Dance 2 and V03.1.

18:08 I'm so proud of my family.

18:11 My sweetie, you're the heart of our family.

18:14 Folks, let's dance together to celebrate.

18:17 I'll bring the music.

18:18 I'll bring the music.

18:21 [music] I'm so proud of my family.

18:26 My sweetie, you're the heart of our family.

18:29 Folks, let's dance together to celebrate.

18:32 I'll bring the music.

18:34 In this domain at least, the difference seems obvious to me.

18:37 So, there we have it.

18:37 Of course, DeepSeek V4 is just around the corner,

18:40 but what did you make of this video and the swirling

18:43 ongoing debate about capturing the true general intelligence of models?

18:46 Thank you so much for watching to the end and have a wonderful day.