DeepSeek Just Fixed One Of The Biggest Problems With AI

0:00 Few people know, but modern AI systems are really silly.

0:04 How?

0:04 Well, imagine having a Michelin-star chef being

0:07 asked for a simple peanut butter sandwich.

0:11 That’s weird, but okay.

0:12 Now, the chef says, you’ll need to wait just a bit,

0:16 because I am going to start planting peanuts,

0:19 wait six months, harvest, churn some peanut butter,

0:22 and then get to work on your bread.

0:25 That sounds really silly,

0:26 and that is exactly what modern AI systems like ChatGPT and Gemini do.

0:32 When they need to recall a simple fact,

0:35 like who Alexander the Great was, something crazy happens.

0:39 They go through complex reasoning layers

0:42 and reconstruct everything from scratch every single time.

0:47 That is crazy.

0:48 Now I have an amazing research paper for you here from folks at DeepSeek AI,

0:53 and this is a piece of technology that might underpin most

0:57 if not all of the amazing AI systems of the future.

1:02 Now every now and then we are going to look at a figure, but for the rest,

1:06 I am going to bring my physics simulations

1:08 and all the goodness we talk about around here.

1:12 Apologies for that.

1:13 Okay, so this is a massive waste of compute.

1:16 But why does this happen?

1:18 Well, standard transformers are a kind of neural

1:21 network that is inside nearly all modern AI assistants.

1:25 And here is the problem:

1:27 they lack a simple and cheap way to just look things up.

1:32 Whatever the question is,

1:33 the answer is a huge bunch of dense mathematical calculations.

1:38 From scratch.

1:39 Yes, it is literally planting that peanut when you ask for a sandwich.

1:44 Now in this work, DeepSeek introduces Engram.

1:47 With this, they are giving our tired little chef a pantry.

1:53 Cutting edge technology brother!

1:55 Instead of growing that peanut butter sandwich from scratch,

1:59 it now just grabs the ingredients from the pantry.

2:02 I’ll explain to you how exactly they did it in a bit.

2:06 Now, this makes the AI way more efficient, okay, I expected that.

2:12 But, what?

2:13 Are you seeing what I am seeing?

2:16 Now this I did not expect at all.

2:18 So here comes the surprise.

2:20 Now hold on to your papers Fellow Scholars,

2:23 because when taking away some of the AI's complex reasoning parts,

2:28 known as mixture of experts, MoE.

2:31 Then, replacing it with the pantry actually makes the AI smarter.

2:36 Lower is better here on the loss curves.

2:40 And not just a little, this is significantly smarter.

2:43 Those dots dipping way down show that this hybrid

2:47 chef makes far fewer mistakes than previous techniques.

2:51 It achieves a perfect balance of active

2:54 cooking and just grabbing from the pantry.

2:57 Genius.

2:58 But that is not the only surprising thing in this paper.

3:01 They also added a way for the AI to check the ingredients before using them.

3:07 You don’t want rotting fish in your strawberry jam.

3:10 To ensure this, they created a context-aware gating mechanism.

3:14 The current context is the dish being cooked.

3:18 Now here, this is compared against the retrieved memory,

3:22 the jar from the pantry.

3:24 If the jar's contents don’t agree with the dish,

3:28 the gate drops to zero, throwing the ingredient away.

3:32 Bye bye rotting fish!

3:34 This mechanism lives right here, inside this jolly little dot product.

3:39 Now let’s see how it actually performs against the current systems.

3:44 I’ll tell you exactly what is going to happen now.

3:47 What happens in nearly all research papers with something new.

3:51 It does something, it is compared to previous methods,

3:55 and it’s better at some things, worse at others.

3:59 And then you sit down and you do your analysis.

4:04 Okay, let’s see…wait what?

4:06 What just happened here?

4:08 The new engram technique makes the neural network better…everywhere.

4:13 Absolutely everything is measurably better.

4:16 This is an absolute miracle work.

4:19 The engram model is actually better on every

4:23 single benchmark compared to the previous techniques.

4:26 It is better everywhere!

4:28 Now this is an amazing life lesson too.

4:32 How?

4:32 Well, essentially what DeepSeek does is automates the easy part,

4:37 and focuses on the more difficult tasks.

4:40 No wonder it works so well!

4:42 What a time to be alive!

4:44 We can learn so much from these research papers,

4:48 and not just about AI, but about life itself.

4:52 Okay, now I’ll tell you how this works,

4:54 and it turns out, there are more surprises ahead.

4:58 Dear Fellow Scholars, this is Two Minute Papers with Dr.

5:01 Károly Zsolnai-Fehér.

5:02 Okay, so how does it do this magic?

5:04 Well, it uses what they call n-gram embeddings combined with multi-head hashing.

5:09 Okay, what the heck does that mean?

5:13 Well, in the kitchen, the chef looks at the order ticket, sees a 3-word phrase,

5:19 and instantly knows exactly which shelf in the pantry has the premade sauce,

5:24 and grabs it quickly.

5:26 And I think this also shows us that there are

5:29 simple and basic ideas in AI that we haven’t found yet.

5:33 I mean, this thing is basically a look up table.

5:37 It is as simple as it gets,

5:39 and it makes everything more efficient and better across the board.

5:43 Just think about it:

5:45 we removed 20 or 25% of the smart experts in this little virtual brain,

5:51 put a spreadsheet there, and it got better!

5:55 I mean what?

5:57 Crazy.

5:57 And I love how we have a little better understanding of the AI system itself.

6:03 Usually, no one knows what is going on inside, but here.

6:07 Look.

6:07 When they switched off the engram memory during testing,

6:10 the AI’s ability to answer trivia went down 70%.

6:16 But its reading comprehension remained at 93%.

6:21 Why?

6:21 Well I think this shows that AI split its brain,

6:25 and it’s using the new part just to store facts.

6:29 Just think about it.

6:31 When they locked the pantry door during testing his ability

6:34 to understand a recipe stayed at a massive 93%!

6:38 What does that mean?

6:40 It shows the chef split the work perfectly.

6:44 He used the pantry strictly as a storage shelf for memorized ingredients,

6:48 but he can still cook an amazing meal.

6:52 I think this is going to lead to even cheaper and even smarter AI systems,

6:57 and this will be an important part of why we

7:00 will all get more systems that we can actually own,

7:04 no subscriptions, these run in our pockets super fast, mostly for free.

7:09 Okay, now not even this technique is perfect.

7:12 One limitation is that if you put the engram module too deep in the network,

7:17 it gets less accurate because the model has

7:20 already wasted time processing what is being asked.

7:23 Of course, there is no need to look up what you already computed.

7:27 I think this is common sense at this point.

7:30 Our chef has to check the pantry at the start of the shift.

7:34 If he checks it after the food was served, the pantry is completely useless.

7:40 A really advanced research paper explained in simple words.

7:44 We are Fellow Scholars, and that’s what we do here.

7:48 And we have a growing club.

7:50 I’ll continue in a moment, but you know who is also watching us?

7:54 The one and only Larry Wheels.

7:57 Yes.

7:57 He is one of our OG Fellow Scholars,

8:01 doing some Scholarly work between two hard sets of bicep curls in the gym.

8:06 You think I am kidding?

8:08 I am not.

8:09 Link is in the description.

8:11 Reading his comment made me instantly more muscular.

8:14 So much value.

8:15 Huge respect to Mr.

8:17 Wheels!

8:17 Honored to have you here.

8:20 And here comes the best part.

8:22 I think this will be a part of every major AI system,

8:26 and it is knowledge out there for free for all of us,

8:30 and now you know exactly how it works!

8:33 No nonsense where everything is hidden in a proprietary

8:36 system that costs 300 dollars per month to run.

8:40 Nope.

8:41 All free for all of us.

8:43 Glorious.

8:44 An epic paper.

8:45 Now, as our chef does, I took a bit longer to cook this video.

8:49 But I promise that I did not

8:51 put together my computer from scratch before starting.

8:54 So I took some more time to make sure you get a better video.

8:59 If you feel this is the right way of doing that, subscribe,

9:03 hit the bell and leave a really kind comment.

9:06 And you can also check out Lambda with our link in the description

9:10 because it is an excellent way of running DeepSeek privately, I do it too.