Mind Reading

0:05 Mind reading?

0:06 Of course not.

0:08 I love reading.

0:11 Look, mind reading might sound

0:14 like pseudoscientific— pardon my language— bullshoot.

0:18 But its scientific counterpart, thought identification,

0:21 is very much a real thing.

0:23 It's based in neuroimaging and machine learning,

0:26 and what's really cool is that experiments in mind

0:29 reading aren't just about spying on what someone is thinking.

0:33 They're about figuring out what thoughts are even made of.

0:37 I mean, when I think of something,

0:39 what does that mental picture actually look like?

0:43 What resolution is it in?

0:44 How high fidelity is a memory, and how do they change over time?

0:49 Well, in this episode, I'm going to look at how reading

0:52 people's minds can help us answer these questions.

0:54 My journey begins right here at the University of Oregon.

0:58 I'm meeting with Dr.

0:59 Brice Kuhl from the Kuhl lab.

1:01 He's a neuroscientist who uses neuroimaging and machine learning

1:04 to figure out what people are thinking without them telling him.

1:31 So tell me what you're doing here.

1:33 Well, I'm in the cognitive neuroscience program here, and I study human memory.

1:38 My lab primarily uses neuroimaging methods,

1:41 so we do a lot of work using functional magnetic resonance imaging, or fMRI.

1:45 And how do you use fMRI to investigate memories?

1:49 We're looking at the pattern of neural activity.

1:51 When you form a memory, there's a certain pattern.

1:54 And we can record that pattern and then test whether

1:58 that pattern is reinstated or reactivated at a later point,

2:01 like when you're remembering it.

2:02 Does that mean we can look at the patterns of brain

2:05 activity and deduce what it is that is being remembered,

2:09 or recalled, or even just thought?

2:11 Yes, and so we call that decoding.

2:13 So it basically takes your input pattern as some

2:17 pattern of activity that we record while you're remembering something.

2:21 And we make a prediction about what you're remembering.

2:23 You can see how this sounds like mind reading.

2:27 [laughs] Yes.

2:27 It sounds like that.

2:28 So, Brice, what are you going to do to me today?

2:32 So, what we're going to be doing today is uncharted territory for us.

2:36 So we're going to be trying out a kind of new variant of the experiment on you.

2:40 So I can't guarantee any particular results.

2:42 But it represents where the field is and where we're trying to go.

2:46 Today, you're going to participate

2:47 in an experiment where you'll be studying faces.

2:50 So we're going to have you study 12 pictures of celebrities.

2:53 People I already am familiar with.

2:54 -People that you know, yeah.

2:56 -Okay.

2:56 And you're going to try to remember those pictures.

2:59 Then we're going to have you go into the MRI scanner.

3:01 Try to bring that picture to mind as vividly as possible.

3:04 And we're going to be recording your brain

3:06 activity as you try to imagine these pictures.

3:08 We're going to try to build the face.

3:10 Essentially draw a picture of what you're remembering.

3:12 -A picture?

3:13 -A picture.

3:14 An actual picture that we can print out and I could, like, hang on my wall.

3:17 [laughs] If you wanted.

3:19 [Michael] The first step is for me to memorize the 12

3:23 specific celebrity photographs Brice will later try to detect me thinking about.

3:28 I sat down to do this graduate student, Max.

3:33 The success of his predictions depend, in part,

3:35 on my ability to recall these faces

3:37 as vividly as possible while inside the fMRI.

3:43 All right, so...

3:44 [sighs] I think I have a pretty good memory of all of those.

3:50 -Great.

3:51 -I feel the stakes are high.

3:53 With the celebrity faces hopefully memorized, it's time for the next step:

3:58 going through the metal detector and into the fMRI,

4:01 where Brice will record and monitor my brain activity,

4:04 and then later feed it into his algorithm to rebuild the faces.

4:08 This will be the first time he's

4:10 attempted to reconstruct faces from long-term memory, which is very difficult,

4:13 because we're relying on how clearly I can

4:15 remember the celebrity photos I saw an hour ago.

4:19 I love its eyes.

4:20 Look at that.

4:21 [woman] Wouldn't the kid be like, "It's going to eat me"?

4:31 An fMRI monitors the activity within the brain by dividing

4:34 it up into thousands of small cubes called voxels, or volumetric pixels.

4:39 Each of these voxels contains hundreds of thousands of neurons.

4:43 Using fMRI, we are able to detect blood flow within these voxels,

4:47 which means that that part of the brain is active.

4:50 If I'm shown several pictures of people with mustaches,

4:53 my brain will react to the features for each face.

4:56 But there will be a common area of my brain that is engaged throughout.

5:00 That may be the area of my brain that reacts to mustaches.

5:04 So later, when I imagine a face, if Brice notices that area is engaged,

5:09 he can predict that I am thinking about a mustache.

5:13 So right now Michael's in the scanner,

5:15 and he's seeing words appear on the screen one at a time,

5:18 and he's trying to visualize the face,

5:20 remember the face in as much detail as possible.

5:23 What you can see here are the images that we're acquiring.

5:25 We get one of these brain volumes every two seconds.

5:28 So these are refreshing in real time as we collect the images.

5:32 [Michael] With part one of the fMRI session over, it's time for part two,

5:37 where Brice and his team will learn the language of my brain activity,

5:41 so they can later decode by brain scans.

5:44 Hi, Michael.

5:45 You doing okay still?

5:46 [Michael] Yup.

5:48 They'll show me hundreds of unique faces,

5:50 and record how my brain reacts to certain facial characteristics.

5:54 They will then use this information to reconstruct the celebrity

5:59 faces I thought about during the first phase of the scan.

6:03 Really, the more faces that we can show Michael, the better.

6:05 So we're going to basically keep him in there as long as he's comfortable.

6:09 [Michael] Two hours was the maximum time we could get in the fMRI.

6:13 But I was able to look at over 400 faces,

6:17 which should be enough to get some pretty interesting results.

6:20 Hey, Michael, you did it.

6:21 That was great.

6:22 We're going to come get you out.

6:23 [Michael] All right.

6:33 Yeah, so these just show some of the pictures

6:34 that we were taking while you were in there.

6:36 Some images of your brain.

6:38 Now we are going to crunch some numbers.

6:39 Max is going to analyze your data.

6:42 We'll meet up again tomorrow, where we'll look at the results,

6:45 where we try to actually reconstruct the face

6:47 images from the brain data that we just collected.

6:49 All right.

6:50 Well, see you tomorrow.

6:51 All right.

6:51 Thanks a lot.

6:52 Max, thank you as well.

6:53 I can't wait.

6:54 You better pull an all-nighter.

6:55 I want this data to be perfect.

7:04 All right, so I am back at Dr.

7:06 Kuhl's lab.

7:06 Overnight, his team crunched the data,

7:08 and I can't wait to see what they think they saw me thinking.

7:15 How are my results?

7:17 I think they look good.

7:18 We're going to take a look in just a moment here.

7:20 All right, I can't wait.

7:22 -So can I just take a seat?

7:23 -Yeah, have a seat.

7:24 All right, so...

7:26 first of all...

7:28 what am I seeing?

7:29 Oh, okay, well, these are the pictures I actually memorized.

7:32 -That's right.

7:32 -And this is what you've reconstructed from my imagination.

7:37 -That's right.

7:38 -Oh, wow.

7:39 Okay.

7:40 [Brice] Okay, so this is one of the reconstructions that was generated.

7:44 [Michael] Interesting.

7:46 [Max] So that's John Cho.

7:47 [Michael] Not bad.

7:49 Not bad.

7:50 -Can we see the side by side?

7:52 -Yeah.

7:53 [Michael] I see, you know,

7:55 similarities in the kind of facial expressions in general.

8:00 You know, you could almost see the hairline matching here.

8:02 The shape of the face I also thought was— It kind of had a square shape to it.

8:06 -Yes.

8:06 Yes.

8:07 -So those are the things that came out to me.

8:09 And so when I was visualizing this image of John Cho,

8:13 the squareness of the face was the first, most salient thing.

8:16 I just kept thinking, he was the square guy.

8:19 Excellent, all right.

8:23 [Brice] So that's Megan Fox.

8:26 [Michael] Mm-hmm.

8:28 You're going to show us the— side by side.

8:30 [Michael] The side by side.

8:31 Right.

8:31 [Brice] You can see the picture you actually saw,

8:33 and that's the reconstruction we generated.

8:36 I'll you this.

8:37 Megan Fox, I was not able to have a really clear picture in my mind.

8:42 For some reason, this image of her was really

8:44 hard for me to bring back into my mind.

8:47 The sternness in the face was something that I did pick up on.

8:50 So I did sense that there was— It looked feminine.

8:53 And you picked up on the sternness.

8:55 And so together, that produces a match.

8:58 [Michael] Keep in mind that Brice and his team have read these from my memory.

9:02 But when I remember a face,

9:04 do I picture every detail simultaneously with photographic accuracy?

9:09 Or do I just attend to a few at a time?

9:10 By reading my mind, they may be seeing how bad my memory is, and how it works.

9:15 -Me!

9:16 Me!

9:16 -[Brice laughs] Okay,

9:19 so that is your reconstruction of me thinking about this image of myself.

9:24 [Brice] That's right.

9:26 Where'd the beard go?

9:28 [Brice] I don't know.

9:29 I was hoping you could tell me.

9:31 [Michael] For instance, this is a picture of me remembering my own face.

9:36 It really doesn't look like me, but the question is:

9:38 how good am I at picturing myself?

9:41 I don't think of my own face that often,

9:44 so the strangeness in the result may be as much about flaws

9:46 in my own memory and mental picture of myself as flaws in the technology.

9:51 So that's Jennifer Lawrence, I believe.

9:53 [Michael] That's Jennifer Lawrence?

9:55 It looks like it's Jennifer Lawrence's much older uncle.

9:59 [all chuckle] Nothing here was too mind-blowingly close.

10:05 But this is something that you're just

10:08 starting out trying these sort of long-term memories.

10:11 What Brice and his team read in my mind might have been more

10:16 accurate if they'd shown me thousands rather

10:18 than hundreds of images in the fMRI,

10:20 because then the algorithm would have learned

10:22 the language of my brain more thoroughly.

10:24 But regardless, the quality of my memories would have still been an issue.

10:29 I mean, look what happens when memory is cut out of the equation entirely.

10:33 Brice also read my brain activity when I was looking at faces in the fMRI.

10:37 not just imagining them.

10:39 And those results were much closer than those reconstructed from my memory.

10:44 Okay, so, what am I looking at right here?

10:47 [Brice] Okay, so what you're seeing here in the top row,

10:49 these are images that you saw while you were in the scanner.

10:53 Below that, in this bottom row, these are the reconstructions that we draw

10:57 from the patterns of brain activity we collected.

11:00 -This is from the source image.

11:03 -Right.

11:03 [Michael] These are from my brain.

11:05 -[Brice] Right.

11:06 -[Michael] They're pretty close.

11:07 Yeah, overall they were pretty close.

11:09 So not perfect.

11:11 These are— you can see there's some variability in these.

11:13 But this is consistent with what we've found before,

11:16 that the reconstructions that we generated, when you're viewing the faces,

11:20 there is some correspondence between the actual face.

11:22 So this is kind of a sanity check,

11:23 that we can actually reconstruct the images -when you're viewing them.

11:27 -Right, right.

11:28 They're pretty good.

11:31 Well, Brice, Max, thank you so much for letting me be a part of this.

11:35 I hope my data's useful.

11:36 Thank you.

11:37 It's been a lot of fun.

11:38 It's always useful for us to think about these things.

11:46 Dr.

11:47 Brice Kuhl's memory research is showing that it's

11:50 possible for a computer to read someone's mind.

11:53 To figure out what they're thinking.

11:56 But a lot of progress still needs to be made.

11:58 I mean, if you want to know what I'm thinking right now,

12:01 for example, it's still easier to just ask me to tell you.

12:05 But what if I can't tell you?

12:07 Dr.

12:08 Yukiyasu Kamitani is a researcher,

12:10 professor and pioneer exploring the frontier behind the wall of sleep.

12:17 I've come here to Kyoto University to meet with him

12:20 and to see what it's like to read not what someone is thinking,

12:24 but what someone is dreaming.

12:29 Kamitani sensei, I'm Michael.

12:31 -Hi, I'm Yuki.

12:32 -Yuki, nice to meet you.

12:33 [Michael] For the last ten years, Dr.

12:36 Kamitani has been at the forefront of machine mind reading.

12:40 The subject is, you know, ready to go in.

12:43 Similar to Brice Kuhl,

12:45 his early experiments explored reconstructing images shown to subjects

12:49 in an fMRI based on their brain activity.

12:52 In Kamitani's case, the images were black-and-white shapes,

12:55 and the reconstructions were strikingly accurate.

12:58 Recently, Kamitani has focused on using

13:01 deep neural networks and machine learning

13:04 to decipher subjects' brain activity while

13:06 they view much more complex photographs.

13:08 What you're seeing is the result of a deep neural network

13:12 processing the brain activity of a subject looking at the photograph.

13:17 This could have myriad applications in the future,

13:20 for example, in criminal investigations and interpersonal communication.

13:26 [Kamitani] This is far from perfect.

13:28 But I think you still see some, you know, eyes and, you know...

13:33 [Michael] Well, yeah.

13:34 And colors too.

13:36 [Kamitani] Yeah, to some extent, yeah.

13:39 His most current work, however, is about the subconscious.

13:42 He's attempting something extremely ambitious: recording our dreams.

13:46 Would you call yourself a sleep researcher, or a vision researcher?

13:50 Maybe a brain decoder.

13:53 A brain decoder.

13:55 That's a pretty cool job description.

13:57 Can you show me anything from what you're doing with dreams?

14:01 [Kamitani] Mm-hmm, yeah.

14:08 Dr.

14:08 Kamitani's work on dream decoding begins with a similar process to Dr.

14:12 Kuhl's: showing the test subject thousands of images while they are in an fMRI

14:17 in order to learn what the brain looks

14:18 like when it is thinking of certain things.

14:21 Once the machine-learning algorithm is pretty good

14:23 at identifying what images the subject is thinking about,

14:26 the subject is placed in an fMRI with an EEG cap on their head,

14:31 and invited to fall asleep.

14:33 When the EEG waves indicate that the person is dreaming,

14:36 the algorithm predicts which kinds of things

14:39 the subject is most likely dreaming about.

14:41 Right now, the algorithm looks for 20 categories.

14:45 Things like buildings, transportation, and characters in a language.

14:50 Researchers then awaken the subject, ask them what they were dreaming about,

14:55 and see if the algorithm's prediction and the person's recollection match.

14:59 Here is actual data from one of Kamitani's experiments.

15:03 Below is a word cloud of categories.

15:05 The name of each category get bigger or smaller in real time

15:09 based on the probability that they are present in the subject's current dream.

15:13 Now, as you can see, activity is currently strongest

15:15 for the category "character," meaning written language.

15:18 At this point the subject was awoken, and this is what they reported.

15:29 That's pretty spooky.

15:32 -[laughs] -Right?

15:32 I mean, you— you spied on their dream.

15:36 Yeah, in a way.

15:38 But...

15:39 the accuracy's not that great, so...

15:42 Well, the accuracy's not that great but, you know,

15:44 my normal accuracy for guessing people's dreams is zero.

15:46 Right.

15:48 While continuing his research into predicting the content of dreams, Dr.

15:52 Kamitani is embarking on his newest project:

15:54 actually reconstructing images from our dreams.

15:58 So you've brought some of the reconstructions that your lab has created...

16:02 Mm-hmm.

16:04 ...of dreams.

16:17 Right, they all look like dreams about blobs.

16:20 [Kamitani] Yeah.

16:21 I mean, I want to just take a step back and...

16:24 appreciate that what we're looking at on this screen are,

16:28 in a way, some of the first photographs of a dream.

16:31 Mm-hmm.

16:33 We are looking at the earliest phase of revolutionary research.

16:38 One day, we may able to have images, or even record movies, of our own dreams.

16:42 And Dr.

16:43 Kamitani is the only person in the world doing this so far.

16:47 He's a lone explorer journeying into our subconscious.

16:50 So this work hasn't even been published yet.

16:53 No.

16:55 -Thank you for showing it to me.

16:57 -[laughs] The insights that researchers like Dr.

17:04 Kuhl and Dr.

17:05 Kamitani might be capable of achieving in the future

17:09 because of mind reading are difficult to fully fathom.

17:13 But let's slow down for a second,

17:15 because we're talking about a technology that can

17:18 know us better than we know ourselves.

17:21 Should we really be doing this?

17:23 Well, to address that question, I'm going to meet with an expert in ethics,

17:27 neuroscience and artificial intelligence: Julia Bossmann.

17:32 She's the director of strategy at Fathom Computing,

17:34 a council member of the World Economic Forum,

17:37 an alum of Ray Kurzweil's Singularity University,

17:40 and a former president of the Foresight Institute,

17:43 a think tank specializing in future technologies and their impacts.

17:50 Julia, thanks for taking some time to chat.

17:52 -Yeah, of course.

17:53 -You are the perfect person for me to bring these questions to.

17:55 -Mm-hmm.

17:56 -And they're deep questions.

17:57 But I think they're extremely important,

17:58 and they're becoming more and more pressing.

18:00 I think we're living in such an interesting time right now,

18:04 because we're in this time where brains

18:05 and machines are actually moving closer together.

18:07 So when it comes to being able to look at brain activity,

18:12 where are the ethical lines here?

18:15 How private should my internal thoughts be?

18:17 Like with any powerful technology, it depends on the hands that wield it.

18:22 All these new technologies are things

18:25 that can make whoever uses them more powerful.

18:28 So we want to not blame the technology,

18:30 but we want to— how is it being used, and who is using it?

18:35 So how do we make sure that this technology is in the right hands?

18:39 So I think it's very important to involve people who act

18:42 on policy and law to understand what is coming in the future.

18:48 I am hopeful about the collaborative aspect of it.

18:51 Let's talk about the good things now.

18:53 I mean, what are the applications here?

18:56 Yeah, so if we think about the late Stephen Hawking, for example,

18:59 if he had a way of richer interfacing with the world or with computers,

19:05 we can only imagine what he could have shared with us.

19:08 Those with locked-in syndrome, right?

19:10 They are there.

19:11 They know that they are there.

19:13 But we just need something to look into their brain

19:15 to see what it is that they are trying to say, -or what they're feeling.

19:19 -Right, exactly.

19:20 So, what do you say to people that have that kind of fear of technology,

19:26 of us surrendering our true natural selves to technology?

19:31 There is something enticing about getting to the next level of what

19:37 some people might call a human evolution or civilization development, and so on.

19:43 In a way, we are already not living natural lives, right?

19:46 Because then most of us would die before the age of, I don't know, 30 or 40.

19:51 We would have all kinds of diseases.

19:53 We would not wear this clothing.

19:55 We wouldn't have eyeglasses or contact lenses.

19:58 We wouldn't have antibiotics.

19:59 [Julia] We are already kind of very futuristic

20:03 cyborgs if we compare ourselves to the human

20:06 that was living 10,000 years ago and was

20:09 genetically almost identical with who we are now.

20:11 [Michael] Yeah, we really are.

20:17 In order to understand cognition, right now we basically have to either just

20:22 ask people to talk about what they're thinking, or observe their behavior.

20:26 But reading thoughts directly would be a lot better.

20:30 That is how Dr.

20:31 Kuhl is studying memory, and it's how Dr.

20:35 Kamitani is studying sleep and dreams.

20:38 But even though the technology has a long way to go,

20:40 it's easy to see how ethical questions could become an issue.

20:44 Well, here's the thing: there is no such thing as a totally wild human.

20:51 We are co-evolving with technology.

20:55 Humans and technology today are inseparable.

21:00 Now, it's true that we need to be careful about every new thing we do,

21:03 but we cannot change the fact that they will happen.

21:08 It's a story we've lived through again and again.

21:11 You know, we could have sat around forever debating whether or not

21:15 a speed limit should exist and who should have the authority to enforce it.

21:19 But we didn't.

21:21 Instead, we went ahead and invented cars,

21:24 and responsibly figured out the details as we went along.

21:29 Ethical questions about new technologies do

21:31 the most good when they facilitate the technology,

21:35 not when they needlessly hinder progress.

21:40 So follow your dreams.

21:41 And, as soon as you can, show them to me.

21:44 And, as always, thanks for watching.