Haptic Rendering - Computerphile

0:00 So, we are going to talk about computer haptics today.

0:03 Uh and I will give you an introduction of what haptics means.

0:07 Okay, so I'm thinking of things like force

0:09 feedback on a controller on a on a computer,

0:12 you know, video game or something.

0:14 Is is this the sort of thing we're talking about?

0:16 Yes.

0:16 Uh it is one aspect of haptics.

0:18 So, uh when you think about mobile phones or game controllers,

0:23 they're typically vibrations.

0:25 Uh and these are um about your tactile sensation

0:29 or cutaneous sensations that are about fingertip receptors or stretches.

0:35 on an Android phone, it makes a little Yes, it makes little buzzers.

0:38 Uh and when when it rings, it buzzes as well.

0:41 Uh this is one aspect of haptic feedback.

0:44 And the other aspect is kinesthetic feedback or proprioceptive feedback,

0:48 which uh talks to our senses of forces, positions.

0:53 So, it is how haptic feedback is transferred through our joints and muscles.

0:57 So, this this must be the sort of hard stuff to get right.

1:00 Like when you shake someone's hand,

1:01 you sort of feel how hard they're shaking and you shake back.

1:04 But a robot, how does that do that?

1:06 Is this the sort of thing we're going for?

1:07 It is similar, but uh when you think about touching a human,

1:11 it is actually very much complex.

1:12 So, the the thing about haptics is is really distributed.

1:16 So, we get information from multiple channels because there

1:20 are a lot of different sensors on our uh body,

1:23 especially in our hands, and we combine them.

1:26 So, it's a combination of how forces, for example,

1:29 how much you're squeezing, which is about pressure,

1:32 and then how much you're moving, for example,

1:34 which is about the forces that you're applying to.

1:37 And robots today, they are only capable of realizing

1:41 some of it because we don't have sensory technologies

1:45 that could uh sensor and actuation technologies that could

1:49 that could make it very comparable to human haptics.

1:54 When I talk about computer haptics, this is a little bit different than how you

1:57 would uh realize oops sense of touch with a robot.

2:00 It is about how you can create a virtual world like a simulation a computer

2:05 graphics game for example and you implement a device to feedback forces to you.

2:12 So when I talk about haptics today uh what

2:14 I refer to will be kinesthetic feedback force feedback uh

2:18 and what we are going to talk about is

2:21 how these forces are computed when you see a scene.

2:26 And this scene would be a simulation uh what

2:29 it can reflect to the real world as well.

2:32 Uh And within this simulation we um assume

2:35 that there are objects that we want to touch.

2:38 So this is if you can see a haptic

2:40 interface and it's moves in uh in the 3D space.

2:47 So this is a uh grounded haptic interface which has actuators.

2:54 And what it reads is the position and the orientation although it doesn't

2:58 uh really drive torques at your hand it only gives you uh force feedback.

3:03 This is an impedance type device which means

3:06 that it reads positions and gives you forces.

3:10 There are other kinds of devices as well uh but this is the probably

3:13 more uh most common of uh haptic force feedback interfaces that you can have.

3:19 We also have different interfaces which you can for example wear like uh

3:23 exoskeletons or force gloves uh and they

3:27 do operate on similar mechanisms as well.

3:29 So we are going to talk about uh how we

3:33 actually use this almost as a mouse and then go

3:37 and touch an object in in the in the on the yeah

3:41 on the virtual space on a 2D screen.

3:44 Uh so I can um actually talk about

3:48 this as a sensing actuation problem where you sense the data,

3:52 which is positions in this case, and you have then a simulation loop which

3:58 consists of a visual loop and the haptic loop.

4:01 And then you have the actuation.

4:04 Uh the visual loop operates typically at a very

4:07 low frame rates like 30 Hz, 60 Hz.

4:11 Uh however, the haptic loop, in order to be stable,

4:14 it needs to operate at a much higher rate, which is traditionally 1,000 Hz.

4:19 Oh, 1,000, okay.

4:20 So, every millisecond you have to make a computation and send it back

4:24 to the device so that the device would give you something meaningful and stable.

4:29 Let's say we have a haptic device here,

4:32 which is similar to this, that can read data from encoders.

4:37 So, we have encoder data, which means I can read the state of the robot arm,

4:43 uh which is typically the joint angles.

4:46 Mhm.

4:46 And I know the model of the haptic device,

4:49 so I can compute the position of the tool tip or the end effector,

4:55 uh or the interface point as we as we call it.

4:57 So, the encoder data is fed into something that we call forward kinematics.

5:04 And this is very traditional in robotics,

5:08 so you can get the state, the joint angles of a robotic device,

5:13 and because you know the model, you can compute where the tool tip is.

5:17 And this is the easiest easy problem.

5:20 And then the forward kinematics computes a position.

5:24 That position is used to interface with a simulation,

5:28 and within the simulation, I have objects.

5:31 These objects can move, these objects can be static,

5:35 uh what I need to do in order to enable haptic

5:39 rendering is very similar to what we do in computer graphics,

5:43 to look into collisions.

5:46 So, the position of the tool tip is taken

5:49 and that is checked for collisions with objects within the scene.

5:55 So, I will call this collision detection.

6:00 People will be thinking of again video games with this, won't they?

6:03 You know, it's It is the same principle.

6:04 Same principle.

6:05 Uh, so haptics and computer graphics, they are extremely similar.

6:10 So, many of the methodologies, especially in haptic rendering,

6:14 uh, is adopted, from computer graphics.

6:17 Some differences is in simulation,

6:19 you want to show visually something that is compelling.

6:23 The aim of haptic rendering, in contrast to the visual rendering,

6:27 is to make someone feel something real.

6:30 Physical, yeah.

6:31 Physical.

6:32 And the one other difference is, uh, the frame rate.

6:36 So, this needs to be really very quick,

6:39 whereas this tolerates failure a little bit more.

6:42 Tactile sensations or haptic sensations are uh, much faster.

6:46 Uh, we wouldn't really see if some frames are

6:48 dropped or if they are injected with some information.

6:52 Uh, we would feel better discontinuities and they are what we,

6:56 uh, put meaning into when understanding the world, explore the world.

7:00 But also, from a robotics point of view,

7:03 the discontinuities make the robot unstable.

7:06 So, the behavior is not realistic.

7:09 So, collision detection takes some information from simulation,

7:13 which is where the objects are,

7:14 where example, the tetrahedral or triangular models that are comprising the, uh,

7:20 simulation elements.

7:21 We have the position of the, uh, of the device,

7:25 the tool tip, and we compute a contact point.

7:30 Once we have the contact point, we need to compute a collision response.

7:35 So, in graphics, a collision response could be you

7:38 push one object and it moves and it touches something, so there is a and effect.

7:43 So, you put you issue a force and there is

7:45 a physics uh model behind the simulation which makes things happen.

7:50 And this is the same principle.

7:52 You touch something and you generate a force.

7:54 The object can move in response.

7:56 The collision response would compute a force, a force vector,

8:02 to be felt at the tooltip or in my hand

8:05 if I was pushing it if the system was completely transparent.

8:09 Uh and then that force is sent to the simulation and to the haptic device.

8:15 The force that you send to objects in the simulation

8:17 and the force that you should send to the device may differ.

8:22 So, you could have some force modulation here,

8:26 which could be scaling or putting it into the ranges

8:29 so that you don't you don't harm the device or Right.

8:32 It could be So, Isabella, I can like graphics, you know,

8:35 you scale to what the monitor's capable of displaying

8:37 rather than I don't know, burning it out.

8:40 Yes, exactly.

8:41 It is very similar.

8:42 So, you move mouse your mouse, for example,

8:44 5 pixels, but then your agent moves 5 m.

8:47 Right.

8:47 So, there is a scaling over there.

8:49 So, I will draw two arrows here.

8:53 One is the force sent to the Mhm.

8:56 simulator if anything is moving and the other one would

8:58 be uh the force that you should feel at the tooltip.

9:03 Mhm.

9:03 When it comes to the device,

9:04 what I can control is the torques that I send to the motors.

9:10 So, there is another block here we call the inverse kinematics.

9:17 Uh and this is the harder problem

9:19 when you compare it with the forward kinematics

9:21 because it solves the inverse problem of what should be the state of this robot.

9:29 It's not a hard problem as it is uh NP-hard or anything,

9:33 but uh it is less straightforward than forward kinematics and we

9:37 do have very established methods for the inverse kinematics as well.

9:40 Uh and this computes torques.

9:43 As I said, this kind of operates at 30 hertz,

9:47 let's say, and this has to operate at at least 1,000 hertz.

9:52 Oh, yeah, sorry.

9:53 So, it can be more than that, then.

9:54 It can be more than that.

9:55 Uh but if you drop down, then you would start feeling it's not really stable.

10:01 And in force feedback robotics as well, that becomes a bottleneck,

10:06 especially if you're doing if you're dealing with, for example,

10:08 plate systems, if you're sending a robot arm to a space

10:12 station and you're doing a delayed teleoperation and so on.

10:15 So, this is our uh the bottleneck.

10:18 Mhm.

10:19 Okay.

10:20 Okay, so this is exactly what we are doing

10:23 and uh I will now talk a little bit about how

10:25 we do the collision detection and how we do the collision

10:28 response so we could feel some objects in a scene.

10:32 So, as you may know,

10:34 if you're interested in computer graphics, uh objects are discretized.

10:40 So, you may have a polynomial polygonal object which consists of triangles.

10:46 Uh and the most naive way could be I have this point,

10:51 this mouse cursor, and I have thousands of triangles.

10:56 In order to check for a collision, I could just check with every triangle.

11:02 But then, this is very slow.

11:04 Yeah.

11:04 Especially if you have to do a thousand operations in a second,

11:07 this gets very slow.

11:09 So, what we do in haptics is we

11:11 use bounding boxes and we use hierarchical bounding boxes,

11:15 which is which is similar to this octree algorithms,

11:18 where you have a bounding box for an object,

11:22 and then you check whether there is a collision with that bigger bounding box.

11:26 If not, you should feel nothing with a haptic device.

11:29 It should be really transparent that as if you're holding nothing in your hand.

11:33 And if you have a collision, you start dividing it into smaller sections,

11:38 so you don't actually search the whole space,

11:40 but you only look into a small set of triangles.

11:45 Uh and why this works is also because

11:49 of the observation that once you detect where you are,

11:55 then the search space really becomes very,

11:57 very small because humans actually can't move that that fast.

12:04 Okay, so that makes the search problem very, very quick.

12:11 And once we know we're reaching an object,

12:15 we have a surface that we are testing against,

12:19 the collision response, the computation of the collision response,

12:22 the forces is the is the problem.

12:24 So, I'll go over the simplest example here for a point to line,

12:31 point to surface collision detection.

12:35 You could know when you're not touching the object,

12:39 and this is your haptic interface point, which I will write as hip.

12:43 The force in this situation should be zero because you're outside,

12:47 and you shouldn't really feel anything.

12:49 The device should feel really empty as if it's not existing,

12:54 which we call transparency in robotics.

12:58 And when you touch the object,

13:00 in haptics we allow the hip to pierce into the object.

13:05 In graphics, you would see something on the object,

13:08 so visually you should see the object as here, but in order to compute forces,

13:14 you need some kind of formulation that allows you to compute

13:19 uh to see how much how much force is Yeah, okay.

13:22 Yes, you should apply to pull you away from the object.

13:25 So, in practice, what you would do is you

13:29 would allow the hip to pierce into the object.

13:37 And then, you would place a little virtual spring here.

13:41 And then, the force you compute would be simply the spring force.

13:46 You would set a stiffness parameter K,

13:48 and that would define your material properties.

13:51 So, I can show you a very simple demo.

13:55 So, what you see here, uh is now I I turned on this uh demo,

14:01 where you have three different surfaces.

14:03 So, one of the surfaces is characterized as being soft,

14:06 the other one is medium, and the other one is hard.

14:09 Uh and if I So, this is my This is a representation of my cursor here,

14:13 which is very similar to the uh end effector.

14:16 And when it touches very slowly,

14:18 you could see a maybe something like a springy movement.

14:22 Yeah.

14:23 And if you look at the device, it's kind of really springy.

14:28 You could see how the force is modeled.

14:30 So, it does act like a spring.

14:33 If it's If I go to a harder object, then you could feel it more.

14:39 And the hardest object,

14:40 then you would really feel that your your accelerator is sliding So,

14:46 we're inside there all the motors are basically saying,

14:48 "Don't let that probe go any lower down, right?" Yes,

14:52 they are saying that because we are I'm very slow.

14:55 So, I'll also show you something different here,

14:58 uh because this is all about movement, right?

15:01 I had a haptic interface point, and I'm moving it in space.

15:05 So, I'm really very slowly approaching this.

15:09 So, I'm not really piercing the object a lot, right?

15:13 So, you don't actually see that this is really getting into the You know,

15:18 the the you could see the the sphere pretty well.

15:20 Maybe you see some kind of piercing motion

15:24 that you see the bottom is getting in.

15:26 But what happens when I become a little bit maybe fastish?

15:31 I'm not really hold holding it very much.

15:34 So it's bouncing.

15:35 So I'm I'm actually making it go into something in stable.

15:41 Uh and you can also see it's sometimes popping through.

15:46 Right.

15:47 So it's getting under the object where it shouldn't be allowed to.

15:52 And why is this happening?

15:54 Um because it's stress positions.

15:56 There um the rendering algorithm doesn't know where I am.

16:01 So what happened over there is I was moving the hip.

16:06 And then I was moving it very fast.

16:07 So I was actually going really like here.

16:12 And then the force that you compute gets quite

16:18 large and it makes you pop out of the object, jump and bounce back.

16:25 And then this can actually escalate as if,

16:28 you know, it can escalate really badly.

16:30 In this In this case it didn't, but it can also get very very unstable.

16:34 So you have to terminate the simulation.

16:38 Another issue that happens is for example when we have a very thin object,

16:44 which was also the case in this example,

16:47 uh where you would compute forces that would push

16:52 you away from the body of the object when

16:58 you approach from this direction and away from the body

17:01 of the object if you approach from this direction.

17:04 But if I'm coming and if I'm really fast I could actually pop through.

17:10 Another thing that could be a problem is if I have

17:14 a complex object then I would represent it as a few subsections.

17:21 And in sub in the subsection, for example, if the hip is here,

17:27 I would find the closest surface and compute

17:31 a force that pushes me in that direction.

17:34 So, this is the forces that you should feel for this surface.

17:38 And if I'm here, I should be actually feeling forces against this surface.

17:45 So, that's why my interaction would feel like I am on these points.

17:51 But if I am approaching, and if I'm moving from here to here,

17:57 then you would feel like somebody's popping me out from the side.

18:01 So, these are not realistic.

18:04 This is not something that we would

18:05 expect from a good force rendering implementation.

18:08 Would that be helped by making it faster?

18:12 Um it could be helped, but also it is about the characteristics of the device.

18:17 Each device has a limit on how much stiffness that they can tolerate,

18:21 and obviously about the the frame rate as well, the server rate.

18:27 Um so, I can still pop through objects, and it is not very desirable.

18:33 Um so, how we uh eliminate these in haptic rendering is through proxy algorithms

18:47 where uh I actually have almost shown you the first idea that comes to mind.

18:55 If I am rendering force,

18:57 instead of computing just the distance about this uh this piercing distance,

19:04 I could compute an error, and I could try to minimize that.

19:08 Right?

19:09 And the error would be the error between

19:12 where I should be and where I am, literally.

19:16 So, instead of actually trying to represent the distance to the line,

19:20 I could define a proxy point, which would solve the issue of jumps,

19:26 and I could compute a spring between the proxy and the and the object.

19:33 So, at any time, you could look into where

19:36 the proxy is by looking into the collision section,

19:40 and you could compute a point where the proxy should be residing,

19:44 and make the proxy walk over the object, and make your hip on the attracted

19:50 to that, so that you would eliminate the error.

19:54 to get back to where the proxy is at all times.

19:56 Yes.

19:56 Okay.

19:57 So, this is sometimes called "end god object algorithm",

20:01 sometimes say "proxy-based rendering".

20:05 Um and it is a much better way to display forces.

20:09 Um So, that's all.

20:11 And this proxy proxy method is a method for haptic rendering,

20:14 but they also are used for human-robot collaboration.

20:18 So, in our research,

20:21 we typically use proxy points for human-human or human-robot collaboration,

20:26 where you have a robot interacting through an interface point,

20:31 a human interacting through another interface point,

20:35 and they get connected through a proxy,

20:38 which then connects to the task and operates it.

20:41 And this way, you could do a lot of intelligent

20:44 things like assigning dominance behaviors to the human and the robot,

20:48 separating their inputs from each other, so you could analyze them better,

20:53 so that the proxy becomes not a proxy for rendering only,

20:56 but proxy for agreement between the two actors.

20:59 This is like an ongoing open problem.

21:01 It's not something that's considered solved, is it?

21:03 Or is it just a different ways of approaching it?

21:05 Um I think you could consider haptic rendering uh in some ways solved.

21:14 So, there are mechanisms that you could simply do things,

21:17 but there are still uh issues,

21:19 for example, when you represent material properties,

21:23 such as uh what exactly I am feeling because we didn't really talk

21:27 a lot about friction or uh material properties that make this feeling realistic.

21:33 What What is it that I feel when I touch a sponge?

21:37 Something like this or something like a table.

21:39 So, these there there are libraries to uh

21:42 enable this kind of learning of object properties.

21:46 Uh also, transferring the haptics over distances and so

21:51 on is a I think still an open problem.

21:54 So, there are coding mechanisms just as we do video coding for haptic coding,

21:59 but uh it is still still open.

22:02 We can't really tolerate very delayed communication with haptics.

22:06 So, there's a lot of work going on in teleoperation architectures,

22:10 but also packaging the haptic information and sending sending in distances.

22:23 If the goal is to move from start to goal,

22:29 and if I can't actually uh have an analytical solution, I'll do things.