CUDA Live: Scaling HPC with Multi-GPU Communication Libraries
NVIDIA Developer
1:04 Hello and thank you for joining us for our CUDA live stream today.
1:08 I'm super excited to present people who
1:11 are working on our NVIDIA communication libraries today.
1:16 Just as a quick reminder in the comments
1:19 you'll see some NVIDIA developers out there.
1:22 Please start putting your questions there.
1:24 We might answer them in line.
1:26 We're going to be taking questions after
1:28 the team has had an opportunity to you know,
1:31 go through a few presentations and demos
1:33 and things like that but please you know,
1:36 start loading up your questions and I'm going to queue
1:38 them up when we get to the that Q&A.
1:42 So just to kick us off.
1:44 My name's Katrina Real.
1:45 I'm an engineering manager for the cuNumeric team
1:49 and with that I am going to be basically just
1:53 acting as moderator and I'm going to hand it
1:56 over to Michael who's going to start off our presentation.
1:59 Thanks.
2:03 Sounds good.
2:05 Hi everyone.
2:06 My name is Michael Wang.
2:08 I work for the CUDA Python team and I contributed
2:11 to the number CUDA compiler frequently and in this presentation
2:16 I'll be talking about nickel and a little bit
2:18 about number bindings for nickel for pi as well.
2:22 I'll hand it over to Ben.
2:26 Sorry, I was on mute.
2:27 I'm Ben.
2:28 I've been with NVIDIA for a couple years.
2:31 I work on NVSHMEM and nickel multi-GPU communications libraries.
2:36 We try to just get the most out of multi-GPUs
2:39 when you're using more than one in your job.
2:41 Make it as efficient as possible.
2:43 And I'll hand it over to Arnav.
2:46 Hi, I'm Arnav Goel.
2:47 I've been at NVIDIA for about 2 and 1/2 years.
2:50 I've been working on multi-GPU communication libraries just like Ben.
2:54 Right now I've been recently focusing on gym multi-GPU
2:56 communications architectures for nickel and NVSHMEM and as Ben
3:00 said we try to squeeze out the max performance
3:02 we can out of the GPUs while doing multi-GPU communication.
3:05 And I'll hand back to Katrina.
3:09 Actually I think Michael you're going to start off our presentation, right?
3:13 Yeah, I mean in that case can Zach bring up the slides?
3:19 Are we going to start with the slides?
3:20 Um Right, right.
3:23 Um So when it comes to like communication libraries I think there are
3:28 quite a few ones out there in the open source community and you
3:32 will often hear about the word nickel and which is stands NCCL
3:37 and that's like in full name
3:38 spelling that's like NVIDIA collective communication library.
3:42 It's like super popular among you know,
3:45 widely adopted across the community like in torch and in other
3:48 uh famous HPC and deep learning library as well.
3:53 Um and for those who are not do do not
3:56 know familiar with this like NCCL pronounced nickel it's pretty much
4:01 a library that's delivered developed
4:03 by NVIDIA for efficient communication and you
4:07 use that to perform communications between multiple GPUs because you know,
4:11 nowadays the workloads always spans across multiple GPUs.
4:15 Single GPU won't be able to host all of the data that you need to process with.
4:20 Um and nickel provides like a super easy to use interface where you could do
4:25 point-to-point communications and collective operations.
4:28 We'll come We'll come back to those terms in a bit about what that means.
4:32 Um and ever since you know,
4:35 version 2.28 we also added a device library for nickel so you could actually
4:41 do perform uh device side communicate device
4:44 side initiated communication on CUDA kernels as well.
4:48 So and we'll also come back to what
4:50 device side initiated communication means in a bit.
4:54 Um next slide.
5:00 Um I'll just give a quick overview of what APIs are
5:05 available across nickel because in a bit we'll have a live
5:09 stream event sorry a live coding event where I'll actually
5:13 show how to use these some of these APIs in live.
5:17 So in general there are two families of API.
5:19 There's send and receive APIs and there's collective operations like all reduce,
5:25 broadcast and reduce.
5:27 For send and receive this is what we call as point-to-point APIs one
5:33 a GPU will initiate one of the send and another GPU will initiate a receive
5:38 and this is a kind of things we call a two-sided communication APIs where both
5:44 side the send sender side and the receiver
5:46 side will have to initiate those API calls.
5:50 And for collective operations I actually have an image below
5:54 to showcase the use of an all reduce collective operations.
5:58 This is kind of operations where you have to take all
6:00 of the GPUs in your coms world to into into account.
6:06 So say you have four GPUs and all of them holds
6:09 separate data in the different data independently and after you have
6:14 initiated all reduce example all of those GPUs will hold exactly
6:20 the same reduced data as the result of this all reduce.
6:24 In this case I'm using a sum reduce operation and which means all of the GPUs
6:30 will hold the the summed result that was
6:34 originally scattered across all the all the GPUs.
6:37 And from this point on we can move on to do some
6:42 live coding but before I do that is there any questions queuing up?
6:48 Regarding the slides.
6:56 I guess not.
6:58 Zach can you bring up the live coding?
7:02 All right, all right.
7:03 So because you know, doing live coding on on on multi-GPUs
7:10 and and parallel programming is pretty unintuitive.
7:13 So I invite our friend Lil Mike right here to come show
7:17 us how to do programming of of nickel APIs in actual Python.
7:23 So first of all thanks to our friend Ben here.
7:27 He has set up this very nice Jupiter hub environment for us
7:30 where we could do multi-GPU multi-process
7:35 programming just within the Jupiter lab environment.
7:39 Um first of all let's take a look
7:41 at this hardware environment that that we currently have.
7:45 So I will go ahead and run this code.
7:49 So I think a lot of people are familiar with this NVIDIA-SMI command.
7:54 I don't think it's widely known that there's this topo NVIDIA-SMI topo-m
7:59 command where you could actually show
8:02 the entire topology of your hardware system.
8:05 So what this thing actually show here is that it it
8:08 gives us a grid view of communication capabilities between those GPUs.
8:15 So if you take a look at row GPU zero and column
8:20 GPU one there's these things that says node and what does node say?
8:24 We can come down to the legend and says it's connect connection traversing
8:28 PCIe as well as interconnection between PCIe host bridges within a NUMA node.
8:33 Okay, that's a lot of a lot of word but at least we can see
8:37 that this gives us a pretty complete topology view of um of our hardware setup.
8:45 And just to save every everybody's time I've actually like
8:49 pre generated an image for our hardware setup over here.
8:55 So we see here is that this system has two NUMA node and each NUMA node has two
9:00 GPUs and between these NUMA node there's a SMP
9:05 interconnect and within single NUMA node the GPUs themselves
9:09 are connected through PCIe host bridges and there are
9:12 three NICs which stands for network network interface cards
9:16 and all of them sits on NUMA node zero
9:18 and connected through the PCIe host bridge as well.
9:21 Okay, so this is the system we have for uh for for this setup and I think
9:29 this is the same system that our friends will
9:33 NVIDIA will also present for other coding example as well.
9:37 So just to run this boilerplate code for starting
9:41 a MPI cluster for um for this lab environment
9:49 and see it's finished um and for those who are
9:53 not familiar with MPI MPI stands for message parsing interface.
9:59 It's pretty much like a very common tool
10:03 that people use to manage environments with multi-node processing,
10:08 multi-GPU environment, and that kind of stuff.
10:11 And a lot people will actually in a terminal
10:15 environment that we use MPI run MP4,
10:17 which will launch something called like an MPI world,
10:21 which is like a global view of all
10:22 the processing power you have in your environment.
10:25 And in in our case, we have four processes,
10:28 and each of them is assigned with a rank number,
10:31 and some of them is like rank zero, one, two, and three,
10:34 and all of them has holds a has access access to a one GPU.
10:39 And And as you can see here, as you can you can imagine this like four
10:43 separate workers who's doing work in your environment separately.
10:48 And you will see this uh%% px cell magic,
10:54 which means that cell that cell is being
10:56 executed on all of the workers in your environment.
11:01 All right.
11:02 All right.
11:02 So, that's in our environment setup, and now we're actually going to talk about
11:07 the examples we're going to show with nickel.
11:10 Um And specifically, I'm going to demonstrate something called a ring shift.
11:15 And what's a ring shift mean?
11:17 That basically means um in we have four GPUs,
11:23 and all of GPUs has their unique data.
11:25 Each of the GPU will pass its own data to the next GPU.
11:31 So, it's basically its next neighbor.
11:33 And uh that happens for all of the GPUs,
11:36 and for the last one, say the in our case,
11:39 I call it rank three, will pass its data back to the first GPU, so rank zero.
11:45 So, basically zero passed to one, one passed to two,
11:48 two passed to three, and three passed back to zero.
11:51 And this is the initial state.
11:53 All of them has data A.
11:54 Rank zero has data A, one has B, two has three two has C, and three has D.
12:01 And after you have finished the whole ring shift,
12:05 so one ring shift step, rank zero will have data D,
12:10 and rank one will have data A, rank two will have data B,
12:12 and rank three will have data C.
12:16 All right.
12:17 Um So, first of all,
12:20 as we do any Jupiter Lab coding, let's import the necessary libraries.
12:24 Something to look out for is we use
12:26 a Python bindings for MPI to do communications.
12:29 We import PyTorch to do host side management memory management.
12:35 And then, the most important thing here is
12:37 we import nickel core library to do the communication.
12:41 And uh we can ignore some of these warnings here.
12:47 Um And the first thing we'll do is we will call MPI comm world to uh access,
12:57 you know, um to access the global information passed down passed down by uh MPI.
13:04 And the first API we use is get rank
13:06 to receive the local rank number of of this current process.
13:13 Cuz remember, this this cell is executed on all of the processes,
13:16 and each of the process will have its own local rank number.
13:20 We also call get size to get n ranks number,
13:24 and that means the total number of processes that exist in the comms world.
13:30 And then, we'll arbitrarily set rank zero as the root rank.
13:35 And then finally, we assign um for for each of the process,
13:40 we assign a CUDA device to it.
13:43 And we do it by torch.cuda.set_device.
13:47 Um And so, let's wait a minute for that to finish.
13:54 Okay.
13:54 So, after we have done the setup,
13:58 the next I'm going to do is actually to set up nickel.
14:02 So, the way how nickel API is is set
14:05 up is that everything is exposed through a communicator.
14:09 So, the first thing we'll have to do is set up a nickel communicator.
14:12 And by doing that, we'll first have to create a a variable called unique ID.
14:19 And that unique ID uh we'll set we'll create
14:22 a unique ID on using just one of the ranks.
14:25 And then, we use a broadcast mechanism to send that unique
14:30 ID to all of the other processes in in in our world.
14:34 And so that all of the uh basically, this is basically like a alignment process.
14:39 So, all of the all of the processes would know, you know,
14:43 how can I actually speak with like a global manager called nickel communicator.
14:49 And then, each of them will use that unique ID to initialize a um
14:55 a communicator uh variable locally for each of the for each of the ranks.
15:01 Um And it all comes down to pretty much
15:04 just like pretty straightforward simple uh three lines of code.
15:10 Um I think it's called get unique ID.
15:15 And that creates a unique unique ID variable.
15:19 And then, we also use a um I think it's the uh a broadcast API
15:31 to do that to send uh the um unique ID to all of the uh processes.
15:47 Um And then, oh.
15:52 This is I just missed one thing.
15:53 So, Oh.
15:58 So, the the unique ID is only created on the on the on the root rank.
16:05 So, this this execution is only happening on on the on the root rank.
16:10 And then, we use MPI comms to broadcast that unique
16:13 ID to all of the uh all of the other processes.
16:17 And finally, we use this unique ID to initialize a nickel communicator.
16:22 And we can do that by uh calling the nickel.communicator .init API.
16:32 [clears throat] And with the arguments we're going to pass in the total
16:34 number of ranks we have in the in the comms world,
16:38 um its local rank number, as well as the uh the unique ID.
16:46 All right.
16:47 Let's see if that works.
16:50 All right.
16:50 It works.
16:51 So, always keep in mind that this cell is executed on all of the processes,
16:55 and now all the processes um will have its own nickel communicator,
16:59 and they can basically talk to each other through nickel in that case.
17:04 And in the next cell, we will set up some of the local buffers,
17:09 GPU buffers to uh so that we can use that to to host data to send,
17:15 as well as buffers to receive data receive data.
17:19 So, let's run that.
17:23 So, as you can see here,
17:26 each of the rank holds a PyTorch tensor with a single float.
17:33 And each of them has unique data.
17:36 So, uh because the they are initialized with 100 plus rank.
17:42 So, rank zero will have uh number 100.
17:46 And rank one will have 101, rank two will have 102,
17:50 and rank three will have 103 in that case.
17:53 And um in the next step,
17:57 we will compute the peer number to send and receive the data to.
18:02 Now, um if you still remember this ring shift mechanism,
18:08 each rank will have will need to actively do a send and receive.
18:14 So, it will each each rank will have to independently compute its own
18:19 peer to send to, as well as its own peer to receive from.
18:23 So, in that case, rank zero will need to compute
18:26 rank one to as as its data as its peer to send data to, and compute rank three
18:33 to receive as the peer number to receive data from.
18:36 And so goes to rank one, rank two, and rank three.
18:40 And how we do that?
18:41 Um in fact, uh we're just going to use a modulo number to do that.
18:50 And uh I think in that case,
18:54 we'll just be using its local rank number to plus one.
18:59 Oh.
19:00 Didn't know that it doesn't automatically do that for me.
19:03 Um the parentheses thing.
19:05 Um And then, you can just use the modulo to uh uh Basically,
19:12 for the for the last for the last rank,
19:14 it will wrap around back to rank zero for that in that case.
19:18 And the same thing goes for receive peer,
19:22 can use rank minus one modulo by n ranks.
19:27 And uh for those who are not familiar with like negative number n ranks,
19:33 it also wraps up to the uh the last rank in that case.
19:38 So, after you computed that, um we will basically each of the each of the rank
19:45 now knows which um uh peer to send data to, and which to receive data from.
19:54 So, rank zero knows in it needs to send data to one,
19:57 and needs to receive data from three.
20:00 Um now, all comes down to actually making
20:04 the nickel uh send and receive calls here.
20:09 But, one thing we have to keep in mind is that um for nickel APIs,
20:13 it's all blocking calls.
20:15 So, if you invoke a a receive call,
20:20 the function won't actually return until the um the data receive has finished.
20:26 And that means um if [clears throat] uh if the user did something wrong,
20:31 it can completely cause a a a deadlock across all of the nodes.
20:34 So, for example, in some cases, uh all of the nodes will be start waiting
20:39 for um some of the other nodes to be sending data,
20:42 and none of the nodes can can progress in in that case.
20:45 So, it is using send and receive, it's up to user's responsibility to make sure
20:52 that um the send and receive pairs match, and all of the ranks are uh has it
20:59 will be able to guarantee uh progress in that case.
21:02 Um and I can pretty much easily show
21:05 an example of a deadlock example [snorts] here.
21:08 So, say, remember we we initialize the communicator as nickel com,
21:12 so we can do um uh I can do a receive.
21:17 I think that's called uh uh receive data.
21:23 So, that means we have to put the data
21:25 into the uh the the torch tensor we initialized earlier,
21:29 and call it by receive peer, and then nickel com dot send send data peer.
21:40 dot send peer.
21:42 Right.
21:42 So, if you try to run this code, it's easily uh deadlocks because in that case,
21:49 all of the ranks are waiting for data for other ranks to send data to.
21:53 So, none of the none of the uh process
21:57 will be able to progress over this line end,
21:59 which means no rank will be able to send any data.
22:03 So, I mean, ring shift is like a super simple example,
22:07 but there are much there are other more complicated use cases
22:10 where user will have to take care of these responsibilities themselves.
22:15 But, luckily, um nickel has invented something
22:19 like I in my opinion that's pretty genius.
22:22 It's called a group call.
22:23 And group call will comes to rescue in our cases.
22:26 Um so, in this in this uh uh uh uh demonstration,
22:31 the first part of it shows the deadlock case.
22:35 But, we could put we could sort of like
22:38 initialize a a group call context in that case,
22:42 and nickel uh will uh collect all
22:46 of the send and and receive uh [snorts] API calls,
22:51 and then automatically resolve for its internal dependencies,
22:55 and then actually figure out, right?
22:57 Like figure out the way to avoid the deadlock.
23:01 And that's one very simple use cases for nickel group calls,
23:06 but there are other things that's uh that group call can can provide.
23:10 For example, it will provide a better performance.
23:13 So, say, if in a group call where you try to one
23:16 of the rank is trying to send data to all of the other uh ranks,
23:21 and uh the group call will actually analyze that thing,
23:25 and then decide that, "Okay, maybe a scatter call is is is a better one."
23:30 Or, if all of the all of the uh sorry,
23:33 a broadcast a broadcast call is is a better option for it.
23:37 So, we'll reduce the number of um nickel calls in [snorts] in uh in there.
23:43 And then, for uh point-to-point operations like send and receive,
23:47 it will actually try to attempts to merge them into into a a single call.
23:52 So, we'll also gives you better performance of that.
23:55 So, we will actually use uh nickel group
23:59 to to write the send and receive in this case.
24:01 How do we do that?
24:03 It's actually pretty simple.
24:04 So, uh nickel for pi is provided uh a very nice python
24:08 pythonic uh context management interface where
24:11 you can just write with nickel.group.
24:15 And this creates a a context scope where um uh everything every nickel
24:21 call executed between will be captured in in this uh nickel group call.
24:26 And uh I will just do exactly the same thing as before.
24:31 Um dot receive receive data peer equals receive peer.
24:39 And I'm like I'm trying to like uh intentionally write the way where
24:43 this would actually go um uh uh
24:47 deadlock without without the the group call thing.
24:51 Send data peer equals send peer.
24:55 And then, um always keep in mind that um within a nickel group,
25:01 the data transaction won't happen at right after this function returns.
25:07 Well, actually, all of them will take place all
25:09 at once after you've exited um this this context.
25:12 So, I'll just go ahead and and call it.
25:18 Uh sorry, I I see there's a question.
25:20 Uh what was that?
25:24 Uh yeah, we can go ahead and take a question.
25:26 We had a question uh coming in from YouTube.
25:31 Does nickel always rely on MPI for setting up inter-node communication?
25:35 Uh I believe not.
25:38 Um for example, I think the only part that depends
25:41 on MPI is in this example is this broadcast,
25:45 but in fact, any kind of uh communication mechanism that you
25:50 can send that unique ID across uh processes will work.
25:55 And that means, you know, TCP/IP protocol, socket-based communication,
25:58 or like even just simple file sharing um will will all of them will work.
26:04 So, nickel does not uniquely depend on MPI to work.
26:10 All right.
26:12 Um and as we speak, this cell has finished executing,
26:17 and we can now um come to verify the result.
26:20 First, we'll synchronize over the device to make sure the data is
26:23 uh all of the operations have finished on each of the streams,
26:27 and then we'll print it out on the result.
26:32 And uh let's see.
26:34 Um Perfect.
26:37 So, the way how we read it is for rank zero, it had sent data 100,
26:43 and received data 103,
26:45 and which means it has already received data from the last rank.
26:48 That's a good that's a good sign.
26:49 For rank one, it sent 101, and received 100.
26:53 Uh for rank two, sent 102, received 101.
26:56 And for rank three, sent 103, and received 102.
27:00 So, great success.
27:02 And after you have finished your um your workload,
27:06 don't forget to always call the nickel com
27:08 destroy to uh for uh environment clean up.
27:14 Um and uh that's it.
27:17 That's uh a simple pretty introductory example for uh nickel for pi usage.
27:23 And now, I'm going to hand it over to Arnav,
27:24 who's going to show us a more intricate and and delicate example here.
27:29 I'm going to run I'm going to I'm going to get
27:32 in here just real quick cuz there was another question that was related
27:35 to this, but um what I heard is that is that you
27:38 can use your multi-call to allow more calls to be made over time.
27:42 Is that correct?
27:48 Um Can you use your multi-call to allow more calls to be made over time?
27:54 I guess is the question.
27:56 I can read that one.
27:57 Yeah, Arnav, please.
27:58 Yeah.
27:59 So, um I think the group extraction that Michael talked about
28:02 is basically like a deferred send or a deferred receive operation.
28:06 So, what it allows you to do,
28:07 it allows you to batch together multiple operations that are not executed
28:10 at the time when the operation exist in program order in the code.
28:14 So, as Michael was showing when he has the receive or the send,
28:17 the operation doesn't get executed right away.
28:20 When the with abstraction ends,
28:22 which is a context manager abstraction in Python,
28:24 there is an implicit group end that happens,
28:26 and that maps to the nickel group start and group end APIs,
28:29 which are on the C side.
28:30 I think folks on from from the team have also shared links to those APIs,
28:34 so do check those out.
28:35 That's the point at which those operations actually get executed,
28:39 and that's what avoids the deadlock issue because
28:40 you don't have the situation where receives and sends
28:43 are not ordered correctly across across multiple processes
28:46 to cause a deadlock where everybody's receiving or everybody's sending,
28:49 and no one's making forward progress.
28:55 Awesome.
28:56 Let's uh move on.
28:57 Thanks.
28:59 Thanks, Michael.
29:00 That was a really good example to go over the basics of nickel API,
29:04 and kind of walk through how to do a point-to-point for a ring shift.
29:07 So, what I'm going to be covering next is I'm going to be explaining how
29:10 a distributed general matrix multiplication problem can be
29:13 implemented using the likes of nickel for pi.
29:17 Um this is a common problem that both HPC and AI workloads
29:20 kind of are involved in where there is a uh um you know,
29:24 matrix multiplication that occurs between two or more um operands computing
29:29 some intermediate results that eventually requires to have um some kind
29:34 of uh reduction operation that that needs to be performed across
29:38 all the participating ranks such that everybody has the final result.
29:41 Think of it like if there are two or four GPUs which
29:44 are doing their own local gem as it's called or general matrix multiplication.
29:50 They in they produce intermediate results and there's
29:53 a requirement to be able to do some type of a collective operation to make
29:57 sure that everybody has the the final result, which is the you know,
30:01 the aggregation of all the results from all the other participants.
30:05 Um, so in this example, what I'm going to be walking over is similar
30:09 to the case that Michael showed how to set up your environment
30:12 and how to use the nickel APIs and this is going
30:15 to be a bit of a live coding exercise as well.
30:17 I have some boiler plate code which is already set up.
30:19 So for example, the first part that runs here,
30:22 much like the example you saw before,
30:24 is going to set up the the the MPI launcher in the background.
30:30 Um, it's basically we're using MPI
30:31 to launch multiple processes mostly because it's it's
30:34 easy enough for for us to kind of use that to bootstrap the processes,
30:37 but as Michael pointed out,
30:38 it could have been any other launcher that we could have used.
30:41 Um, um, and then in this cell,
30:44 what happens is we're importing all of the different
30:47 Python libraries that we are using in this exercise.
30:49 So we're going to be using MPI for Pi
30:52 to basically do the bootstrapping of the control plane to exchange
30:54 your unique IDs or to exchange any seed or any
30:58 other information we need in the control plane of the application.
31:01 We're going to be using torch to set up our data,
31:03 our tensors essentially that each processes are
31:06 going to be using to store the results,
31:08 either intermediates or inputs and outputs.
31:10 We're going to be using nickel for Pi for our multi-GPU communication and we're
31:14 going to be using the likes of a library from Nvidia called NV math,
31:18 which is basically you know, a really cool another really good Python
31:22 ecosystem library as part of our you know,
31:24 as part as part of our CUDA Python stack that allows us to do you know,
31:28 fast and efficient matrix multiplication and other HPC algorithms.
31:32 So this just sets up pretty much that environment.
31:36 Um, once that's done, the next thing we ought to do is,
31:39 you know, make sure that the MPI bootstrap is actually working.
31:42 So what I'll have here is basically just
31:45 an output that kind of demonstrates you know,
31:47 how big my world is, how many ranks do I have in the system.
31:50 I launched this particular instance with two ranks,
31:53 so that's what you see in the output here that I'm I'm I'm
31:57 I initialize my MPI comm world and I got back the number of ranks
32:01 in this variable and that's what I I'm I'm basically showing here
32:04 as the environment is set up and I'm printing that output on the first rank,
32:08 which is rank zero in my use case.
32:11 Um, and then next I'm going to be essentially
32:15 having my ranks bind to an individual CUDA device.
32:19 So on the system I have you know, two or four devices.
32:22 I'm using two devices, device zero and device one.
32:25 So I have this mapping of rank zero to device zero or rank one to device one,
32:30 which I'm I'm setting up using torch.
32:32 Torch again has these abstractions on top of CUDA runtime
32:35 and CUDA driver APIs that allows us to basically you know, set up the device um,
32:41 and and initialize CUDA runtime and then
32:43 now we're ready to use the other libraries.
32:46 So at this point, what has happened so
32:48 far is the the processes have been launched,
32:51 the local CUDA devices have been created and initialized,
32:54 MPI bootstrap is ready to go and now we need to set up the nickel communicator.
33:00 Communicator is basically the abstraction that allows
33:03 us to do the multi-GPU communication.
33:05 It abstracts away all of the underlying communication
33:08 resources that are being used for multi-GPU communication.
33:10 This communication could be happening over various different interconnects.
33:15 It could be a a networking based interconnect that's on the system.
33:18 It could be an NVLink based interconnect on the system.
33:21 It's the it's the responsibility of the nickel
33:24 library to decide what's the most efficient
33:26 interconnect to use for a given piece
33:27 of communication or for a given collective operation.
33:31 Um, so the first thing I'm going to do
33:33 here is I'm going to basically allocate my unique ID.
33:37 So I'm going to call this get unique ID function and I'm
33:41 going to do this in a way where I I check if um,
33:45 my rank is equal to zero, else it'll be none.
33:50 Um, so here I would have essentially allocated a unique ID for the root process,
33:56 which I'm assuming in my example to be zero and then I would have
34:00 broadcasted that unique ID across multiple processes
34:03 such that everybody has the same unique ID.
34:05 Think of unique ID as just like a an address
34:09 or an IP address or a location or some sort of a unique
34:13 identifier that tells nickel essentially and the ranks that how
34:17 to connect to each other and how to bootstrap to each other.
34:20 Um, in in one implementation, it could be an IP address of you know,
34:25 one of the underlying NICs on that system that is
34:28 being used to bootstrap the communication or to bootstrap the communicator.
34:32 So once I run this cell,
34:34 the unique ID should be created and broadcasted to all ranks and then
34:37 subsequently now I need to be able to initialize my nickel communicator.
34:41 So I will do this using nickel communicator dot
34:47 init and I would pass in my rank is rank,
34:51 my n ranks is is n ranks and my unique ID is my unique ID.
34:59 I hope I got the I'm just checking.
35:01 I hope I got the the variables correct.
35:03 Yep, that should be the right way to do it.
35:06 Um, and then once I execute
35:08 this, my communicator should be initialized across the processes.
35:13 Yep, so nickel communicator is initialized across the two ranks.
35:16 Um, and this is a collective API, meaning once this API is invoked until
35:21 all ranks have have joined the communicator,
35:23 the API will not make forward progress.
35:25 Once this API completes,
35:26 that means all the ranks in my in my in my application have
35:30 the communicator created and they they can
35:32 now participate in point-to-point or collective communication.
35:36 Um, now we can dive into the part of the application
35:39 that we're trying to basically build this for, which is a distributed gem.
35:42 So the first thing we have to do is we have to create the input data,
35:46 which is typically organized as multi-dimensional matrices.
35:49 So we have this A and B matrices, which are basically these torch tensors.
35:54 So one is M by K, the other is K by N.
35:57 These are typically the shape and dimensions
35:59 of of those multi-dimensional arrays as you see.
36:03 So what this piece of code is doing is this piece of code
36:06 is basically creating random tensor objects on on both the devices and it is
36:12 making sure that all processes are aware about what each of those individual
36:16 A and B inputs are and then once those once those local tensors are created,
36:23 it's making sure that you know,
36:25 just printing out basically that what the shapes of the tensors
36:28 are and and whether those torch objects have been created.
36:31 At this point, these tensors or these input data
36:35 or these these these these these these these matrix, they live in host memory,
36:39 meaning they are not yet visible to the GPU memory under the hood.
36:43 Um, what we're doing in the next step is we're
36:46 now going to be able to move them into GPU
36:48 memory and we want to partition the the matrix such
36:52 that each of the of the of the two GPUs, they operate on a local part.
36:58 So you can see here I have this matrix on the left,
37:01 which is A, which is of dimensions M by K and I have a matrix on the right,
37:05 which is B, which is of N by K and I'm just going
37:08 to splitting the matrix into two halves
37:10 because I'm splitting it by number of participants.
37:12 So here I have two ranks, therefore two halves.
37:15 And I have the first rank operate on this part on the part of M by K local,
37:20 which is going to be this first half and I
37:22 will have the second rank operate on the second part,
37:25 which is this in in as explained in this in this slicing scheme.
37:30 It's going to be the second half of this M by K local matrix
37:33 and I'm going to a similar partitioning
37:35 for but a column column major partitioning
37:38 for for the other matrix that you can see here where the first half is
37:42 going to be rank zero and the second half is going to be rank one.
37:45 And the idea is that you now want to do
37:47 local matrix multiplications on the first individual halves so
37:51 that you you do an you do you compute your local
37:55 partial results on each of this of the two GPUs.
37:58 We call that result as partial C.
38:00 Um, and then there needs to be a multi-GPU collective operation,
38:04 which is an all reduce to make sure that we aggregate the results
38:08 across all the GPUs so that at the end of that operation,
38:11 all the GPUs have the exact same total value of of C that's
38:17 visible and then it can proceed to do some other computation on it.
38:21 Um, so this is how we're slicing the data across the the the two
38:26 ranks and this is what we will see each of the ranks compute.
38:30 So here I have a piece of code that basically partitions my data by based
38:34 on my rank rank ID and this dot two operation that you see here,
38:39 this is an operation that torch exposes to allow you to move
38:43 this tensor from the host to the GPU memory and at that point
38:47 the data lives in GPU memory and we can use it
38:49 to perform both the matrix multiplication as well as the all reduce operation.
38:55 So I'm going to execute this piece of code and it's basically going to have
38:59 the data be partitioned and it will have
39:02 the data move be moved to the device memory
39:04 of the GPU memory on this on this platform and you can see the outputs of what
39:10 are the individual local pieces that each
39:12 of the rank A and rank B will operate on.
39:17 And then next I'm using a another CUDA Python library,
39:21 which is called NV math to basically perform the matrix multiplication.
39:24 This is just a convenience that that that I
39:26 really have from the CUDA Python ecosystem.
39:29 There are a bunch of other math libraries you could also use,
39:31 but this is one of the more popular ones.
39:33 So I'm using this NV math.mat mul
39:36 to perform the local matrix multiplication between
39:39 A and B and it's going to give giving back the partial result C operation.
39:45 Um, and this is summing over the partial products
39:47 to give you the result of A and B.
39:51 So, this computes this operation.
39:54 Now, at this point, this operation is being done on the GPU.
39:58 It's being launched on the GPU.
40:00 And and it's executing on the GPU.
40:02 And then one thing that that that that that you do is you could either
40:08 synchronize the operation right here and then perform
40:12 the nickel operation for doing the all reduce.
40:14 Or what would be a preferred thing to do is you can just go ahead,
40:18 do nickel all reduce.
40:21 And you basically save partial C.
40:25 Partial C.
40:27 And you say reduce.sum.
40:30 I'll just make sure my my syntax is correct there.
40:35 I think it's called nickel.sum.
40:37 Probably that's the typo I have here.
40:40 Yep, it's called nickel.sum.
40:42 All right, so that should that should theoretically just work.
40:46 Okay, so let's say go ahead and execute that.
40:50 And now I've taken the partial sum then the all reduce
40:55 and now the partial C variable actually holds the total results,
41:00 which is the result of the entire
41:01 matrix multiplication being done from both GPUs, right?
41:05 So, it has it has the accumulated results.
41:08 And once this operation is done,
41:10 then I have the step to verify and make sure my operation was done successfully.
41:15 So, what what is happening here is I'm
41:18 also doing a complementary matrix multiplication of the total
41:21 data without the partitioning on on the CPU
41:25 and I'm comparing the result of that against the results that the GPU gave me
41:28 for the all reduce and I'm checking is it within
41:31 the error tolerance that I have for using
41:34 the torch.odclose API and the answer is yes, it does.
41:38 And once you're done with the all reduce operation at that point,
41:42 now you can destroy your communicators.
41:44 And that's it.
41:46 That completes the exercise for how you would want
41:48 to do a distributed gem using nickel plus plus all reduce.
41:53 Um using gem plus all reduce.
41:56 Um, I hope folks found that to be useful.
42:00 If there any questions, we can take them now.
42:02 Otherwise, we can switch over to Ben.
42:06 All right, we're going to go ahead and take a couple of questions here.
42:09 So, um First one, [sighs and gasps]
42:14 how do you CUDA communication libraries abstract
42:16 hardware differences across GPU generations and interconnects?
42:21 That is a that's a great question.
42:23 So, and the answer is of course as all good questions,
42:27 it depends on what you mean.
42:29 So, the nickel API and the the NVSHMEM API that we're about to talk about
42:33 in general in the in the next section
42:36 are we call them they're they're somewhat portable.
42:39 So, they hide the complexity for you.
42:41 We do a lot of library engineering inside of our libraries to make sure
42:45 that the user experience is that the same
42:47 thing happens from GP generation to GP generation.
42:51 Um there are other sort of more
42:54 complexities related to the different underlying hardware transports
42:59 that you're using whether you're doing data movement
43:01 across NVLink or um network or things like that.
43:06 And I think the the section of the presentation that Michael is going to cover
43:09 right at the end is going to directly answer that question kind of perfectly.
43:16 Awesome.
43:16 Let's take one more.
43:17 We got another question from the same poster.
43:22 What are the biggest bottlenecks when scaling from single
43:24 GPU to multi GPU systems in real world HPC workloads?
43:30 Um, I'll do the the simple answer which is locality and then
43:34 I'll let Arnav who I saw wanted to talk expand on that.
43:38 Yeah, I mean it it really depends on the kind of communication patterns
43:41 that you run into when you go from single GPU to multi GPU, right?
43:44 So, one of the examples I gave was you know,
43:47 you have this really dense gem and you want
43:49 to be able to partition this across multiple GPUs.
43:52 And now all of a sudden you've converted a compute bound
43:55 problem into both partial compute and communication and memory bound problem.
43:59 And if you have a really good underlying system
44:02 like the GPUs that we're that we're building, you know,
44:05 we can solve some of the memory bound problem
44:07 by making sure as Ben said have the data relocated
44:10 on my GPU memory and access it as quickly as I
44:12 can through the HBMs that that are there on our systems.
44:15 And also by having these highly efficient interconnects like NVLink,
44:19 we can basically move pump a lot of data
44:22 something like an all reduce operation more efficiently.
44:26 So, you you would want to take you would want to think
44:28 about scaling of a problem as as a function of number one,
44:33 if my original problem was compute bound and now I've
44:36 turned it into a memory or an IO bound problem,
44:38 then what are the components of the hardware that I'm actually using like
44:42 what are the components of software I'm
44:44 using to use those hardware pieces efficiently.
44:47 Am I going over the network multiple times in a latency bound problem?
44:51 If that's the case, I should probably think
44:53 about reducing the number of trips I do
44:55 over the the network and that requires restructuring
44:58 the algorithm so that I have fewer trips.
45:00 If I'm going constantly to you know,
45:03 my non-local memory versus my own local memory,
45:06 then I'm again introducing some latency components there.
45:08 Or if I'm using an interconnect which
45:10 is more efficient for latency and not bandwidth,
45:13 then I'm basically making my communication part much slower.
45:17 So, I've shifted the problem statement somewhere else.
45:19 So, you know, the more detailed answer is you need to think
45:23 about your algorithm as a function of [music] what parts are compute,
45:27 what parts are memory and what parts are IO bound.
45:30 And you want to restructure those pieces such
45:32 that you balance all those stages together with as much
45:36 overlap as you can to make the end-to-end problem
45:39 more more efficient on on on a given system.
45:42 But it really depends on the use case that you have that how
45:45 to make that particular use case more more scalable and more more efficient.
45:50 And and it is no one general solution for that other
45:53 than just those first principles that I kind of briefly mentioned.
45:59 Um, thanks Arnav for that answer and um the there's two kinds of questions.
46:05 There's a good question, which is one that that leads a presenter
46:08 to give a long in-depth detailed answer.
46:11 And there's a great question, which is one that you already have a slide about.
46:14 And we just heard a good question and a great question.
46:17 So, thanks.
46:19 If you could Zach, if you could pull up my my two slides.
46:23 So, we've been talking about nickel so far.
46:25 I'm going to change gears and and have us talk about NVSHMEM now.
46:30 Um, so NVSHMEM is another communications library that our team maintains.
46:36 Um the interesting thing about NVSHMEM is that it's a more GPU-centric.
46:44 It's targeted at finer grained communications and it was the place where
46:48 we did our first experiments with GPU-initiated communication in a coms library.
46:54 So, if you look in this picture on the right-hand side of the slide,
46:56 you can see that the darker green thing on the left side is
47:00 a CPU and the brighter green things on the right side is a GPU.
47:06 And the and the put API that's being called, we'll talk about that in a moment.
47:10 It's being called from the GPU itself.
47:12 So, you don't actually have to go back to the host
47:14 in order to ask the network to do some work for you.
47:18 Um, the other thing I really want to mention is it's one-sided API semantics.
47:23 So, Michael talked about two-sided communication where one um process
47:28 does a send and the other process does a receive.
47:31 In NVSHMEM as we're going to see in the code,
47:34 um only one side is ever involved in the communications itself.
47:38 And and you'll see what that means when we pull up the code.
47:41 Um, next slide.
47:44 The other really important thing to know about NVSHMEM
47:46 is we use what's called a partition global address space.
47:49 You'll you might have heard of symmetric memory.
47:52 Those are sort of interchangeable terms in in this context.
47:55 What it means is that we at setup time when we initialize our library,
47:59 we reserve a special type of address space
48:02 such that across each GPU in the process,
48:05 all the objects that we allocate on that GPU's memory have the same address.
48:10 And that lets our library do some tricks under the hood that speed up
48:14 performance and make it kind of easier to reason about what data lives where.
48:20 So, the the simple thing you need to remember is that for symmetric memory,
48:24 you allocate it with a function like NVSHMEM.core.array.
48:27 That'll give you an array in symmetric memory.
48:30 And then for private memory that only your GPU only one GPU can access,
48:34 you'll just make it however you want.
48:36 So, this example I'm using CuPy APIs CuPy arrays.
48:40 Next slide.
48:43 The only other thing I want to introduce before we code is the concept of a put.
48:48 So, the way you think of a put is just like a mem copy.
48:52 Um, so a mem copy that takes an extra argument called PE,
48:57 which is the destination device.
48:59 So, this first call that you can see
49:00 at the bottom does a put into some destination array
49:05 from some source array and the PE is where
49:09 the destination where which GPU the destination array should be updated.
49:14 So, what we're doing is we're copying the green thing from the light
49:17 green thing from GPU zero to the same address on GP one.
49:21 And with that, I think we can go move on to trying this in practice.
49:29 So, we're using um we're using NVSHMEM for Pi in this.
49:35 Is the is the code visible?
49:37 I can't see it on my side.
49:39 There we go.
49:40 Um so we're using NVSHMEM for Pi.
49:43 Um so we're going to do all of this in Python.
49:45 Uh you can see the same kind of boilerplate
49:47 setup at the top that Michael and Arnav had.
49:49 I'll skip that.
49:51 Um what I'll mention is that uh similar to what Michael and Arnav were showing,
49:55 we have uh a set of CPU ranks that are
49:58 connected to each other via an an MPI communicator.
50:01 I've drawn them in blue here.
50:03 And then we have a set of GPU PEs.
50:06 Um the NVSHMEM word for a rank is a PE, a processing element.
50:11 So just keep that in mind.
50:12 That are connected to each other and they're sort
50:15 of pinned one to one um with the MPI ranks.
50:17 So there's a CPU process and then each one has
50:20 a corresponding GPU and set of GPU resources that it can use.
50:26 Uh this code shows you how to use um MPI to initialize NVSHMEM.
50:32 So we have a special way that we can pass a communicator,
50:36 an MPI communicator in um to NVSHMEM and then it will
50:42 for us it'll sort of automatically give us an out-of-band communication.
50:46 So it'll it'll automatically give all these green
50:48 guys um ability to use the blue MPI path to talk to each other out of band
50:53 and it'll do that bootstrap process for us.
50:56 Um NV NVSHMEM also has a unique ID bootstrap mechanism like nickel has.
51:00 So if you want to use Kubernetes or uh TCP/IP
51:03 or files or whatever to do a bootstrap, you can do that.
51:06 But a lot of our users care about MPI um as a bootstrap mechanism.
51:11 So we have a special convenience function for them.
51:13 So what this code is doing is it's
51:15 just doing some library setup and then it's going
51:18 to call this NVSHMEM init function and it's
51:20 going to print out however many PEs we found.
51:23 Um we should find I don't know where my output went, but we should find four.
51:34 Hm.
51:35 I'm just going to restart this kernel.
51:37 Oh, never mind.
51:38 It worked.
51:40 Uh so here's that's running.
51:43 The init is sometimes takes a minute.
51:44 And then here's the my PE.
51:46 So you'll notice that the the MPI rank
51:49 here is pinned the same as the NVSHMEM PE.
51:53 So that's what this tag and this tag mean.
51:55 Um I mentioned put.
51:57 So I have some code here that creates uh an empty symmetric array.
52:02 Uh it's two actually, a source and a destination.
52:06 I'm assigning the source to be my PE plus one.
52:09 And the reason I'm doing plus one is just so that we never see any zeros.
52:13 And then the dest is zero.
52:15 Right now I do um just print out before and after.
52:18 So you'll see the same thing.
52:19 It's it's before it's zeros and then after it's uh also zeros on the dest side.
52:26 Um what we're going to do is we're going to add
52:28 a an NVSHMEM API call here which is going to be a put.
52:34 Just like um just like we saw before on the slide.
52:40 So we're going to do uh NVSHMEM.
52:42 We're going to first calculate the peer and we're going
52:45 to do that exactly the same way that Michael did.
52:47 So I'm not going to talk about that in too much detail.
52:50 It's uh NVSHMEM.core.
52:53 dot my PE plus one mod NVSHMEM.core.
53:04 dot n PEs.
53:06 And uh again, PE and rank are the same thing basically.
53:11 And then we're going to do NVSHMEM.core.put from uh
53:16 into dest from source on peer on stream stream.
53:25 So NVSHMEM for Pi APIs are always stream ordered
53:29 which means that we put them on a CUDA work
53:31 stream and they execute um on that stream and then
53:34 we'll sync the device um before we look at results.
53:40 The other thing we need to do is we
53:42 need to add something called a quiet because NVSHMEM APIs,
53:45 let me see if I can show the picture.
53:47 NVSHMEM APIs are weakly ordered which
53:49 means that their effects are not guaranteed to be visible until you do some
53:54 kind of synchronization or or or visibility guarantee.
53:59 So in NVSHMEM we have this API called
54:00 a quiet which guarantees that all prior puts are visible.
54:04 So we're going to do NVSHMEM.core.
54:07 quiet and it's just going to take a stream.
54:13 Okay.
54:14 And let's try to run this.
54:17 And it seems to have worked.
54:21 So if you look at the uh before here,
54:25 the dest was all zeros, the source was threes.
54:28 And then after, the dest is twos and the source
54:32 is still threes because we didn't change the source.
54:34 But what happened was uh PE2 put to PE one put to PE2,
54:40 PE2 put to PE3, PE3 put to PE4, PE4 put to PE0.
54:44 So we did the same kind of ring shifting as we did before.
54:47 Um there is uh another way to uh to do
54:53 this which I want to explain cuz it's really useful.
54:57 It's called a signal.
54:58 So um a signal is a special API in NVSHMEM which does two things at once.
55:05 First, it does um a put.
55:08 So it does some data it updates some data somewhere.
55:11 And then it it does what we call a signal.
55:14 So it it sets um one more special variable uh on the destination.
55:20 But it only does that after the put is guaranteed to be complete.
55:25 So what this kernel that we have above is going
55:27 to do is first do that mem copy that we just did.
55:31 And then instead of having to do a quiet um what we're going to do
55:35 what it's going to do is it's automat
55:37 it's going to inline automatically update a signal.
55:41 And what we can do there is um the PE that is being signaled to can just wait.
55:48 And all it does is locally poll on that single address.
55:52 And by the time that that uh signal is
55:55 updated NVSHMEM promises that the data is all visible.
55:59 And this is really useful because quiet is very
56:01 computationally expensive call that takes a long time to run.
56:04 So if you can rewrite the program using a the data flow using a signal
56:09 you can uh end up using much
56:11 finer grain control and getting uh better performance.
56:14 So the code for that uh looks like this.
56:21 Just wait a second.
56:24 I'll just put it in here.
56:29 Uh it's actually I'm going to put it here where it belongs.
56:34 Um so we're calculating the peer the same way.
56:37 And then we call this NVSHMEM.core.put_signal.
56:40 We pass our dest, our source, and then we also pass our signal which you can see
56:45 is a special array of type uint64 and it's one sized.
56:51 Um we're using set and one.
56:56 So this means that when we're done moving the data from source to dest,
57:00 we're also going to update the signal to be set to one.
57:05 And then the remote PE is peer and the stream is some stream.
57:09 And then on the we're also going to wait for the other PE to update our signal.
57:15 So the wait uh takes the signal pointer basically.
57:21 It's an array.
57:22 Uh it waits for comparison of equal to one on the same stream.
57:28 So what's going to happen is all the PEs are going to first do the put
57:31 signal and then they're going to wait
57:33 for their neighbor to update their stream their signal.
57:37 And let's see if that worked.
57:40 So um before we can for example take PE3 here.
57:44 Uh before the signal was zero, the dest was zero, and the source was fours.
57:49 And after, the signal was updated to one and the dest
57:53 uh was updated to threes because PE3 put into uh PE1.
57:58 And again, both the signal and the wait are completely local operations.
58:04 So um this the put signal is done by PE0 without PE1 knowing about it.
58:11 And the wait is done by PE1 without PE0 knowing about it.
58:16 Um thanks.
58:18 I think I will hand it back to Michael.
58:23 Uh thanks Ben.
58:25 Um that was a really cool presentation.
58:27 Um so I think we're closing to the end of the hour
58:31 and I think it's time to uh wrap up our uh presentation for today.
58:37 Um uh Zach, could you bring up the last slide, please?
58:45 Uh just just the we can just jump over to the summary slide over here.
58:50 Um yeah, so everything you've presented here you've seen here today,
58:55 um you could actually get it from PiPI
58:57 from today using this one line from above.
59:00 Just use pip install um nickel for Pi and NVSHMEM for Pi
59:03 and you can get all of functionality out of the box.
59:07 Um and uh we also provide you with a couple
59:10 links to uh where you can read up the API documentations.
59:14 If you're interested in reading up the code,
59:16 you can also just scan the QR code uh over here and you
59:19 can jump over there and take a look at the code yourself.
59:22 All of them are open sourced.
59:24 Um also, if you run into any any problems while using it,
59:27 please feel free to submit issues um or feedbacks over our GitHub.
59:32 Um and that wraps up for our live stream today.
59:36 Please drop us a star on on GitHub.
59:40 Yeah, absolutely.
59:41 Yeah, the best part about all of this software
59:44 is that you can actually open issues.
59:46 You can communicate with the developers.
59:48 Like please, you know, go check out everything on GitHub.
59:52 Um and I can't thank you enough for all joining us today.
59:56 It's been fantastic.
59:58 Um and just a reminder that this is a monthly live stream.
1:00:01 So we will have more people each month and so please come join
1:00:05 us and you get a chance to ask questions directly to the experts.
1:00:09 So hope to see you next time.
1:00:11 Thanks.
1:00:15 Thanks all.