RDM-05 · RDMA and HPC Interconnect
Move GPU data across the cluster without a CPU copy.
Build a GPU-to-GPU path that skips the host, then keep it fast when several tenants share the machine.
You build: a GPUDirect path with a multi-tenant performance test.
Promise
Move GPU data across the cluster without a CPU copy, and keep the path fast in a multi-tenant system.
Prerequisites
RDM-01 or RDM-03; basic GPU concepts.
Units
1. Overview ch 01–02
GPUDirect at a glance.
2. The GPUDirect stack ch 03–06, 11
GPUDirect Storage, GPUDirect RDMA, SR-IOV on the GPU, the stack.
3. Networks ch 07–10
InfiniBand, RoCE, coordinating GPU and RDMA traffic, virtualization.
4. Software ch 12–14
MPI with GPUDirect, the NCCL library, CUDA-aware MPI.
5. Evaluation ch 15–17
system software stack, method, micro-benchmarks.
6. Case studies ch 18–21
molecular dynamics, CFD, AI training, climate models.
7. Multi-tenant and cloud ch 22–24
SR-IOV in shared systems, cloud-native supercomputing, troubleshooting.
8. System configuration ch 25–28
building and binding virtual functions, programming models and APIs.
9. Outlook ch 29–30
.
Labs
Each lab names its environment and a pass check. Labs are not marked complete on "it ran" — the pass check has to hold.
| Lab | What you do | Environment |
|---|---|---|
| RDM-05ach 13 | NCCL all-reduce testPass check: Bus bandwidth recorded | hardware60 min |
| RDM-05bch 17 | Micro-benchmark for GPU and NIC pathsPass check: Table of message sizes | hardware90 min |
| RDM-05cch 27 | Create and assign SR-IOV virtual functionsPass check: `lspci` shows the VFs | hardware60 min |
| RDM-05dch 14 | CUDA-aware MPI send and receivePass check: Device buffers pass correctly | hardware90 min |
Capstone project · CAP-16
GPU Communication Optimization Study · 3 weeks
A training job spends too long in all-reduce.
Profile the communication, change one thing at a time, and measure each change.
Deliverables: Profiles, before-and-after tables, report.
Outcomes
- I can explain why GPUDirect RDMA skips the host copy.
- I can run an NCCL all-reduce test and read its bandwidth.
- I can create and assign SR-IOV virtual functions.
Also in
How it runs
Work through the units in order. Each unit groups the chapters of the study guide with its share of the labs. You can start any time; there is no cohort date and nothing expires.
Labs run in a VM, a container, a simulator or on real hardware, and the card above names which. Where a lab needs hardware — an RDMA NIC, a programmable switch model, an FPGA — there is a simulator fallback so you are never blocked by equipment.
Questions
Do I get a certificate?
You get a completion record for the course. Path badges and project credentials are earned by passing the labs and the project review, not by attendance.
Can my team take this together?
Yes — and if you would rather have it delivered live, with the instructor in the room, this course also exists as a scheduled live cohort. See the live dates.
How long does it take?
≈ 30 h of study including the labs. Most engineers take it part-time over a few weeks.