RDM-03 · RDMA and HPC Interconnect
Find out where your MPI job really spends its time.
Follow MPI from the application to the wire through OFI and UCX, then tune a real job until it earns the speed its hardware promised.
You build: a tuned MPI application with before-and-after measurements.
Promise
Explain how MPI reaches the wire through OFI and UCX, and tune a real MPI job.
Prerequisites
RDM-01; MPI basics.
Units
1. HPC and RDMA basics ch 01–03
HPC fundamentals, RDMA principles.
2. InfiniBand and the MPI standard ch 04–07
the architecture, the standard, and the implementations.
3. MPI over RDMA internals ch 08–12
integration architecture, OpenMPI's OOB and BTL, MPICH's CH4 and Netmod, OFI, UCX.
4. Communication primitives ch 13–16
RDMA primitives mapped to MPI, point-to-point and collective optimization.
5. Registration, completions, overlap ch 17–21
memory registration, completion queues, overlapping computation and communication.
6. Tuning and launch ch 22–25
runtime parameters, hardware tuning, large-scale launch, fault tolerance.
7. Accelerators and advanced modes ch 26–27
heterogeneous computing, persistent and partitioned communication.
8. Case studies ch 28–29
OpenMPI with UCX; MPICH with OFI.
9. Outlook ch 30
smart NICs and programmable switches for MPI.
Labs
Each lab names its environment and a pass check. Labs are not marked complete on "it ran" — the pass check has to hold.
| Lab | What you do | Environment |
|---|---|---|
| RDM-03ach 28 | OSU latency and bandwidth with OpenMPI and UCXPass check: Both results recorded | container90 min |
| RDM-03bch 11 | Use OFI to list providers and run a testPass check: Chosen provider reported | sim60 min |
| RDM-03cch 16 | MPI_Put example on two ranksPass check: Data arrives after fence | sim90 min |
| RDM-03dch 22 | Change one MCA parameter and measurePass check: Before and after compared | sim60 min |
Capstone project · CAP-03
MPI Application Benchmark and Tuning · 3 weeks
A research team's MPI job runs slower on the RoCEv2 fabric than on the old InfiniBand cluster.
Benchmark the application's communication, find the dominant pattern, tune transport and collective choices, and report the gain with its uncertainty.
Deliverables: Benchmark scripts, before-and-after table, a recommendation with risks.
Outcomes
- I can explain which transport MPI selects and why.
- I can run the OSU benchmarks and read the results.
- I can set OpenMPI MCA parameters and show their effect.
- I can explain what an RMA put does on RDMA hardware.
Also in
How it runs
Work through the units in order. Each unit groups the chapters of the study guide with its share of the labs. You can start any time; there is no cohort date and nothing expires.
Labs run in a VM, a container, a simulator or on real hardware, and the card above names which. Where a lab needs hardware — an RDMA NIC, a programmable switch model, an FPGA — there is a simulator fallback so you are never blocked by equipment.
Questions
Do I get a certificate?
You get a completion record for the course. Path badges and project credentials are earned by passing the labs and the project review, not by attendance.
Can my team take this together?
Yes — and if you would rather have it delivered live, with the instructor in the room, this course also exists as a scheduled live cohort. See the live dates.
How long does it take?
≈ 30 h of study including the labs. Most engineers take it part-time over a few weeks.