GPUDirect & SR-IOV for Supercomputing
Remove the CPU from the data path: GPUDirect RDMA and Storage, SR-IOV for multi-tenant HPC, NCCL and CUDA-aware MPI.
Advanced30 lessons
written lessons≈ 15 h
High-Performance Networking & Cloud-Native · 30 illustrated lessons (text/HTML)
Outcomes
- Configure GPUDirect RDMA and GPUDirect Storage
- Use SR-IOV virtual functions in multi-tenant HPC and containers
- Diagnose communication bottlenecks in GPU clusters
Where we teach this
- AIC-210 · RDMA & AI Cluster Networking live cohort
- VRT-110 · VFIO Device Passthrough & virtio Drivers live cohort
The lessons
Every lesson in this curriculum, in the order we teach it.
- 1Supercomputing trends
- 2HPC course
- 3GPUDirect Storage: overview and core principles
- 4Principles and applications
- 5SR-IOV technology in depth
- 6SR-IOV on GPUs
- 7InfiniBand networking basics: architecture, protocol stack and dominance in supercomputing
- 8RoCE networks: principles, comparison and applications
- 9Core concept: why GDR is needed
- 10SR-IOV and network virtualisation
- 11The NVIDIA GPUDirect stack explained
- 12MPI and GPUDirect: RDMA support and integration
- 13The NCCL library in depth: collective primitives and optimisation
- 14CUDA-aware MPI: concepts, implementation and use
- 15Supercomputer software stack: job scheduling, containers and GPUDirect integration
- 16Performance evaluation methodology: GPUDirect and SR-IOV in supercomputing
- 17Micro-benchmarking: profiling HPC network performance
- 18Case study 1: GPUDirect acceleration of molecular dynamics simulation
- 19Case study 2: GPU communication optimisation in computational fluid dynamics (OpenFOAM)
- 20Case study 3: large-scale GPU communication in AI training
- 21Case study 4: GPU-accelerated communication patterns in weather and climate models (WRF)
- 22SR-IOV in multi-tenant HPC: resource isolation, QoS and security
- 23Cloud-native HPC with SR-IOV: Kubernetes device plugins, Kata Containers and GPU virtualisation
- 24Troubleshooting and debugging: GPUDirect and SR-IOV high-performance applications
- 25System configuration in practice, part 1: Linux kernel, drivers and firmware
- 26System configuration in practice, part 2: InfiniBand/RoCE subnet manager, partitions and routing
- 27Creating and binding SR-IOV VFs, and passing them through to VMs and containers
- 28Programming models and APIs: CUDA IPC, NVSHMEM and UCX compared
- 29Outlook: NVLink Switch System, Quantum-2 InfiniBand and how DPUs/IPUs change architectures
- 30Course summary and best practices
All-access library
Every curriculum on this page, with the labs and lab sequences, for a year.