RDM-04 · RDMA and HPC Interconnect
Storage that stops waiting on TCP.
Trace where a storage cluster loses time, switch on RDMA in Ceph, and prove the gain with numbers you can defend.
You build: a before-and-after Ceph study with latency and throughput data.
In final review — the chapters for this course are still being finalised.
Promise
Find where a storage cluster loses time to TCP, enable RDMA in Ceph, and prove the gain with measurements.
Prerequisites
RDM-01; basic storage concepts.
Units
1. RDMA foundations ch 01–03
RDMA basics, the protocol stack.
2. Ceph and its bottlenecks ch 04–06
Ceph architecture, where TCP costs time, why RDMA helps.
3. Messenger and daemons ch 07–10
the MSGR v2 protocol, OSD and client communication, BlueStore.
4. Build and deploy ch 11–15
RDMA environment, building Ceph with RDMA, cluster configuration, deployment.
5. Benchmarks ch 16–20
latency, large-block throughput, CPU utilization comparisons.
6. Tuning ch 21–25
memory and cache tuning, monitoring and logs.
7. Ecosystem ch 26–30
DPDK and SPDK, containers, public cloud, summary.
Labs
Each lab names its environment and a pass check. Labs are not marked complete on "it ran" — the pass check has to hold.
| Lab | What you do | Environment |
|---|---|---|
| RDM-04ach 13 | Build Ceph with RDMAPass check: `ceph` shows the RDMA messenger | sim180 min |
| RDM-04bch 15 | Deploy a three-node test clusterPass check: Health OK | sim120 min |
| RDM-04cch 18 | Small-block latency: RDMA vs TCPPass check: Both curves in one chart | sim90 min |
| RDM-04dch 20 | CPU use during the same testPass check: Per-core usage table | sim60 min |
Capstone project · CAP-15
RDMA vs TCP Storage Benchmark · 2 weeks
A storage team asks whether RDMA is worth the operational cost.
Benchmark small-block latency, large-block throughput, and CPU per gigabyte, with and without RDMA. Recommend.
Deliverables: Data, charts, recommendation with operational cost.
Outcomes
- I can build Ceph with RDMA support and confirm it is in use.
- I can design a fair RDMA-vs-TCP benchmark.
- I can tune locked memory and huge pages for RDMA.
Also in
How it runs
Work through the units in order. Each unit groups the chapters of the study guide with its share of the labs. You can start any time; there is no cohort date and nothing expires.
Labs run in a VM, a container, a simulator or on real hardware, and the card above names which. Where a lab needs hardware — an RDMA NIC, a programmable switch model, an FPGA — there is a simulator fallback so you are never blocked by equipment.
Questions
Do I get a certificate?
You get a completion record for the course. Path badges and project credentials are earned by passing the labs and the project review, not by attendance.
Can my team take this together?
Yes — and if you would rather have it delivered live, with the instructor in the room, this course also exists as a scheduled live cohort. See the live dates.
How long does it take?
≈ 30 h of study including the labs. Most engineers take it part-time over a few weeks.