RDMA for Distributed Storage (Ceph)

High-Performance Networking & Cloud-Native · 30 illustrated lessons (text/HTML)

Outcomes

• Build and configure an RDMA-enabled Ceph cluster
• Benchmark RDMA against TCP for small and large I/O
• Tune memory, hugepages and BlueStore for RDMA

Where we teach this

• STO-110 · NVMe, NVMe-oF & Storage Acceleration live cohort

The lessons

Every lesson in this curriculum, in the order we teach it.

• 1Introduction: RDMA in Ceph distributed storage
• 2RDMA fundamentals
• 3The RDMA protocol stack explained
• 4Ceph architecture in depth
• 5Analysis of Ceph's traditional network bottlenecks
• 6Why combine RDMA and Ceph
• 7The Ceph MSGR v2 protocol
• 8RDMA-based Ceph OSD communication optimisation
• 9RDMA-based Ceph client communication
• 10Ceph BlueStore and RDMA: architecture and acceleration
• 11Deployment planning: RDMA in Ceph
• 12Setting up an RDMA environment on Linux
• 13Building Ceph from source with RDMA support
• 14Ceph configuration in detail: RDMA networking and CRUSH map tuning
• 15Deploying an RDMA-enabled Ceph cluster in practice
• 16RDMA in Ceph
• 17RDMA in Ceph: performance-testing methodology
• 18Latency comparison: RDMA vs TCP/IP for small I/O
• 19Throughput comparison: RDMA vs TCP/IP bandwidth for large sequential I/O
• 20CPU utilisation comparison: verifying the benefit of RDMA kernel bypass
• 21RDMA in Ceph: advanced tuning parameters
• 22Network-layer tuning
• 23Memory and cache tuning: locked memory, hugepages, Ceph caching and RDMA
• 24RDMA in Ceph: troubleshooting basics
• 25Monitoring and log analysis: RDMA in Ceph
• 26Working with DPDK/SPDK: user-space I/O stacks combined with RDMA, and architecture options
• 27RDMA Ceph in containers: Kubernetes deployment and device management
• 28Public cloud practice: deploying Ceph on RDMA-capable cloud instances
• 29RDMA in Ceph: emerging technologies and outlook
• 30RDMA in Ceph: summary and best practices