CUDA Core Programming & Parallel Optimisation

Heterogeneous Computing & Edge AI · 40 illustrated lessons (text/HTML)

Outcomes

• Write CUDA kernels and manage the memory hierarchy
• Optimise reductions, scans, matrix multiply and sorting
• Use streams, multi-GPU, Tensor Cores and profiling tools

Where we teach this

• AIC-220 · CUDA & HIP Programming live cohort

The lessons

Every lesson in this curriculum, in the order we teach it.

• 1Introduction to parallel computing with CUDA
• 2Setting up a CUDA development environment
• 3CUDA programming model basics
• 4Your first CUDA program in practice
• 5The CUDA thread hierarchy
• 6The CUDA memory hierarchy in depth
• 7Shared-memory optimisation in CUDA
• 8Constant and texture memory in CUDA
• 9CUDA streams and concurrent execution
• 10CUDA events and timing
• 11Atomic operations and synchronisation in CUDA
• 12Warp-level programming optimisation
• 13Optimising matrix multiplication in practice
• 14Optimising reduction algorithms
• 15Scan algorithms in CUDA in practice
• 16Sorting algorithms in CUDA in practice
• 17Image processing optimisation with CUDA in practice
• 18Deep-learning inference acceleration
• 19Dynamic parallelism in CUDA
• 20Multi-GPU programming
• 21CUDA and OpenMP integration
• 22CUDA and MPI integration
• 23CUDA profiling tools
• 24Optimising CUDA memory-access patterns
• 25Instruction-level optimisation in CUDA
• 26Tensor Core programming
• 27Advanced CUDA C++ features
• 28The CUDA graph execution model
• 29Virtual memory management in CUDA
• 30The streaming multiprocessor architecture
• 31Optimising the CUDA cache hierarchy
• 32Asynchronous operations and overlap in CUDA
• 33Controlling numerical precision
• 34Optimising random-number generation in CUDA
• 35Graph algorithms accelerated with CUDA in practice
• 36Parallelising physics simulations
• 37Accelerating financial computing
• 38CUDA in bioinformatics
• 39CUDA best practices summary
• 40The future of large chips