Accelerating Transformers with NVIDIA cuDNN 9

The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library focused on accelerating deep learning primitives with state-of-the-art performance. According to the NVIDIA Technical Blog, cuDNN 9 introduces significant enhancements aimed at improving the performance of transformers and other complex deep learning models.

Integration with Popular Frameworks

cuDNN is integrated with widely-used deep learning frameworks such as PyTorch, TensorFlow, and XLA (Accelerated Linear Algebra). These frameworks abstract the complexities of direct GPU programming, allowing developers to concentrate on designing and training their models. cuDNN acts as the performance engine, ensuring that operations are executed with maximum efficiency.

Scaled Dot Product Attention Performance

Scaled dot product attention (SDPA) has become a performance-critical primitive in important workloads like large language models (LLMs). cuDNN 9 has added support for this primitive, enhancing its performance through flash attention and other optimizations. For example, on the NVIDIA H200 Tensor Core GPU, cuDNN can achieve up to 1.2 PFLOPS in FP8. A speedup of 1.15x was observed in an experiment using cuDNN FP8 SDPA for Llama2 70B LoRA fine-tuning, conducted on an 8-GPU H200 node.

SDPA as cuDNN Graphs

SDPA in cuDNN can be specified as a cuDNN graph of tensor operations. For any given graph, the cuDNN library has some set of engines that can execute it. While some graphs may not have any suitable engines, the intent is to provide at least one engine for any cuDNN graph that can be executed atomically on the GPU. This section explains the flexibility and support available today.

SDPA Usage Walkthrough

There are several API entry points available for creating and running cuDNN graphs:

• Frontend API (with both C++ and Python variants)
• Backend API (C only)

The walkthrough begins with the simplest case, an SDPA node created in Python. The SDPA Python example demonstrates the configuration options and the basic usage flow.

Other Notable cuDNN 9 Features

In addition to the SDPA improvements, cuDNN 9 introduces several other important features:

• Mixed input precision support for matmuls and convolution
• Improved error reporting
• Hardware forward compatibility
• Streamlined installation

Mixed Input Precision Support

cuDNN now supports mixed input precision matmuls and convolutions, where A and B operands can be different data types, with online fused type conversion for performance and memory optimization. This is particularly useful for scenarios like Activation-aware Weight Quantization (AWQ).

Improved Error Reporting

cuDNN 9 enhances error reporting with more specific error codes and nested logging levels, which align with logging conventions. The new function cuDNNGetLastErrorString allows developers to get the last error message programmatically, simplifying debugging.

Hardware Forward Compatibility

cuDNN 9 introduces hardware forward compatibility for a large subset of the API. This means that programs using this subset will be functional on future GPUs, and users won’t need to upgrade their cuDNN installation to use new GPU architectures.

Streamlined Installation

The installation process has been streamlined for RPM and Debian meta-packages. For Python environments, you can use pip to install the new Python frontend along with the library.

Next Steps

If you have feedback, questions, or comments, you can post on the cuDNN NVIDIA Developer forum. To get started, download cuDNN today.

As AI continues to drive the industry to the limits of hardware and software integration, NVIDIA is committed to optimizing performance and improving user experience, ensuring that cuDNN can be used more effectively and broadly across deep learning frameworks and graph compilers.

Acknowledgments

The NVIDIA cuDNN team contributed the technical content for this post, in collaboration with many other teams across the company.