NVIDIA Unveils Grouped GEMM APIs in cuBLAS 12.5 to Boost DL and HPC Performance

The latest release of the NVIDIA cuBLAS library, version 12.5, brings significant updates aimed at enhancing the functionality and performance of deep learning (DL) and high-performance computing (HPC) workloads, according to NVIDIA Technical Blog. Key updates include the introduction of Grouped GEMM APIs, improved matrix multiplication (matmul) performance on NVIDIA Hopper (H100 and H200) and Ada (L40S) GPUs, and enhanced performance tuning options.

Grouped GEMM APIs

The newly introduced Grouped GEMM APIs generalize batched APIs by allowing different matrix sizes, transpositions, and scaling factors to be grouped and executed in one kernel launch. This approach has shown a 1.2x speedup in certain scenarios, such as the generation phase of a mixture-of-experts (MoE) model with batch sizes of 8 and 64 and FP16 inputs and outputs.

Two new sets of APIs support Grouped GEMM:

1. cublas<t>gemmGroupedBatched for FP32 (including TF32) and FP64 precisions.
2. cublasGemmGroupedBatchedEx for FP16, BF16, FP32 (including TF32), and FP64 precisions.

These APIs support variable shapes, transpositions, and scaling factors. Examples can be found on the NVIDIA/CUDALibrarySamples GitHub repository.

Latest LLM Matmul Performance on NVIDIA H100, H200, and L40S GPUs

Recent performance snapshots show significant speedups for Llama 2 70B and GPT3 training phases on NVIDIA H100, H200, and L40S GPUs. The H200 GPU, in particular, demonstrates nearly 3x and 5x speedups compared to the A100 for Llama 2 70B and GPT3 training phases, respectively. These improvements are measured without locking GPU clocks and account for the number of times each GEMM is repeated in the workload.

Library Performance and Benchmarking

Several enhancements have been made to runtime performance heuristics and performance tuning APIs. The cuBLAS library uses a recommender system at runtime to dispatch the fastest available configuration for any user-requested matmuls. This system is trained on actual timing data from a wide range of problems and configurations.

For advanced users, the cublasLtMatmulAlgoGetHeuristic API enables performance tuning to achieve faster implementations. Examples of auto-tuning in cuBLAS can be found on the NVIDIA/CUDALibrarySamples repository.

Better Functionality and Performance in cuBLASLt

Since cuBLAS 12.0, numerous enhancements have been introduced:

1. Fused epilogue support parity between BF16 and FP16 precisions on NVIDIA Ampere and Ada.
2. Additional fused epilogues on NVIDIA Hopper and Ampere.
3. Support for FP8 on Ada GPUs and performance updates on Ada L4, L40, and L40S.
4. Removal of M, N, and batch size limitations of cuBLASLt matmul API.
5. Improved performance of heuristics cache for workloads with high eviction rate.
6. cuBLAS symbols are available in CUDA Toolkit symbols for Linux repository.

For more information on cuBLAS, see the documentation and samples.