ThunderKittens Framework Enhanced for NVIDIA Blackwell GPUs

Together AI has announced significant advancements in its ThunderKittens framework, optimizing it for the latest NVIDIA Blackwell GPUs. This development includes the open-sourcing of several high-performance GPU kernels, which are now available for use on Together AI's GPU Clusters, as reported by Together AI.

Collaboration with Stanford Researchers

The ThunderKittens framework, a collaborative effort with researchers at Stanford University, aims to simplify the process of writing efficient GPU kernels. This recent update sees the introduction of new kernels specifically designed for NVIDIA’s Blackwell architecture, leveraging the GPU's advanced capabilities.

New Kernel Features

The update includes BF16 and FP8 ThunderKittens GEMM kernels, which reportedly achieve speeds comparable to or exceeding those of cuBLAS on NVIDIA's H100 GPUs. Furthermore, the attention forward and backward kernels operate at near-cuDNN speeds on the B200, offering up to twice the performance of previous iterations on the H100.

Innovative Use of Hardware Features

ThunderKittens' developers have utilized innovative techniques to harness the full potential of NVIDIA Blackwell GPUs. The framework employs advanced dataflow strategies to maximize the throughput of tensor cores, essential for high-performance computations. The new features, such as fifth-generation tensor cores and tensor memory, play a crucial role in these enhancements.

CTA Pairs and Tensor Memory

One of the standout features in the new architecture is the use of CTA pairs, which allows for deeper coordination between CUDA thread blocks. This coordination is crucial for executing tensor core instructions more efficiently. Additionally, the introduction of tensor memory provides an extra layer of register memory, facilitating the development of complex dataflow pipelines.

Implications for Developers

These advancements are set to benefit developers looking to optimize their applications for NVIDIA’s latest hardware. The open-source nature of the ThunderKittens kernels means developers can experiment with and adapt these tools to suit their specific needs, potentially leading to further innovations in GPU computing.

For more detailed insights into these developments, visit the official announcement by Together AI.