ThunderKittens: Simplifying AI Kernel Development with an Embedded DSL

ThunderKittens, a newly introduced embedded Domain Specific Language (DSL) for AI kernel development, aims to simplify the complexities of creating AI kernels. Developed by together.ai, ThunderKittens is designed to make it easier to express key technical ideas and improve code clarity, according to together.ai.

Addressing the Challenges of AI Kernel Development

The development of AI kernels often involves intricate processes that can be daunting for many. FlashAttention, another open-source AI project, has been regarded as complex and almost alien technology by some. To bridge the gap between complex concepts and practical application, the team at together.ai presented their ideas at a NeurIPS keynote, yet found that the key technical ideas remained difficult for many to grasp.

In response, together.ai embarked on creating a simple framework named ThunderKittens. This small library or DSL aims to enable developers to write clean, simple-to-understand code that effectively utilizes GPU capabilities. The fundamental object in ThunderKittens is a matrix (tile) that fits into the tensor core, ensuring high utilization and performance optimization.

Key Features and Performance

ThunderKittens is designed with several key observations in mind:

• Maximizing Tensor Core Utilization: Tensor cores constitute 94% of the compute power on an H100 GPU. ThunderKittens ensures these cores are kept busy by making the matrix tile the fundamental object.
• Familiarity for AI Developers: The API is designed to be similar to PyTorch, making it accessible for AI professionals familiar with this framework.
• Transparency and Performance: ThunderKittens does not hide the workings of accelerators, allowing AI developers to fully leverage the power of CUDA or HIP. The DSL aims to strike a balance between being embedded in raw CUDA and focusing on AI kernels, similar to NVIDIA’s approach.

Performance-wise, ThunderKittens has shown promising results. On GPUs like the 4090s and A100s, ThunderKittens matches the performance of FlashAttention 2 (FA2) with just a few lines of code. On H100s, it even surpasses FA2 in both forward and backward operations, offering significant speed improvements without sacrificing code simplicity.

Community and Future Prospects

The developers at together.ai have integrated ThunderKittens with several other projects, including a version of the popular NanoGPT project by Andrej Karpathy. This integration aims to make key ideas clearer and more accessible to the AI community.

While ThunderKittens is currently regarded as an art project by its creators, it offers a fun and useful tool for those interested in AI kernel development. The team encourages feedback, although they note that they may not address all complaints or issues raised by users. The focus remains on creating a simple, effective tool for their own use and for the broader AI community to experiment with and learn from.

For more detailed information, visit the original together.ai post.