Enhancing llama.cpp AI Inference Using CUDA Graphs

The open-source llama.cpp code base, initially released in 2023, has become a popular choice for AI inference on Meta Llama models due to its efficiency and lightweight structure. Built on the GGML library, llama.cpp has gained significant traction among developers, particularly those using C/C++ on personal workstations. According to the NVIDIA Technical Blog, recent advancements have been made by integrating CUDA Graphs into llama.cpp, further enhancing its performance on NVIDIA GPUs.

CUDA Graphs

GPUs have seen rapid advancements, with each new generation offering increased speed. However, individual GPU activities such as kernel execution or memory copying often complete quickly, leading to cumulative overheads when scheduled separately by the CPU. The CUDA Graphs facility addresses this by allowing multiple GPU activities to be scheduled as a single computational graph, reducing overheads significantly.

In previous posts, NVIDIA has detailed the basics of CUDA Graphs and their application in scientific software like GROMACS. Traditionally, each GPU activity in a stream model is scheduled separately, but CUDA Graphs enable unified scheduling, which is relatively straightforward to implement by capturing stream execution into a graph via a few extra CUDA API calls.

Implementing CUDA Graphs in llama.cpp

Before the introduction of CUDA Graphs, llama.cpp faced significant overheads during AI inference due to the separate scheduling of GPU activities. Profiling data showed that these overheads were primarily GPU-side, associated with kernel launches. By integrating CUDA Graphs, these overheads have been substantially reduced.

Overheads in Pre-existing Code

Profiling the pre-existing code revealed gaps between GPU activities, corresponding to CPU-related sampling and preparation tasks. These gaps indicated significant launch overheads, which were addressed by introducing CUDA Graphs.

Introducing CUDA Graphs to Reduce Overheads

In llama.cpp, CUDA Graphs were introduced during the GPU graph evaluation stage. The existing stream was captured into a graph, instantiated into an executable graph, and launched on the GPU to perform token evaluation. Efficient graph reuse was crucial to offset the capture and instantiation overheads. The GGML graph was only recaptured when necessary, with minor adjustments made to the graph to accommodate changes in kernel parameters.

Impact of CUDA Graphs in Reducing Overheads

By submitting all kernels as part of a single computational graph, the overheads between kernels were minimized, leading to significant performance improvements. The highest achieved speedup was 1.2x for the smallest Llama 7B model on the fastest NVIDIA H100 GPUs.

Performance Results

The integration of CUDA Graphs has resulted in notable speedups across various model sizes and GPU variants, with the highest benefits observed for smaller models on faster GPUs. These improvements have been incorporated into the main branch of llama.cpp for batch size 1 inference on NVIDIA GPUs.

Ongoing Work to Reduce CPU Overheads

Further efforts are underway to reduce CPU overheads associated with the preparation of the GGML graph and sampling. These enhancements are expected to offer up to a 10% improvement in performance.

Summary

The introduction of CUDA Graphs to llama.cpp has significantly enhanced AI inference performance on NVIDIA GPUs. Ongoing work promises additional improvements, making llama.cpp an even more efficient framework for AI-enabled workflows.