NVIDIA Enhances LLM Inference with ReDrafter Support in TensorRT-LLM

NVIDIA Enhances LLM Inference with ReDrafter Support in TensorRT-LLM

Caroline Bishop Dec 18, 2024 17:01 0 Min Read

NVIDIA has integrated Apple's ReDrafter, an innovative speculative decoding technique, into its TensorRT-LLM library to enhance large language model (LLM) inference. This collaboration aims to optimize the performance of LLM workloads on NVIDIA GPUs, according to NVIDIA's official blog.

Understanding ReDrafter

ReDrafter, developed and open-sourced by Apple, employs recurrent neural network (RNN)-based sampling, or 'drafting', combined with tree-style attention. This speculative decoding technique accelerates LLM inference by generating multiple tokens in parallel, maintaining output quality while reducing response times. The method is especially effective during low traffic periods, utilizing resources for low-latency inference.

Technical Integration

ReDrafter's integration into TensorRT-LLM facilitates drafting and validation within a single engine, minimizing runtime overhead. This update allows TensorRT-LLM greater flexibility in kernel selection and scheduling, optimizing network performance. The ReDrafter implementation in TensorRT-LLM simplifies runtime changes and enhances speculative decoding efficiency compared to previous methods like Medusa.

Inflight-Batching and In-Engine Validation

TensorRT-LLM supports inflight-batching, which improves throughput by batching context-phase and generation-phase requests. ReDrafter's validation logic is incorporated into the model definition, necessitating similar logic during validation. This approach enhances TensorRT-LLM's flexibility, enabling more complex model definitions in the future.

Performance and Potential

According to Apple benchmarks, ReDrafter in TensorRT-LLM can achieve up to 2.7x throughput improvements on NVIDIA H100 GPUs. Factors influencing performance include GPU utilization, average acceptance rate, and task specificity. Speculative decoding is particularly beneficial for low-traffic scenarios and tasks like code completion, which predict future tokens more accurately.

Future Implications

This integration marks a significant advancement in LLM inference optimization. By enhancing TensorRT-LLM's capabilities, NVIDIA and Apple's collaboration opens new possibilities for developing sophisticated models. The LLM community can now leverage these advancements to deploy models with unparalleled performance on NVIDIA GPUs.