NVIDIA's TensorRT-LLM MultiShot Enhances AllReduce Performance with NVSwitch

NVIDIA has unveiled TensorRT-LLM MultiShot, a new protocol designed to enhance the efficiency of multi-GPU communication, particularly for generative AI workloads in production environments. According to NVIDIA, this innovation leverages the NVLink Switch technology to significantly boost communication speeds by up to three times.

Challenges with Traditional AllReduce

In AI applications, low latency inference is crucial, and multi-GPU setups are often necessary. However, traditional AllReduce algorithms, which are essential for synchronizing GPU computations, can become inefficient as they involve multiple data exchange steps. The conventional ring-based approach requires 2N-2 steps, where N is the number of GPUs, leading to increased latency and synchronization challenges.

TensorRT-LLM MultiShot Solution

TensorRT-LLM MultiShot addresses these challenges by reducing the latency of the AllReduce operation. It utilizes NVSwitch's multicast feature, allowing a GPU to send data simultaneously to all other GPUs with minimal communication steps. This results in only two synchronization steps, irrespective of the number of GPUs involved, vastly improving efficiency.

The process is divided into a ReduceScatter operation followed by an AllGather operation. Each GPU accumulates a portion of the result tensor and then broadcasts the accumulated results to all other GPUs. This method reduces the bandwidth per GPU and improves the overall throughput.

Implications for AI Performance

The introduction of TensorRT-LLM MultiShot could lead to nearly threefold improvements in speed over traditional methods, particularly beneficial in scenarios requiring low latency and high parallelism. This advancement allows for reduced latency or increased throughput at a given latency, potentially enabling super-linear scaling with more GPUs.

NVIDIA emphasizes the importance of understanding workload bottlenecks to optimize performance. The company continues to work closely with developers and researchers to implement new optimizations, aiming to enhance the platform's performance continually.