AMD Collaborates with Microsoft to Boost Performance and Reduce Memory Footprint on AI Workloads

AMD and Microsoft Enhance AI Workload Performance

AMD, in close partnership with Microsoft, has introduced 4-bit quantization support and acceleration for Large Language Models (LLMs) to integrated and discrete AMD Radeon GPU platforms running with ONNXRuntime->DirectML. This advancement, as reported by AMD, is geared towards boosting the performance of AI workloads and reducing their memory footprint.

Addressing Memory Bottlenecks in LLMs

Large Language Models are often bottlenecked by memory bandwidth and availability. The memory consumption tends to escalate significantly depending on the number of parameters used by the LLM. This issue often makes some systems unsuitable for running such workloads. The introduction of 4-bit quantization for LLM parameters aims to make a broader set of integrated and discrete GPUs accessible for such LLM workloads, thus significantly reducing memory usage while boosting performance.

Introducing Activation-Aware Quantization

With the latest DirectML and AMD driver preview release, Microsoft and AMD have introduced Activation-Aware Quantization (AWQ) based LM acceleration on AMD GPU platforms. The AWQ technique compresses weights to 4-bit wherever possible with minimal impact on accuracy, thereby reducing the memory footprint of running these LLM models significantly while increasing performance simultaneously. This technique identifies the top 1% of salient weights that are crucial for maintaining model accuracy and quantizes the remaining 99% of weight parameters.

Improved Performance

The transition to the 4-bit quantization technique for LM parameters not only improves memory utilization but also enhances performance by significantly reducing bandwidth. Performance testing conducted by AMD on a test system configured with a Ryzen 9 7950X CPU and Radeon RX 7900 XTX GPU demonstrated memory footprint reduction compared to running 16-bit variant of the weights.

Running Sample Application of LLM using 4-bit quantized models

AMD provided a sample application of an LLM using 4-bit quantized models, which can be run using the ONNXRuntime GenAI backend. This technique now makes it possible to run these language models (LM) on device on low memory equipped system which was not possible before.