MACaQ: Model-Agnostic Quantization for Efficient Neural Network Inference

A new paper titled "MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations" has been published on arXiv by researchers Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, and Jiawei Zhuang. The study introduces MACaQ, a novel post-training quantization method designed to improve the efficiency of neural network inference by reducing memory and computational costs while maintaining accuracy.

Key Features of MACaQ

MACaQ is an MXFP4 W4A4 post-training quantization method that operates independently on each linear layer of a neural network. It employs per-channel scaling and block-Hadamard transforms to process activations online, while inverse transforms are absorbed into the weights. This approach eliminates cross-layer dependencies, enabling a simple streaming implementation where only one layer resides in memory at a time.

To enhance quantization accuracy, MACaQ introduces Flip-Refinement, a Hessian-aware coordinate-descent procedure that adjusts FP4 codes to adjacent representable values. Despite the additional activation transforms during inference, MACaQ adds only 2.3% end-to-end overhead compared to pure MXFP4 on the Qwen3-32B model in vLLM.

Performance Benchmarks

MACaQ demonstrates significant performance improvements over existing methods:

• It achieves lower perplexity than DuQuant++ and MR-GPTQ and matches their best perplexity on Llama-3-8B in 11 times less quantization time.
• It successfully quantizes a 122-billion-parameter model in just 64 minutes using 32.5 GB of GPU RAM.
• The method is applicable across various architectures and modalities, including text, image, video, and audio models.

Open-Source Implementation

The MACaQ implementation is available on GitHub at https://github.com/huawei-csl/MACaQ. The repository supports multiple backends, including llama.cpp, vLLM, Diffusers, and PyTorch, allowing users to quantize and run models across different domains. Additionally, MACaQ-R is provided for refining existing GGUF models to improve quality and adapt them to specific workloads while maintaining compatibility with standard llama.cpp.

Implications and Applications

MACaQ's model-agnostic design and ability to operate without architecture-specific modifications make it a versatile tool for optimizing neural network inference. Its efficiency in quantizing large-scale models with minimal resource requirements positions it as a valuable solution for both researchers and practitioners in AI and machine learning.