MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspective

2025-07-25

AI Summary

To address performance degradation caused by low-bit quantization in Vision Transformers, this paper proposes MixA-Q—the first framework integrating window-level activation sparsity with mixed-precision quantization. Its core innovation lies in a dual-branch Swin module that enables intra-layer window importance estimation and dynamic bit-width allocation (e.g., higher bit-width for critical windows, lower bit-width for sparse ones), supporting both Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ). MixA-Q significantly suppresses quantization error: under PTQ on COCO, it achieves 1.35× speedup without accuracy loss; under QAT, it delivers either 1.25× speedup with no mAP degradation or 1.53× speedup with only a 1% mAP drop. For the W4A4 model, it improves mAP by 0.7% and reduces quantization-induced degradation by 24%.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Large pretrained models with web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In this paper, we propose MixA-Q, a mixed-precision activation quantization framework that leverages intra-layer activation sparsity (a concept widely explored in activation pruning methods) for efficient inference of quantized window-based vision transformers. For a given uniform-bit quantization configuration, MixA-Q separates the batched window computations within Swin blocks and assigns a lower bit width to the activations of less important windows, improving the trade-off between model performance and efficiency. We introduce a Two-Branch Swin Block that processes activations separately in high- and low-bit precision, enabling seamless integration of our method with most quantization-aware training (QAT) and post-training quantization (PTQ) methods, or with simple modifications. Our experimental evaluations over the COCO dataset demonstrate that MixA-Q achieves a training-free 1.35x computational speedup without accuracy loss in PTQ configuration. With QAT, MixA-Q achieves a lossless 1.25x speedup and a 1.53x speedup with only a 1% mAP drop by incorporating activation pruning. Notably, by reducing the quantization error in important regions, our sparsity-aware quantization adaptation improves the mAP of the quantized W4A4 model (with both weights and activations in 4-bit precision) by 0.7%, reducing quantization degradation by 24%.

Problem

• Leveraging activation sparsity for efficient vision transformer inference
• Improving performance-efficiency trade-off via mixed-precision quantization
• Reducing quantization error in critical regions for accuracy preservation

Innovation

• Mixed-precision activation quantization framework
• Two-Branch Swin Block processing
• Sparsity-aware quantization adaptation