BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers

2026-03-10

AI Summary

This work addresses the high computational complexity of attention mechanisms in Vision Transformers and Diffusion Transformers, which poses a fundamental trade-off between efficiency and accuracy. The authors propose BinaryAttention, the first method to theoretically guarantee that binarizing attention preserves essential similarity relationships. By retaining only the sign bits of queries and keys, attention computation is reformulated as highly efficient bitwise operations. Integrated with learnable biases, quantization-aware training, and self-distillation, the approach enables end-to-end 1-bit query-key attention. Experiments demonstrate that BinaryAttention matches or exceeds full-precision performance on both Vision Transformer and Diffusion Transformer architectures, while achieving more than a 2× speedup over FlashAttention-2 during inference on an A100 GPU.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Hardware-aware ML
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

Transformers have achieved widespread and remarkable success, while the computational complexity of their attention modules remains a major bottleneck for vision tasks. Existing methods mainly employ 8-bit or 4-bit quantization to balance efficiency and accuracy. In this paper, with theoretical justification, we indicate that binarization of attention preserves the essential similarity relationships, and propose BinaryAttention, an effective method for fast and accurate 1-bit qk-attention. Specifically, we retain only the sign of queries and keys in computing the attention, and replace the floating dot products with bit-wise operations, significantly reducing the computational cost. We mitigate the inherent information loss under 1-bit quantization by incorporating a learnable bias, and enable end-to-end acceleration. To maintain the accuracy of attention, we adopt quantization-aware training and self-distillation techniques, mitigating quantization errors while ensuring sign-aligned similarity. BinaryAttention is more than 2x faster than FlashAttention2 on A100 GPUs. Extensive experiments on vision transformer and diffusion transformer benchmarks demonstrate that BinaryAttention matches or even exceeds full-precision attention, validating its effectiveness. Our work provides a highly efficient and effective alternative to full-precision attention, pushing the frontier of low-bit vision and diffusion transformers. The codes and models can be found at https://github.com/EdwardChasel/BinaryAttention.

Problem

• attention
• quantization
• vision transformers
• diffusion transformers
• computational efficiency

Innovation

• BinaryAttention
• 1-bit quantization
• QK-attention
• bit-wise operations
• quantization-aware training