Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few

2025-09-20

AI Summary

Transformer self-attention achieves strong performance but suffers from an ill-defined optimization objective and quadratic computational complexity, O(n²), hindering both interpretability and efficiency. To address this, we propose Contract-and-Broadcast Self-Attention (CBSA), the first unified framework that jointly enhances interpretability and efficiency. CBSA employs optimization-driven token contraction and information broadcasting to compress attention into low-dimensional structured representations, achieving linear complexity, O(n). Theoretically, CBSA subsumes multiple mainstream attention variants as special cases, demonstrating strong generalizability. Extensive experiments across diverse vision tasks show that CBSA matches or surpasses baseline performance while incurring significantly lower computational overhead—validating its effectiveness, interpretability, and practical utility.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Satisfaction

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Attention mechanisms in Transformers have gained significant empirical success. Nonetheless, the optimization objectives underlying their forward pass are still unclear. Additionally, the quadratic complexity of self-attention is increasingly prohibitive. Unlike the prior work on addressing the interpretability or efficiency issue separately, we propose a unified optimization objective to alleviate both issues simultaneously. By unrolling the optimization over the objective, we derive an inherently interpretable and efficient attention mechanism, which compresses all tokens into low-dimensional structures by contracting a few representative tokens and then broadcasting the contractions back. This Contract-and-Broadcast Self-Attention (CBSA) mechanism can not only scale linearly but also generalize existing attention mechanisms as its special cases. Experiments further demonstrate comparable performance and even superior advantages of CBSA on several visual tasks. Code is available at this https URL.

Problem

• Address unclear optimization objectives in Transformer attention mechanisms
• Solve quadratic complexity issue of self-attention for better efficiency
• Provide unified approach for interpretability and efficiency simultaneously

Innovation

• Compresses tokens into low-dimensional structures
• Contracts and broadcasts representative tokens
• Achieves linear complexity and generalizes existing attention