Cost-Effective Attention Mechanisms for Low Resource Settings: Necessity&Sufficiency of Linear Transformations

2024-03-03

AI Summary

In low-resource settings, Scaled Dot-Product Attention (SDPA) incurs excessive computational and memory overhead due to redundant linear transformations. Method: This paper systematically analyzes the necessity and redundancy of SDPA’s linear projections and proposes three lightweight variants, with Super Attention as the core innovation—employing structured pruning and redesigned linear layers to eliminate parameter inefficiency. Contribution/Results: Super Attention reduces model parameters by 25%, slashes memory consumption and FLOPs significantly, and surpasses standard SDPA by 10% in accuracy. It maintains near-lossless performance across diverse NLP and vision tasks, achieving overall parameter compression of 25–50%. The method delivers simultaneous improvements in accuracy, inference speed, and model compactness. To rigorously validate generalization, the paper introduces the first cross-modal unified evaluation framework, demonstrating consistent efficacy across modalities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

From natural language processing to vision, Scaled Dot Product Attention (SDPA) is the backbone of most modern deep learning applications. Unfortunately, its memory and computational requirements can be prohibitive in low-resource settings. In this paper, we improve its efficiency without sacrificing its versatility. We propose three attention variants where we remove consecutive linear transformations or add a novel one, and evaluate them on a range of standard NLP and vision tasks. Our proposed models are substantially lighter than standard SDPA (and have 25-50% fewer parameters). We show that the performance cost of these changes is negligible relative to size reduction and that in one case (Super Attention) we succeed in outperforming SDPA by up to 10% while improving its speed and reducing its parameters by 25%.

Problem

• Optimize attention mechanisms for low-resource settings
• Reduce memory and computational requirements of SDPA
• Maintain performance while decreasing model parameters

Innovation

• Linear transformations optimization
• Reduced parameter attention variants
• Enhanced speed and efficiency