Hybrid Gated Attention

2026-08-12

AI Summary

This work addresses the attention sink problem in attention mechanisms and the inherent trade-off between representational capacity and computational efficiency by proposing a Hybrid Gated Attention (HyGA) framework. HyGA introduces, for the first time, a multi-view, multi-stage hybrid gating mechanism that integrates learnable attention sinks, low-rank matrix decomposition, and element-wise and head-wise collaborative modulation to enable fine-grained control over information flow. The proposed method consistently outperforms existing gated attention approaches across diverse backbone architectures and benchmark tasks, achieving state-of-the-art performance under varying computational budgets. Furthermore, HyGA effectively reduces training loss while simultaneously enhancing model stability and expressive power.

Abstract

Gated attention is an effective approach to mitigate attention sinks and enhance the representational capacity of attention. To further extend its effectiveness-efficiency Pareto frontier, we propose a Hybrid Gated Attention (HyGA) framework that contains three types of gating strategies. Specifically, these gates leverage diverse information from multiple stages of attention, and collaboratively build element-wise/head-wise gating from multiple perspectives, capturing intra-head and cross-head information interactions. Through our hybrid gating components, HyGA could provide multi-source modulation signals, enabling more comprehensive control over information flow and improving the representational capacity of attention. We also introduce low-rank matrix decomposition and learnable attention sink to further enhance training efficiency and stability. In experiments, we evaluate HyGA on widely-used benchmarks based on different backbones. The experimental results show that our HyGA comprehensively improves both training loss and various downstream performances compared with Gated attention. HyGA has also been verified to achieve the best performance at different computation costs, with comprehensive model analyses for better understanding. The proposed HyGA sheds light on a more effective, efficient, and stable attention mechanism.

Problem

• attention sinks
• representational capacity
• attention mechanism
• information flow
• training efficiency

Innovation

• Hybrid Gated Attention
• attention gating
• low-rank decomposition
• learnable attention sink
• multi-source modulation