Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers

2025-10-10

AI Summary

Large Transformer models are vulnerable to extreme token phenomena—such as attention sinks and value-state drain—where attention is erroneously concentrated on low-information tokens, degrading performance, distorting quantization, and impairing interpretability. To address this, we propose Value-State Gated Attention (VGA), the first method to use the value vector itself as the gating signal source. VGA introduces a data-dependent, learnable gating function at the attention output layer, explicitly decoupling attention score computation from value-state update and thereby breaking their pathological coupling. Guided by gradient-based theoretical analysis, the gating mechanism is optimized for dynamic, precise regulation of value-state magnitudes. Experiments demonstrate that VGA significantly mitigates attention sinks, stabilizes value-state norms, and enhances inference stability, quantization fidelity, and attention interpretability—without compromising model capacity.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large models based on the Transformer architecture are susceptible to extreme-token phenomena, such as attention sinks and value-state drains. These issues, which degrade model performance, quantization fidelity, and interpretability, arise from a problematic mutual reinforcement mechanism where the model learns an inefficient 'no-op' behavior by focusing attention on tokens with near-zero value states. In this paper, we propose Value-State Gated Attention (VGA), a simple, dedicated, and stable architectural mechanism for performing 'no-op' attention efficiently by directly breaking this cycle. VGA introduces a learnable, data-dependent gate, computed directly from the value vectors (V), to modulate the output. Through a theoretical analysis of the underlying gradients, we show that gating the value-state with a function of itself is more effective at decoupling value and attention score updates than prior methods that gate on input embeddings. This creates a direct regulatory pathway that allows the model to suppress a token's contribution based on its emergent value representation. Our experiments demonstrate that VGA significantly mitigates the formation of attention sinks and stabilizes value-state norms, leading to improved performance, robust quantization fidelity, and enhanced model interpretability.

Problem

• Mitigating attention sinks and value-state drains in Transformer models
• Breaking mutual reinforcement between attention scores and value states
• Improving model performance, quantization fidelity, and interpretability

Innovation

• Value-State Gated Attention breaks mutual reinforcement cycle
• Learnable gate modulates output using value vectors directly
• Suppresses token contribution based on emergent value representation