It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention

2026-09-08

AI Summary

研究分析了大型语言模型中因自注意力集中和值非混合导致的'注意力沉陷'及伴随的大激活现象，为量化策略提供新见解。

Abstract

Large Language Models (LLMs) often exhibit"Attention Sink"(AS) and the accompanying"Massive Activations"(MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it. Our experiments suggest that self-concentration of attention, resulting from the causal mask, and the subsequent Value-non-mixing in attention outputs contribute to AS and MAs. These findings provide new empirical evidence on the internal dynamics of LLMs, offering insights that may inform future quantization strategies and advance our understanding of the internal mechanisms of attention layers.

Problem

• Attention Sink
• Massive Activations
• Large Language Models

Innovation

• self-concentration
• value-non-mixing
• attention sink