It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
研究分析了大型语言模型中因自注意力集中和值非混合导致的'注意力沉陷'及伴随的大激活现象，为量化策略提供新见解。
0 citationsRead paper
Selected work
Representative Papers
Recent publications
Latest Papers
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
研究分析了大型语言模型中因自注意力集中和值非混合导致的'注意力沉陷'及伴随的大激活现象，为量化策略提供新见解。
0 citationsRead paper