Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

2026-08-12

AI Summary

This study investigates the formation mechanisms and structural regularities of large-scale activations in hybrid linear-attention large language models. Through systematic outlier analysis, controlled pretraining, output gating ablation, and cross-architecture comparisons, the work identifies and formally names two architecture-aligned activation patterns: Pre-Attention Spikes (PAS) preceding full attention layers and Inter-Spike Plateaus (ISP) within intermediate linear layers. The authors propose a unified lifecycle framework to explain these phenomena, demonstrating its validity across models ranging from 1.2B to 397B parameters. Controlled experiments on a 1.3B model further reveal that PAS and ISP emerge early in training and exhibit asymmetric responses to output gating, providing empirical support for the delayed cancellation hypothesis.

Abstract

We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs. Our code is available at https://github.com/StartluxLabs/Massive-Activations-HLA.

Problem

• Massive Activations
• Hybrid Linear Attention
• Pre-Attention Spikes
• Inter-Spike Plateaus
• Large Language Models

Innovation

• Massive Activations
• Hybrid Linear Attention
• Pre-Attention Spikes
• Inter-Spike Plateaus
• GDN gating