What Does Layer-Importance Reveal About Transformers and State-Space Models?

2026-09-14

AI Summary

研究通过分析层重要性（必要性和可塑性）揭示了Transformer和状态空间模型在序列建模中的行为差异及其对下游任务适应性的影响。

Abstract

Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical knowledge built for transformers transfers to SSMs. We address this through the lens of layer importance which underpins compression, selective fine-tuning, and interpretability across both families. We decompose layer importance into two distinct notions. \emph{Necessity} captures how much the pretrained model depends on a layer's existing contribution, measured by the loss increase from bypassing it. \emph{Plasticity} captures where the model absorbs new information during fine-tuning, measured by the magnitude of task-specific weight updates. Our analysis reveals that the two families behave fundamentally differently: in every evaluated residual transformer up to $14$B parameters, Necessity and Plasticity anti-align across depth, whereas in the evaluated Mamba-style SSMs they point to overlapping regions. The sign of this alignment also predicts downstream adaptation behavior. In the evaluated transformers, concentrating updates in the most plastic layers increases catastrophic forgetting, while this tier-dependent effect disappears in the evaluated Mamba-style SSMs.

Problem

• Transformers
• state-space models
• layer importance
• compression
• selective fine-tuning

Innovation

• Layer Importance
• Necessity
• Plasticity
• Transformers
• State-Space Models