🤖 AI Summary
This study addresses catastrophic forgetting in large language models during continual fine-tuning, wherein learning new tasks leads to the loss of previously acquired knowledge. Through systematic investigation on Transformer models ranging from 109B to 400B parameters, the work reveals for the first time that this phenomenon is jointly driven by interference in attention weight gradients, drift in intermediate layer representations, and flattening of the loss landscape. Employing sequential task fine-tuning, gradient alignment analysis, representational similarity metrics, and perturbation tracking of attention heads, the authors demonstrate a strong correlation between forgetting severity and task similarity (Pearson r = 0.87). Notably, 15%–23% of critical attention heads suffer significant degradation during fine-tuning, with lower layers exhibiting heightened sensitivity, thereby providing both mechanistic insight and empirical grounding for continual learning in large-scale models.
📝 Abstract
Large language models exhibit remarkable performance across diverse tasks through pre-training and fine-tuning paradigms. However, continual fine-tuning on sequential tasks induces catastrophic forgetting, where newly acquired knowledge interferes with previously learned capabilities. Despite widespread observations of this phenomenon, the mechanistic understanding remains limited. Here, we present a comprehensive mechanistic analysis of catastrophic forgetting in transformer-based LLMs during sequential fine-tuning. Through systematic experiments across multiple model scales (109B to 400B total parameters) and task sequences, we identify three primary mechanisms driving forgetting: gradient interference in attention weights, representational drift in intermediate layers, and loss landscape flattening. We demonstrate that forgetting severity correlates strongly with task similarity (Pearson r = 0.87) and gradient alignment metrics. Our analysis reveals that approximately 15 to 23 percent of attention heads undergo severe disruption during fine-tuning, with lower layers showing greater susceptibility. These findings establish mechanistic foundations for developing targeted mitigation strategies in continual learning systems.