Attribution-Guided Continual Learning for Large Language Models

2026-05-06

AI Summary

This work addresses the challenge of catastrophic forgetting in large language models (LLMs) during continual learning, where performance on previously learned tasks degrades significantly after acquiring new ones. The authors propose an attribution-guided continual fine-tuning framework that, for the first time, incorporates fine-grained, task-specific parameter importance attribution into LLM continual learning. By evaluating element-wise parameter importance and modulating gradients based on attribution scores, the method adaptively controls the magnitude of parameter updates across all Transformer layers, enabling semantic-aware preservation of prior knowledge. Extensive experiments demonstrate that the proposed approach substantially outperforms existing methods on multiple continual learning benchmarks, effectively retaining knowledge from old tasks while maintaining strong learning capacity on new tasks.

Abstract

Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization. However, these methods lack semantic awareness of internal knowledge distribution in LLMs. As a result, they cannot distinguish parameters that should be preserved or updated. We propose an attribution-guided continual fine-tuning framework for LLMs. Our method estimates task-specific, element-wise parameter importance in each Transformer layer and uses these scores to modulate gradients. Parameters important to previous tasks receive smaller updates, while less relevant ones remain plastic for learning new tasks. Experiments on continual learning benchmarks show that our method consistently outperforms baselines, achieving better retention of old tasks while maintaining competitive performance on new tasks.

Problem

• catastrophic forgetting
• continual learning
• large language models
• task retention

Innovation

• attribution-guided
• continual learning
• large language models
• parameter importance
• catastrophic forgetting