Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning

2026-04-15

AI Summary

This work addresses the pervasive challenges of task interference and catastrophic forgetting in supervised fine-tuning, which existing parameter isolation methods fail to adequately resolve due to their reliance on static assumptions about parameter importance. To overcome this limitation, the authors propose the Evolving Parameter Isolation (EPI) framework, which is the first to explicitly model and exploit the temporal drift of parameter importance during training. EPI employs online gradient estimation and a dynamic mask-updating mechanism to adaptively adjust parameter isolation strategies throughout the training process. This enables the dynamic protection and release of critical parameters, substantially mitigating interference and forgetting in multi-task settings. Empirical results demonstrate that EPI consistently outperforms both static isolation approaches and standard fine-tuning across multiple benchmarks, leading to significantly improved model generalization.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Supervised Fine-Tuning (SFT) of large language models often suffers from task interference and catastrophic forgetting. Recent approaches alleviate this issue by isolating task-critical parameters during training. However, these methods represent a static solution to a dynamic problem, assuming that parameter importance remains fixed once identified. In this work, we empirically demonstrate that parameter importance exhibits temporal drift over the course of training. To address this, we propose Evolving Parameter Isolation (EPI), a fine-tuning framework that adapts isolation decisions based on online estimates of parameter importance. Instead of freezing a fixed subset of parameters, EPI periodically updates isolation masks using gradient-based signals, enabling the model to protect emerging task-critical parameters while releasing outdated ones to recover plasticity. Experiments on diverse multi-task benchmarks demonstrate that EPI consistently reduces interference and forgetting compared to static isolation and standard fine-tuning, while improving overall generalization. Our analysis highlights the necessity of synchronizing isolation mechanisms with the evolving dynamics of learning diverse abilities.

Problem

• parameter importance
• supervised fine-tuning
• task interference
• catastrophic forgetting
• temporal drift

Innovation

• Evolving Parameter Isolation
• parameter importance drift
• dynamic parameter isolation
• supervised fine-tuning
• catastrophic forgetting