Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance

2025-08-29

AI Summary

To address the pervasive “seesaw effect” (i.e., performance trade-offs across tasks) and catastrophic forgetting in multi-task fine-tuning of large language models (LLMs), this paper proposes Core Parameter Isolation (CPI), a novel fine-tuning framework. CPI first clusters tasks based on overlap of task-specific core parameter regions, then applies core parameter transplantation alongside spherical linear interpolation (SLERP) to smoothly fuse non-core parameters, while freezing task-dedicated core regions during fine-tuning. Its technical pipeline includes standalone supervised fine-tuning (SFT) for core parameter identification and streamlined lightweight multi-task training. Evaluated on multiple public benchmarks, CPI consistently outperforms conventional multi-task and sequential fine-tuning baselines, significantly mitigating inter-task interference and enhancing both overall performance and training stability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Supervised fine-tuning (SFT) is a pivotal approach to adapting large language models (LLMs) for downstream tasks; however, performance often suffers from the ``seesaw phenomenon'', where indiscriminate parameter updates yield progress on certain tasks at the expense of others. To address this challenge, we propose a novel emph{Core Parameter Isolation Fine-Tuning} (CPI-FT) framework. Specifically, we first independently fine-tune the LLM on each task to identify its core parameter regions by quantifying parameter update magnitudes. Tasks with similar core regions are then grouped based on region overlap, forming clusters for joint modeling. We further introduce a parameter fusion technique: for each task, core parameters from its individually fine-tuned model are directly transplanted into a unified backbone, while non-core parameters from different tasks are smoothly integrated via Spherical Linear Interpolation (SLERP), mitigating destructive interference. A lightweight, pipelined SFT training phase using mixed-task data is subsequently employed, while freezing core regions from prior tasks to prevent catastrophic forgetting. Extensive experiments on multiple public benchmarks demonstrate that our approach significantly alleviates task interference and forgetting, consistently outperforming vanilla multi-task and multi-stage fine-tuning baselines.

Problem

• Addresses task interference in multi-task fine-tuning
• Mitigates catastrophic forgetting during sequential training
• Identifies and isolates core parameters per task

Innovation

• Core Parameter Isolation Fine-Tuning framework
• Parameter fusion with SLERP integration
• Lightweight pipelined SFT with frozen cores