Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection

2025-08-23

AI Summary

Fine-tuning pretrained models often degrades their domain generalization capability. To address this, we propose a joint parameter selection mechanism that updates only a sparse, cross-domain gradient-consistent subset of parameters during fine-tuning. Grounded in theoretical analysis linking parameter sparsity to generalization error, our method introduces a dual-operator gradient consistency filter to dynamically identify parameters with high generalization potential for update. This strategy integrates parameter-efficient fine-tuning with gradient-driven structured sparsity, preserving the model’s original domain generalization while enhancing task-specific adaptation. Extensive experiments on multiple standard domain generalization benchmarks demonstrate significant improvements over existing state-of-the-art methods, validating the effectiveness of our approach in balancing generalization and adaptability.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Domain generalization seeks to develop models trained on a limited set of source domains that are capable of generalizing effectively to unseen target domains. While the predominant approach leverages large-scale pre-trained vision models as initialization, recent studies have highlighted that full fine-tuning can compromise the intrinsic generalization capabilities of these models. To address this limitation, parameter-efficient adaptation strategies have emerged, wherein only a subset of model parameters is selectively fine-tuned, thereby balancing task adaptation with the preservation of generalization. Motivated by this paradigm, we introduce Joint Parameter Selection (JPS), a novel method that restricts updates to a small, sparse subset of parameters, thereby retaining and harnessing the generalization strength of pre-trained models. Theoretically, we establish a generalization error bound that explicitly accounts for the sparsity of parameter updates, thereby providing a principled justification for selective fine-tuning. Practically, we design a selection mechanism employing dual operators to identify and update parameters exhibiting consistent and significant gradients across all source domains. Extensive benchmark experiments demonstrate that JPS achieves superior performance compared to state-of-the-art domain generalization methods, substantiating both the efficiency and efficacy of the proposed approach.

Problem

• Preserve generalization in fine-tuning pre-trained models
• Select sparse parameter subset for efficient adaptation
• Balance task adaptation with domain generalization capability

Innovation

• Joint Parameter Selection method
• Sparse parameter updates strategy
• Dual operators gradient mechanism