FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation

2025-03-23

AI Summary

Visual foundation models often suffer from degraded generalization in domain-generalized semantic segmentation (DGSS) fine-tuning due to task and distribution shifts. To address this, we propose a robust fine-tuning method based on a Domain-Relevant Fisher Information Matrix (DR-FIM). First, we formally define DR-FIM to quantify parameter sensitivity across domains and tasks. Second, we integrate variational inference for stable DR-FIM estimation and incorporate pretrained weight priors into Fisher-guided optimization, jointly enhancing both cross-domain adaptability and out-of-distribution generalization. Third, we adopt Gaussian parameter modeling with prior regularization to further improve robustness. Extensive experiments on multiple DGSS benchmarks demonstrate that our method significantly outperforms selective fine-tuning and adapter-based approaches, achieving higher cross-domain segmentation accuracy while preserving strong generalization capability.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision Foundation Models (VFMs) excel in generalization due to large-scale pretraining, but fine-tuning them for Domain Generalized Semantic Segmentation (DGSS) while maintaining this ability remains challenging. Existing approaches either selectively fine-tune parameters or freeze the VFMs and update only the adapters, both of which may underutilize the VFMs' full potential in DGSS tasks. We observe that domain-sensitive parameters in VFMs, arising from task and distribution differences, can hinder generalization. To address this, we propose extbf{FisherTune}, a robust fine-tuning method guided by the Domain-Related Fisher Information Matrix (DR-FIM). DR-FIM measures parameter sensitivity across tasks and domains, enabling selective updates that preserve generalization and enhance DGSS adaptability. FisherTune incorporates variational inference to stabilize DR-FIM estimation, treating parameters as Gaussian-distributed variables and leveraging pre-trained priors. Extensive experiments show that FisherTune achieves superior cross-domain segmentation while maintaining generalization, outperforming selective-parameter and adapter-based methods.

Problem

• Fine-tuning Vision Foundation Models for domain generalization
• Identifying domain-sensitive parameters hindering generalization
• Enhancing adaptability while preserving model generalization

Innovation

• Fisher-guided tuning for domain generalization
• Variational inference stabilizes Fisher estimation
• Selective parameter updates enhance adaptation