SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning

2024-12-05

AI Summary

To address the core challenge in domain generalization—where parameter-efficient fine-tuning (PEFT) struggles to simultaneously preserve pre-trained model generalizability and enhance task-discriminative capacity—we propose Singular Value Decomposition (SVD)-based Minor Component Adaptation. Our method freezes the dominant singular vectors to retain generic representations, while selectively fine-tuning only the minor components associated with the smallest singular values to learn domain-invariant discriminative features. This work is the first to introduce SVD into domain generalization fine-tuning. We further devise a novel module-level freezing scheme and an annealed weight decay strategy, eliminating the need for auxiliary regularization or inference overhead. The approach is fully compatible with arbitrary backbone architectures and detection heads. Extensive experiments on multiple domain generalization benchmarks for semantic segmentation and object detection demonstrate state-of-the-art performance, significantly improving cross-domain transferability.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Domain generalization (DG) aims to adapt a model using one or multiple source domains to ensure robust performance in unseen target domains. Recently, Parameter-Efficient Fine-Tuning (PEFT) of foundation models has shown promising results in the context of DG problem. Nevertheless, existing PEFT methods still struggle to strike a balance between preserving generalizable components of the pre-trained model and learning task-specific features. To gain insights into the distribution of generalizable components, we begin by analyzing the pre-trained weights through the lens of singular value decomposition. Building on these insights, we introduce Singular Value Decomposed Minor Components Adaptation (SoMA), an approach that selectively tunes minor singular components while keeping the residual parts frozen. SoMA effectively retains the generalization ability of the pre-trained model while efficiently acquiring task-specific skills. Moreover, we freeze domain-generalizable blocks and employ an annealing weight decay strategy, thereby achieving an optimal balance in the delicate trade-off between generalizability and discriminability. SoMA attains state-of-the-art results on multiple benchmarks that span both domain generalized semantic segmentation to domain generalized object detection. In addition, our methods introduce no additional inference overhead or regularization loss, maintain compatibility with any backbone or head, and are designed to be versatile, allowing easy integration into a wide range of tasks.

Problem

• Balancing generalizable and task-specific features in domain generalization
• Analyzing pre-trained weights via singular value decomposition for insights
• Selectively tuning minor singular components to retain generalization ability

Innovation

• Selectively tunes minor singular components
• Freezes domain-generalizable blocks
• Employs annealing weight decay strategy