SALT: Singular Value Adaptation with Low-Rank Transformation

2025-03-20

AI Summary

To address the challenge of balancing parameter efficiency and representational capacity in few-shot fine-tuning for medical image segmentation, this paper proposes a novel parameter-efficient fine-tuning (PEFT) method integrating singular value–selective adaptation with low-rank updates. Its core contribution is a learnable singular value scaling-and-shifting mechanism that jointly refines the dominant subspace and compensates via residual low-rank updates, enabling precise domain adaptation of large foundation models with minimal trainable parameters. Evaluated on five medical imaging datasets (20–1,000 annotated samples), the method achieves average Dice score improvements of 2%–5% using only 3.9% trainable parameters—significantly outperforming LoRA and full-rank SVD. It effectively overcomes LoRA’s underfitting and SVD’s limited adaptability, establishing a new state-of-the-art in parameter-efficient medical segmentation adaptation.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The complex nature of medical image segmentation calls for models that are specifically designed to capture detailed, domain-specific features. Large foundation models offer considerable flexibility, yet the cost of fine-tuning these models remains a significant barrier. Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), efficiently update model weights with low-rank matrices but may suffer from underfitting when the chosen rank is insufficient to capture domain-specific nuances. Conversely, full-rank Singular Value Decomposition (SVD) based methods provide comprehensive updates by modifying all singular values, yet they often lack flexibility and exhibit variable performance across datasets. We propose SALT (Singular Value Adaptation with Low-Rank Transformation), a method that selectively adapts the most influential singular values using trainable scale and shift parameters while complementing this with a low-rank update for the remaining subspace. This hybrid approach harnesses the advantages of both LoRA and SVD, enabling effective adaptation without relying on increasing model size or depth. Evaluated on 5 challenging medical datasets, ranging from as few as 20 samples to 1000, SALT outperforms state-of-the-art PEFT (LoRA and SVD) by 2% to 5% in Dice with only 3.9% trainable parameters, demonstrating robust adaptation even in low-resource settings. The code for SALT is available at: https://github.com/BioMedIA-MBZUAI/SALT

Problem

• Efficiently adapts large models for medical image segmentation.
• Addresses underfitting in low-rank adaptation methods.
• Combines low-rank and full-rank updates for robust performance.

Innovation

• Selective adaptation of influential singular values
• Combines LoRA and SVD for efficient fine-tuning
• Achieves robust performance with minimal trainable parameters