GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation

2025-06-03

AI Summary

To address the insufficient robustness of the Segment Anything Model (SAM) under input degradations (e.g., noise, blur, rain, fog), this paper proposes a Gated Rank-Adaptive (GaRA) mechanism. GaRA inserts lightweight adapters into frozen intermediate layers of SAM’s backbone and employs learnable gating to dynamically select and activate rank-1 weight components, enabling parameter-efficient, input-aware, and fine-grained robustness enhancement. Crucially, GaRA requires no backbone fine-tuning, introduces zero additional inference latency, and remains fully compatible with standard supervised training. Evaluated on robust segmentation benchmarks—including ACDC—GaRA achieves state-of-the-art performance: on ACDC, it improves mean IoU by 21.3 percentage points over baseline SAM. This substantial gain significantly enhances SAM’s reliability for safety-critical applications such as autonomous driving and robotics.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Improving robustness of the Segment Anything Model (SAM) to input degradations is critical for its deployment in high-stakes applications such as autonomous driving and robotics. Our approach to this challenge prioritizes three key aspects: first, parameter efficiency to maintain the inherent generalization capability of SAM; second, fine-grained and input-aware robustification to precisely address the input corruption; and third, adherence to standard training protocols for ease of training. To this end, we propose gated-rank adaptation (GaRA). GaRA introduces lightweight adapters into intermediate layers of the frozen SAM, where each adapter dynamically adjusts the effective rank of its weight matrix based on the input by selectively activating (rank-1) components of the matrix using a learned gating module. This adjustment enables fine-grained and input-aware robustification without compromising the generalization capability of SAM. Our model, GaRA-SAM, significantly outperforms prior work on all robust segmentation benchmarks. In particular, it surpasses the previous best IoU score by up to 21.3%p on ACDC, a challenging real corrupted image dataset.

Problem

• Enhancing SAM robustness to input degradations
• Maintaining parameter efficiency and generalization
• Achieving fine-grained input-aware adaptation

Innovation

• Gated-rank adaptation for robust segmentation
• Lightweight adapters in frozen SAM layers
• Dynamic rank adjustment via learned gating