BALR-SAM: Boundary-Aware Low-Rank Adaptation of SAM for Resource-Efficient Medical Image Segmentation

2025-09-28

AI Summary

Medical image segmentation poses challenges for foundational vision models (e.g., SAM) due to poor domain adaptability and high fine-tuning costs. To address this, we propose Boundary-Aware Low-Rank Adaptation (BALA), a novel framework integrating boundary enhancement, LoRA adapters, and low-rank tensor attention—enabling prompt-free adaptation with only 1.8% (11.7M) of parameters updated, yet outperforming full-parameter fine-tuning. Leveraging depthwise separable convolutions, multi-scale feature fusion, and ViT embedding optimization, BALA achieves state-of-the-art performance across multiple standard medical segmentation benchmarks—surpassing MedSAM and other SOTA methods. It reduces GPU memory consumption by 75%, accelerates inference, and preserves high-fidelity boundary delineation accuracy—demonstrating superior efficiency and generalizability for clinical deployment.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision foundation models like the Segment Anything Model (SAM), pretrained on large-scale natural image datasets, often struggle in medical image segmentation due to a lack of domain-specific adaptation. In clinical practice, fine-tuning such models efficiently for medical downstream tasks with minimal resource demands, while maintaining strong performance, is challenging. To address these issues, we propose BALR-SAM, a boundary-aware low-rank adaptation framework that enhances SAM for medical imaging. It combines three tailored components: (1) a Complementary Detail Enhancement Network (CDEN) using depthwise separable convolutions and multi-scale fusion to capture boundary-sensitive features essential for accurate segmentation; (2) low-rank adapters integrated into SAM's Vision Transformer blocks to optimize feature representation and attention for medical contexts, while simultaneously significantly reducing the parameter space; and (3) a low-rank tensor attention mechanism in the mask decoder, cutting memory usage by 75% and boosting inference speed. Experiments on standard medical segmentation datasets show that BALR-SAM, without requiring prompts, outperforms several state-of-the-art (SOTA) methods, including fully fine-tuned MedSAM, while updating just 1.8% (11.7M) of its parameters.

Problem

• Adapting SAM for medical image segmentation efficiently
• Reducing computational resources while maintaining segmentation accuracy
• Enhancing boundary detection in medical imaging with fewer parameters

Innovation

• Low-rank adapters optimize medical feature representation
• Depthwise convolutions capture boundary-sensitive segmentation features
• Low-rank tensor attention reduces memory usage by 75%