Focus on Background: Exploring SAM's Potential in Few-shot Medical Image Segmentation with Background-centric Prompting

2026-03-22

AI Summary

This work addresses the limitations of conventional few-shot medical image segmentation methods and the over-segmentation issues arising from the direct application of the Segment Anything Model (SAM) due to ambiguous anatomical boundaries. To overcome these challenges, the authors reformulate few-shot segmentation with SAM as a prompt localization task and introduce FoB, a background-centric prompt generator. FoB leverages class-agnostic background prompts to model spatial dependencies between foreground and background regions and exploits structural priors inherent in medical image backgrounds for progressive prompt refinement. Evaluated on three diverse medical imaging datasets, the proposed method significantly outperforms existing approaches, achieving state-of-the-art performance in few-shot segmentation and demonstrating strong cross-domain generalization capabilities.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Conventional few-shot medical image segmentation (FSMIS) approaches face performance bottlenecks that hinder broader clinical applicability. Although the Segment Anything Model (SAM) exhibits strong category-agnostic segmentation capabilities, its direct application to medical images often leads to over-segmentation due to ambiguous anatomical boundaries. In this paper, we reformulate SAM-based FSMIS as a prompt localization task and propose FoB (Focus on Background), a background-centric prompt generator that provides accurate background prompts to constrain SAM's over-segmentation. Specifically, FoB bridges the gap between segmentation and prompt localization by category-agnostic generation of support background prompts and localizing them directly in the query image. To address the challenge of prompt localization for novel categories, FoB models rich contextual information to capture foreground-background spatial dependencies. Moreover, inspired by the inherent structural patterns of background prompts in medical images, FoB models this structure as a constraint to progressively refine background prompt predictions. Experiments on three diverse medical image datasets demonstrate that FoB outperforms other baselines by large margins, achieving state-of-the-art performance on FSMIS, and exhibiting strong cross-domain generalization. Our code is available at https://github.com/primebo1/FoB_SAM.

Problem

• few-shot medical image segmentation
• Segment Anything Model
• over-segmentation
• ambiguous anatomical boundaries
• clinical applicability

Innovation

• background-centric prompting
• prompt localization
• few-shot medical image segmentation
• Segment Anything Model
• over-segmentation constraint