Toward Robust In-Context Segmentation via Concept Guidance

2026-06-26

AI Summary

This work addresses the instability and lack of robustness in existing in-context segmentation methods, which produce inconsistent results for the same query image under varying reference images. To tackle this issue, the study reformulates the task from a robustness perspective and introduces a concept-guided in-context segmentation paradigm. It leverages a multimodal large language model to generate high-level semantic concepts, which—combined with visual exemplars—jointly activate a frozen SAM3 model. A concept reasoning module and a tree-search optimization mechanism are further integrated to enable synergistic semantic guidance and spatial localization. The proposed approach achieves state-of-the-art accuracy on standard benchmarks while significantly reducing output variance across different reference images, thereby substantially enhancing system robustness.

Abstract

In-context segmentation (ICS) requires a model to segment target regions in a query image using only a few reference images and their corresponding masks, without updating any parameters. Despite recent progress, prior ICS studies have largely overlooked a critical aspect: system robustness, ie, whether the model can produce stable segmentation results for the same query under different references. In this work, we revisit ICS from the robustness perspective and introduce a novel paradigm, Concept-Guided In-Context Segmentation (CG-ICS), which performs segmentation by extracting high-level semantic concepts from references rather than relying solely on low-level visual matching. Specifically, CG-ICS introduces a concept reasoning module that uses an MLLM to propose candidates and a SAM3-driven scoring function with tree-search refinement to select reliable textual concepts, together with a parallel visual exemplar route that provides query-side spatial grounding via a simple context construction. Both the textual concept and the visual exemplar are then used to activate the segmentation capability of a frozen SAM3 backbone. Extensive experiments on standard ICS benchmarks demonstrate that CG-ICS not only achieves state-of-the-art accuracy but also substantially improves robustness, yielding a more reliable ICS system with significantly reduced variance across diverse reference choices.

Problem

• in-context segmentation
• robustness
• reference variability
• stable segmentation
• semantic consistency

Innovation

• Concept Guidance
• In-Context Segmentation
• Robustness
• Semantic Reasoning
• Frozen SAM3