DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentation

2025-06-29

AI Summary

Segment Anything Model (SAM) struggles to refine object boundaries effectively under user interactions—especially for camouflaged objects and multi-part targets—due to interference among multiple prompts and limited adaptability at inference time. Method: We propose a test-time adaptation (TTA) framework for interactive segmentation (IS) that partitions user clicks into mutually exclusive subsets, performs lightweight, localized adaptation independently on each subset, and fuses the resulting masks—enabling fine-grained, “divide-and-conquer” knowledge updating. Contribution/Results: Our approach avoids prompt conflict while preserving SAM’s strong prompt generalization and leveraging TTA’s local optimization capability. Evaluated across multiple IS benchmarks, it significantly outperforms zero-shot SAM and state-of-the-art TTA methods using fewer clicks—particularly excelling in camouflaged object segmentation, where boundary precision is most challenging.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Active Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Interactive segmentation (IS) allows users to iteratively refine object boundaries with minimal cues, such as positive and negative clicks. While the Segment Anything Model (SAM) has garnered attention in the IS community for its promptable segmentation capabilities, it often struggles in specialized domains or when handling complex scenarios (e.g., camouflaged or multi-part objects). To overcome these challenges, we propose DC-TTA, a novel test-time adaptation (TTA) framework that adapts SAM on a per-sample basis by leveraging user interactions as supervision. Instead of forcing a single model to incorporate all user clicks at once, DC-TTA partitions the clicks into more coherent subsets, each processed independently via TTA with a separated model. This Divide-and-Conquer strategy reduces conflicts among diverse cues and enables more localized updates. Finally, we merge the adapted models to form a unified predictor that integrates the specialized knowledge from each subset. Experimental results across various benchmarks demonstrate that DC-TTA significantly outperforms SAM's zero-shot results and conventional TTA methods, effectively handling complex tasks such as camouflaged object segmentation with fewer interactions and improved accuracy.

Problem

• Adapts SAM for specialized domains using user interactions
• Reduces conflicts among diverse cues via Divide-and-Conquer
• Improves accuracy in complex scenarios like camouflaged objects

Innovation

• Divide-and-Conquer strategy for user click subsets
• Test-time adaptation with separated model updates
• Merging adapted models for unified prediction