Efficient Universal Models for Medical Image Segmentation via Weakly Supervised In-Context Learning

2025-10-07

AI Summary

Medical image segmentation foundation models—such as interactive and in-context learning (ICL) approaches—typically rely on dense pixel-level annotations or frequent manual prompts, incurring prohibitive annotation costs. To address this, we propose Weakly Supervised In-Context Learning (WS-ICL), the first framework to integrate weak prompts (e.g., bounding boxes or points) into the ICL paradigm, thereby unifying interactive and context-driven segmentation. WS-ICL synergistically combines weakly supervised learning, prompt-conditioned segmentation networks, and in-context adaptation mechanisms, enabling high-fidelity segmentation from only a few weak prompts. Evaluated on three independent benchmarks, WS-ICL achieves performance on par with fully supervised ICL models and remains highly competitive in interactive settings. Crucially, it reduces annotation overhead by orders of magnitude, significantly enhancing the practicality and scalability of general-purpose segmentation models.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Universal models for medical image segmentation, such as interactive and in-context learning (ICL) models, offer strong generalization but require extensive annotations. Interactive models need repeated user prompts for each image, while ICL relies on dense, pixel-level labels. To address this, we propose Weakly Supervised In-Context Learning (WS-ICL), a new ICL paradigm that leverages weak prompts (e.g., bounding boxes or points) instead of dense labels for context. This approach significantly reduces annotation effort by eliminating the need for fine-grained masks and repeated user prompting for all images. We evaluated the proposed WS-ICL model on three held-out benchmarks. Experimental results demonstrate that WS-ICL achieves performance comparable to regular ICL models at a significantly lower annotation cost. In addition, WS-ICL is highly competitive even under the interactive paradigm. These findings establish WS-ICL as a promising step toward more efficient and unified universal models for medical image segmentation. Our code and model are publicly available at https://github.com/jiesihu/Weak-ICL.

Problem

• Reducing annotation effort for medical image segmentation models
• Replacing dense labels with weak prompts like bounding boxes
• Achieving comparable performance at lower annotation cost

Innovation

• Uses weak prompts like bounding boxes for context
• Reduces annotation cost by eliminating fine-grained masks
• Achieves performance comparable to regular ICL models