ARTEMIS: Agent-guided Reliability-aware Temporal Mask Evolution for Imperfectly Supervised Video Polyp Segmentation

2026-06-18

AI Summary

This work addresses the challenges of boundary ambiguity, motion blur, specular highlights, and sparse annotations in weakly supervised (point- or scribble-based) and semi-supervised video polyp segmentation, which often lead to geometrically distorted masks and temporal inconsistency. To tackle these issues, the authors propose ARTEMIS, a unified framework that initializes masks using SAM2 and employs a vision-language debate-and-critique agent to identify reliable temporal anchor frames. ARTEMIS further integrates reliability-aware reference selection, prototype transfer, and bidirectional temporal propagation to dynamically evolve temporally coherent and geometrically accurate segmentation masks. A reliability-weighted robust loss is introduced to preserve informative hard samples during training. Evaluated under various sparse annotation settings on the SUN-SEG and CVC-ClinicDB-612 datasets, ARTEMIS achieves state-of-the-art performance, significantly outperforming existing methods.

Abstract

Imperfectly supervised video polyp segmentation (VPS) aims to learn dense, temporally consistent masks from inexpensive supervision, including weak annotations (points, scribbles) and semi-supervision with few densely labeled frames. This setting is clinically valuable but challenging due to weak contrast, ambiguous boundaries, motion blur, and specular highlights, compounded by sparse pixel-level guidance. While SAM2 can generate dense masks from sparse inputs, direct pseudo-labeling often yields geometry-degraded masks with boundary leakage, underutilizes temporal consistency, and ignores reliability. To address these issues, we propose ARTEMIS, a unified framework for imperfectly supervised VPS driven by agent-guided reliability-aware temporal mask evolution. ARTEMIS initializes coarse masks from available supervision: SAM2 converts points/scribbles, while dense labels serve as reliable anchors. A debate-and-judge vision-language agent selects reliable temporal anchors under weak supervision, which are propagated bidirectionally with SAM2 to refine unreliable or unlabeled frames. Finally, ARTEMIS trains the segmenter using temporal reliability-aware robust learning, incorporating reliability-guided reference selection, a Reference Prototype Transport Module, and reliability-aware robust loss. These components assess mask reliability, evolve anchors over time, transport target identity across frames, and down-weight noisy supervision instead of discarding difficult samples. Experiments on SUN-SEG and CVC-ClinicDB-612 under scribble, point, and limited-label settings demonstrate that ARTEMIS achieves state-of-the-art performance. Code will be released at https://github.com/wangtong627/ARTEMIS.

Problem

• video polyp segmentation
• imperfect supervision
• temporal consistency
• weak annotation
• mask reliability

Innovation

• reliability-aware learning
• temporal mask evolution
• agent-guided segmentation
• imperfectly supervised VPS
• SAM2-based propagation