AgentRVOS for MeViS-Text Track of 5th PVUW Challenge: 3rd Method

2026-04-20

AI Summary

This work addresses the task of referring video object segmentation (Ref-VOS), where the goal is to segment visual targets specified by textual expressions in videos. The authors propose Sa2VA, a multi-agent framework that explicitly models the collaboration between target existence verification and mask generation. The approach employs a role-specialized agent architecture—comprising a planner, partitioner, scout, refiner, and critic—to iteratively validate, correct, and refine initial semantic hypotheses. Temporal propagation and closed-loop refinement are further integrated via SAM3 to enhance consistency and accuracy across frames. Evaluated on the MeViS-Text benchmark, the method achieves third place, demonstrating significant improvements in both target existence classification accuracy and segmentation mask quality.

Technology Category

• Computer Vision: Segmentation
• Multiagent Systems: Adversarial Agents
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This report describes a Ref-VOS pipeline centered on Sa2VA and organized with explicit agent roles. The key idea is that Sa2VA should provide the first dense semantic hypothesis, while an agent loop decides whether that hypothesis should be accepted, revised, or refined. The pipeline starts with a target-presence judgment stage. If the referred object does not exist in the video, the system directly outputs zero masks. Otherwise, Sa2VA receives the video and referring prompt and produces a coarse mask trajectory over the full video. This trajectory is treated as a semantic prior rather than a final answer. A planner agent decomposes the query, temporal partition agents identify informative blocks, scout agents search for anchor frames, and refinement agents convert reliable Sa2VA masks into boxes and points for SAM3 propagation. A critic scores candidate trajectories, a reflection controller repairs weak hypotheses, and a collaboration controller reconciles multiple agent branches. The result is a Ref-VOS system in which Sa2VA is responsible for dense grounded understanding, while the agent layer handles presence verification, temporal search, confidence-aware revision, and final mask refinement.

Problem

• Referring Video Object Segmentation
• Video Object Segmentation
• Text-to-Video Grounding
• Ref-VOS
• Visual Referring Expression

Innovation

• Ref-VOS
• multi-agent collaboration
• Sa2VA
• semantic prior
• SAM3 propagation