VOS-Agent: The 1st Place Solution for the 8th LSVOS Challenge (MOSEv2 Track)

2026-08-12

AI Summary

This work addresses the instability in propagating object masks in complex video object segmentation (VOS) scenarios, where targets suffer from severe occlusion, disappearance, and reappearance. To tackle this challenge, the authors propose a collaborative VOS-Agent framework built upon SAM3 as a shared dense segmentation backbone. The framework introduces a target-aware routing mechanism that dynamically assigns each target to one of three specialized pathways—regular, tiny, or semantics-dominated—each handled by a dedicated agent. These agents integrate visual trackers to generate confidence-aware bounding box prompts and leverage multimodal large language models (MLLMs) for description-guided localization and candidate verification. Evaluated on the MOSEv2 test set, the method achieves a 𝒥&F score of 69.82%, securing first place in the MOSEv2 track of the 8th LSVOS Challenge at ECCV 2026.

Abstract

Complex video object segmentation requires robust target propagation under severe occlusion, disappearance and reappearance. Although SAM3 provides strong promptable mask propagation, a uniform inference path remains unreliable for tiny targets with insufficient visual evidence and semantic-dominated targets whose identities depend on explicit attributes. To this end, we present VOS-Agent, a collaborative framework that retains SAM3 as the shared dense segmentation module and conditionally activates specialized agents according to target characteristics. A Target Perception and Routing Agent assigns each sequence to a regular, tiny, or semantic-dominated route. Tiny targets are supported by a Visual Tracking Agent through confidence-aware box prompts, while semantic-dominated targets are handled by an MLLM-based Semantic Agent through description-guided localization and candidate verification. On the MOSEv2 test set, VOS-Agent achieves 69.82% on the official $\mathcal{J}\&\dot{\mathcal{F}}$ metric and ranks first in the MOSEv2 Track of the 8th LSVOS Challenge at ECCV 2026.

Problem

• video object segmentation
• occlusion
• tiny targets
• semantic-dominated targets
• target propagation

Innovation

• VOS-Agent
• conditional agent routing
• tiny object tracking
• semantic-guided segmentation
• SAM3