Agreement-Based Audio-Visual Segmentation:Champion Report for the MeViS-Audio Track in the 8th LSVOS Challenge

2026-08-10

AI Summary

This work addresses the task of referring dynamic object segmentation in videos guided by spoken language, where the model must output an empty mask when the referred object is absent. To tackle this challenge, the authors propose a staged approach: first transcribing speech to text using Qwen3-ASR, then generating candidate mask trajectories through multiple complementary vision-based localization and segmentation models. A novel mask-consistency-based selection mechanism is introduced, augmented with explicit linguistic rules—such as spatial direction, cardinality, and plural semantics—and a joint audio-visual multimodal classifier to jointly assess target presence and select the optimal segmentation result. The method achieves state-of-the-art performance on the MeViS-Audio track, securing first place with a J&F score of 0.5952, absence accuracy of 0.7931, presence accuracy of 0.9205, and an overall score of 0.7696.

Abstract

The MeViS-Audio track asks a system to segment the objects described by a spoken motion expression throughout a video and to return empty masks when the described target is absent. We present a simple staged solution. Qwen3-ASR first converts speech into text. Several video mask tracks are then produced with complementary grounding and segmentation models. Instead of trusting a single prediction, we select the track that has the highest average mask agreement with the other candidates. A small set of explicit direction, count, and plural rules corrects queries that require more than ordinary single-object tracking. Finally, a video-level classifier combines visual, audio-visual, and within-video query scores to decide whether any target is present. The submitted system obtains 0.5952 J &F, 0.7931 no-target accuracy, 0.9205 target accuracy, and a final score of 0.769589. The challenge organizers notified our team that this result ranked first in the track.

Problem

• audio-visual segmentation
• spoken motion expression
• video object segmentation
• absent target detection

Innovation

• audio-visual segmentation
• mask agreement
• staged pipeline
• query grounding
• video-level classification