Weakly-Supervised Referring Video Object Segmentation through Text Supervision

2026-04-20

AI Summary

This work addresses the reliance on costly pixel-level annotations in referring video object segmentation by proposing WSRVOS, the first weakly supervised method that operates solely with textual descriptions. Leveraging a multimodal large language model, WSRVOS generates positive and negative text-augmented samples and employs bidirectional vision-language feature interaction, instance-aware expression classification, and a positive-prediction fusion strategy to produce high-quality pseudo-masks. To enhance temporal consistency, it further introduces a temporal clip ordering constraint. Evaluated on four benchmarks—A2D Sentences, J-HMDB Sentences, Ref-YouTube-VOS, and Ref-DAVIS17—the method achieves state-of-the-art performance, demonstrating the feasibility and effectiveness of purely text-supervised referring video object segmentation.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Referring video object segmentation (RVOS) aims to segment the target instance in a video, referred by a text expression. Conventional approaches are mostly supervised learning, requiring expensive pixel-level mask annotations. To tackle it, weakly-supervised RVOS has recently been proposed to replace mask annotations with bounding boxes or points, which are however still costly and labor-intensive. In this paper, we design a novel weakly-supervised RVOS method, namely WSRVOS, to train the model with only text expressions. Given an input video and the referring expression, we first design a contrastive referring expression augmentation scheme that leverages the captioning capabilities of a multimodal large language model to generate both positive and negative expressions. We extract visual and linguistic features from the input video and generated expressions, then perform bi-directional vision-language feature selection and interaction to enable fine-grained multimodal alignment. Next, we propose an instance-aware expression classification scheme to optimize the model in distinguishing positive from negative expressions. Also, we introduce a positive-prediction fusion strategy to generate high-quality pseudo-masks, which serve as additional supervision to the model. Last, we design a temporal segment ranking constraint such that the overlaps between mask predictions of temporally neighboring frames are required to conform to specific orders. Extensive experiments on four publicly available RVOS datasets, including A2D Sentences, J-HMDB Sentences, Ref-YouTube-VOS, and Ref-DAVIS17, demonstrate the superiority of our method. Code is available at \href{https://github.com/viscom-tongji/WSRVOS}{https://github.com/viscom-tongji/WSRVOS}.

Problem

• Referring Video Object Segmentation
• Weakly-Supervised Learning
• Text Supervision
• Video Object Segmentation
• Vision-Language Alignment

Innovation

• weakly-supervised learning
• referring video object segmentation
• text-only supervision
• vision-language alignment
• pseudo-mask generation