FeVOS: Foresight Expression Video Object Segmentation

2026-06-24

AI Summary

This work addresses the limitation of existing video referring expression segmentation methods, which lack the ability to anticipate future events and thus struggle to support applications requiring early understanding of subsequent actions. To bridge this gap, we introduce a novel task termed Future-aware Video Object Segmentation (FeVOS), which generates segmentation masks for relevant objects in observed video frames based on questions about future events. We construct the first dataset annotated to support chained spatiotemporal reasoning and propose a two-stage training framework leveraging multimodal large language models—combining supervised fine-tuning and reinforcement learning—to develop our FeVOS-R1 model. Experimental results demonstrate that FeVOS-R1 achieves state-of-the-art performance on our dataset and exhibits strong generalization capabilities on the RVOS benchmark, advancing predictive reasoning in video perception.

Abstract

Existing Referring Video Object Segmentation tasks focus on referring expressions describing events, actions or appearances of relevant objects within the observed frames, lacking evaluation in scenarios that require pre-decisive spatio-temporal reasoning, thereby limiting their applicability. To address this, we propose Foresight Expression Video Object Segmentation, a task that queries future events in upcoming video segments and requires masks of the objects in the observed frames as visual answers. For example, in ego-centric scenes, the question "What tool will be used?" demands reasoning over spatio-temporal cues to predict the masks of the next tool to be used, which helps with the understanding of future actions and decisions. To support this task, we introduce FeVOS, a dataset with 968 video clips, 14,525 foresight expressions, and 2,904 chain-of-thought annotations to provide explicit and interpretable reasoning steps. We further develop FeVOS-R1, an MLLM-based model trained on our dataset via a two-stage pipeline of supervised fine-tuning and reinforcement learning. FeVOS-R1 not only achieves state-of-the-art performance on FeVOS, but also demonstrates strong generalization to existing RVOS benchmarks. We hope this work can inspire more research on predictive reasoning in video perception.

Problem

• Referring Video Object Segmentation
• Foresight Expression
• Spatio-temporal Reasoning
• Video Object Segmentation
• Predictive Reasoning

Innovation

• Foresight Expression
• Video Object Segmentation
• Predictive Reasoning
• Multimodal Large Language Model
• Chain-of-Thought Annotation