ReVSeg: Incentivizing the Reasoning Chain for Video Segmentation with Reinforcement Learning

2025-12-02

AI Summary

Existing video object segmentation methods implicitly compress dynamic modeling, causal reasoning, and temporal interactions into opaque black-box embeddings, resulting in uninterpretable inference and limited generalization. To address this, we propose an explicit three-stage reasoning framework grounded in a pre-trained vision-language model: (1) semantic parsing, (2) temporal evidence selection, and (3) pixel-level spatial localization. The entire inference process is formalized as an interpretable sequential decision chain, and reinforcement learning is employed to jointly optimize the multi-step policy end-to-end based on segmentation feedback. Our approach achieves state-of-the-art performance across multiple standard video segmentation benchmarks. Crucially, it generates human-readable intermediate reasoning traces—enhancing model transparency and controllability—while maintaining high segmentation accuracy.

Technology Category

• Computer Vision: Segmentation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reasoning-centric video object segmentation is an inherently complex task: the query often refers to dynamics, causality, and temporal interactions, rather than static appearances. Yet existing solutions generally collapse these factors into simplified reasoning with latent embeddings, rendering the reasoning chain opaque and essentially intractable. We therefore adopt an explicit decomposition perspective and introduce ReVSeg, which executes reasoning as sequential decisions in the native interface of pretrained vision language models (VLMs). Rather than folding all reasoning into a single-step prediction, ReVSeg executes three explicit operations -- semantics interpretation, temporal evidence selection, and spatial grounding -- aligning pretrained capabilities. We further employ reinforcement learning to optimize the multi-step reasoning chain, enabling the model to self-refine its decision quality from outcome-driven signals. Experimental results demonstrate that ReVSeg attains state-of-the-art performances on standard video object segmentation benchmarks and yields interpretable reasoning trajectories. Project page is available at https://clementine24.github.io/ReVSeg/ .

Problem

• Explicitly decomposes reasoning for video object segmentation
• Optimizes multi-step reasoning chain using reinforcement learning
• Enhances interpretability and performance in video segmentation tasks

Innovation

• Decomposes reasoning into sequential decisions using VLMs
• Uses reinforcement learning to optimize multi-step reasoning chain
• Executes explicit operations for semantics and temporal evidence