RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

2026-05-08

AI Summary

This work addresses the limitations in video referring segmentation caused by restricted keyframe selection, which undermines temporal reasoning and compromises robustness in multi-object localization. To overcome these challenges, the authors propose a Video Chain-of-Thought (CoT) framework that decouples the task into two stages: Temporal Video Reasoning (TVR) and Keyframe Target Perception (KTP). The approach leverages a task-reward-guided GRPO reinforcement learning agent to self-evaluate and iteratively reselect keyframes, while explicitly separating temporal reasoning from spatial perception through CoT initialization, SAM2-based mask propagation, and high-resolution segmentation. This design significantly outperforms current state-of-the-art methods, enhancing both temporal logical modeling and cross-frame segmentation consistency.

Abstract

Video Reasoning Segmentation (VRS) aims to segment target objects in videos based on implicit instructions that convey human intent and temporal logic. Existing MLLM-based methods predict masks with a [SEG] token after selecting frames via simple sampling or an auxiliary MLLM, where limited supervision and frame-language similarity rules often yield narrow-scope keyframe choices that weaken holistic temporal understanding and lead to brittle localization in complex multi-object scenes. To address these issues, we introduce RCoT-Seg, a video-of-thought framework that factorizes VRS into temporal video reasoning (TVR) and keyframe target perception (KTP), explicitly separating temporal reasoning from spatial perception. Specifically, in the TVR stage, an agentic keyframe selection module, initialized with a curated CoT-start corpus and refined by GRPO under task-aligned rewards, is proposed to generate and reselect the keyframe through self-evaluation, strengthening moment localization and temporal reasoning. In the KTP stage, RCoT-Seg performs high-resolution segmentation on the selected frame and propagates masks with SAM2-based methods across the sequence, replacing heuristic sampling and external selectors while improving spatial precision and inter-frame consistency. Extensive experimental results demonstrate that the proposed RCoT-Seg achieves favorable performance against the state-of-the-art methods. The code and models will be publicly released at https://github.com/Victor-wjw/RCoT-Seg.

Problem

• Video Reasoning Segmentation
• keyframe selection
• temporal reasoning
• multi-object scenes
• mask localization

Innovation

• Reinforced Chain-of-Thought
• Temporal Video Reasoning
• Keyframe Selection
• Video Reasoning Segmentation
• SAM2-based Propagation