Video-ToC: Video Tree-of-Cue Reasoning

2026-04-22

AI Summary

Existing video large language models suffer from insufficient reasoning capabilities in complex video understanding, a tendency to generate hallucinations, and a lack of perceptual adaptability to input content. To address these limitations, this work proposes the Video-ToC framework, which introduces a tree-structured chain-of-thought reasoning mechanism that enables perception-driven dynamic inference through structured visual cue localization, an on-demand dynamic reward scheme, and an automated data construction pipeline. To support training, we introduce two new datasets—Video-ToC-SFT-1k for supervised fine-tuning and Video-ToC-RL-2k for reinforcement learning. Experimental results demonstrate that our approach significantly outperforms current state-of-the-art methods across six video understanding benchmarks and one hallucination evaluation benchmark, confirming its effectiveness and robustness.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Existing Video Large Language Models (Video LLMs) struggle with complex video understanding, exhibiting limited reasoning capabilities and potential hallucinations. In particular, these methods tend to perform reasoning solely relying on the pretrained inherent reasoning rationales whilst lacking perception-aware adaptation to the input video content. To address this, we propose \textbf{Video-ToC}, a novel video reasoning framework that enhances video understanding through tree-of-cue reasoning. Specifically, our approach introduces three key innovations: (1) A tree-guided visual cue localization mechanism, which endows the model with enhanced fine-grained perceptual capabilities through structured reasoning patterns; (2) A reasoning-demand reward mechanism, which dynamically adjusts the reward value for reinforcement learning (RL) based on the estimation of reasoning demands, enabling on-demand incentives for more effective reasoning strategies; and (3) An automated annotation pipeline that constructs the Video-ToC-SFT-1k and Video-ToC-RL-2k datasets for supervised fine-tuning (SFT) and RL training, respectively. Extensive evaluations on six video understanding benchmarks and a video hallucination benchmark demonstrate the superiority of Video-ToC over baselines and recent methods. Code is available at https://github.com/qizhongtan/Video-ToC.

Problem

• Video Large Language Models
• complex video understanding
• reasoning capabilities
• hallucinations
• perception-aware adaptation

Innovation

• tree-of-cue reasoning
• visual cue localization
• reasoning-demand reward
• video large language models
• reinforcement learning