Rethinking Chain-of-Thought Reasoning for Videos

2025-12-10

AI Summary

Traditional chain-of-thought (CoT) reasoning for video understanding suffers from low efficiency due to reliance on lengthy inference chains and excessive visual tokens. Method: We propose a lightweight, annotation-free, and supervision-free framework that jointly optimizes visual token compression and short reasoning trajectory generation via light-weight post-training and inference enhancement of multimodal large language models. Contribution/Results: Challenging the prevailing assumption that long-chain CoT is indispensable, we empirically demonstrate that sparse visual inputs and concise reasoning paths suffice for effective video understanding. Our approach significantly accelerates inference—up to several-fold—while maintaining state-of-the-art performance across multiple video understanding benchmarks (e.g., Video-MME, NExT-QA, TGIF-QA). It exhibits strong generalization across diverse video domains and is highly deployment-friendly, requiring no task-specific fine-tuning or human-annotated rationales.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Chain-of-thought (CoT) reasoning has been highly successful in solving complex tasks in natural language processing, and recent multimodal large language models (MLLMs) have extended this paradigm to video reasoning. However, these models typically build on lengthy reasoning chains and large numbers of input visual tokens. Motivated by empirical observations from our benchmark study, we hypothesize that concise reasoning combined with a reduced set of visual tokens can be sufficient for effective video reasoning. To evaluate this hypothesis, we design and validate an efficient post-training and inference framework that enhances a video MLLM's reasoning capability. Our framework enables models to operate on compressed visual tokens and generate brief reasoning traces prior to answering. The resulting models achieve substantially improved inference efficiency, deliver competitive performance across diverse benchmarks, and avoid reliance on manual CoT annotations or supervised fine-tuning. Collectively, our results suggest that long, human-like CoT reasoning may not be necessary for general video reasoning, and that concise reasoning can be both effective and efficient. Our code will be released at https://github.com/LaVi-Lab/Rethink_CoT_Video.

Problem

• Enhance video reasoning efficiency with compressed visual tokens
• Reduce reliance on lengthy reasoning chains in video MLLMs
• Achieve competitive performance without manual CoT annotations

Innovation

• Compressed visual tokens for efficiency
• Brief reasoning traces before answering
• Post-training without manual annotations