Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1

2025-03-31

AI Summary

This work addresses the lack of systematic evaluation for video understanding in multimodal large language models (MLLMs), particularly the unclear impact of reinforcement learning (RL) on jointly enhancing perception and reasoning. We propose SEED-Bench-R1, the first benchmark specifically designed for video understanding, coupled with a novel three-tier generalization evaluation framework—assessing performance under in-distribution, cross-environment, and cross-environment-task settings. Leveraging Qwen2-VL-Instruct-7B, we conduct comparative training via proximal policy optimization (PPO)-based RL and supervised fine-tuning (SFT), integrating multi-stage reward modeling and a large-scale, verifiable video question-answering dataset. Experiments demonstrate that RL substantially outperforms SFT on SEED-Bench-R1 and even surpasses it on general benchmarks such as LongVideoBench. We further identify a new phenomenon: RL improves visual perception yet risks breaking logical reasoning chains. Results confirm RL’s superior data efficiency and stronger cross-scenario generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent advancements in Chain of Thought (COT) generation have significantly improved the reasoning capabilities of Large Language Models (LLMs), with reinforcement learning (RL) emerging as an effective post-training approach. Multimodal Large Language Models (MLLMs) inherit this reasoning potential but remain underexplored in tasks requiring both perception and logical reasoning. To address this, we introduce SEED-Bench-R1, a benchmark designed to systematically evaluate post-training methods for MLLMs in video understanding. It includes intricate real-world videos and complex everyday planning tasks in the format of multiple-choice questions, requiring sophisticated perception and reasoning. SEED-Bench-R1 assesses generalization through a three-level hierarchy: in-distribution, cross-environment, and cross-environment-task scenarios, equipped with a large-scale training dataset with easily verifiable ground-truth answers. Using Qwen2-VL-Instruct-7B as a base model, we compare RL with supervised fine-tuning (SFT), demonstrating RL's data efficiency and superior performance on both in-distribution and out-of-distribution tasks, even outperforming SFT on general video understanding benchmarks like LongVideoBench. Our detailed analysis reveals that RL enhances visual perception but often produces less logically coherent reasoning chains. We identify key limitations such as inconsistent reasoning and overlooked visual cues, and suggest future improvements in base model reasoning, reward modeling, and RL robustness against noisy signals.

Problem

• Evaluating reinforcement learning for video understanding in MLLMs
• Assessing generalization in perception and reasoning tasks
• Identifying limitations in RL's logical coherence and visual cues

Innovation

• Reinforcement learning enhances video understanding performance
• SEED-Bench-R1 evaluates MLLMs with complex video tasks
• RL improves perception but needs better reasoning coherence