CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMs

2026-06-18

AI Summary

Existing reinforcement learning–based approaches for multimodal video reasoning often employ fixed or oversimplified strategies for controlling reasoning length, struggling to balance thorough exploration with computational efficiency. This work proposes CARE, a novel framework that introduces, for the first time, a capability-aware reward shaping mechanism. CARE dynamically estimates model proficiency via exponential moving average and adaptively shifts reward preferences across training stages—initially encouraging longer exploratory reasoning and gradually favoring concise, efficient inference. To prevent misinterpreting verbosity as complexity, it further incorporates batch-level reasoning effort normalization and a posterior amplification strategy for historically challenging instances. Evaluated on multiple video understanding benchmarks, CARE achieves substantial gains in both accuracy and token efficiency, exhibiting a characteristic inverted U-shaped trajectory of reasoning length during training and ultimately producing shorter yet more informative reasoning paths.

Abstract

In multimodal video reasoning, reinforcement learning-based methods typically rely on simplistic and inflexible reasoning-length control strategies that fail to adapt to the model's evolving competence. This mismatch may suppress necessary exploration at early stages, while encouraging redundant reasoning and inefficient decoding once the model becomes more competent. In this paper, we propose CARE, a competence-aware reward shaping framework for adaptive reasoning length optimization in multimodal reasoning. Specifically, CARE maintains a smoothed competence estimate via an exponential moving average of pass rates, and uses it to route training into progressive stages that shift the reward preference from exploration-oriented long-form reasoning to efficiency-oriented concise reasoning. To avoid conflating verbosity with intrinsic task complexity, CARE further normalizes reasoning effort with batch-level statistics, and introduces a posterior amplifier to strengthen reward signals for unexpectedly strong performance on historically difficult samples. The proposed mechanism is seamlessly integrated into the GRPO training pipeline and incurs no additional inference-time overhead. Extensive experiments on multiple video reasoning and general video understanding benchmarks demonstrate that CARE consistently improves reasoning accuracy, stabilizes reinforcement learning, and significantly enhances token efficiency. Moreover, CARE exhibits a characteristic inverted-U trajectory of reasoning length during training, and yields shorter yet more informative reasoning traces at convergence, indicating effective adaptive allocation of reasoning budget. We provide the source code for our proposed CARE framework and experiments at https://github.com/1Pansy/Video-CARE.

Problem

• reasoning length
• video reasoning
• reinforcement learning
• adaptivity
• multimodal reasoning

Innovation

• reward shaping
• adaptive reasoning length
• competence-aware learning
• video-MLLMs
• token efficiency