When Does a Video-Language Model Stop Watching? Reward Strength Controls the Formation and Reversal of Visual Shortcuts in Multimodal RLVR

2026-06-20

AI Summary

This work addresses the tendency of vision-language models in multimodal reinforcement learning to rely on linguistic priors while neglecting visual inputs—a phenomenon termed “visual shortcuts.” The authors propose dynamically modulating the formation and reversal of such shortcuts during training via an adjustable visual grounding penalty strength, λ. They systematically reveal, for the first time, that visual shortcuts exhibit emergent behavior, dose-response monotonicity, and a critical intervention window. Notably, shortcut mitigation displays temporal asymmetry and hysteresis: penalization is effective only when applied prior to shortcut formation. By integrating reinforcement learning with verifiable rewards (RLVR), visual grounding regularization, and out-of-distribution evaluation, the approach reframes shortcut collapse as a controllable, time-varying process.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is increasingly applied to large vision-language models (LVLMs), yet outcome-only optimization can drive a model to stop attending to the video and instead exploit linguistic priors -- a failure we call a visual shortcut. While the existence of such perception bypass is by now documented, how it forms, whether it can be undone, and when intervention still helps remain open. We treat the strength of a grounding penalty, lambda, as a control knob and characterize the formation-reversal dynamics of visual shortcuts along the training time axis. On a held-out, out-of-distribution diagnostic set, we find: (i) a sharp onset -- shortcut reliance emerges abruptly over a narrow window of optimization steps and is robust across random seeds; (ii) a monotone dose-response -- increasing lambda progressively suppresses the shortcut, and at an intermediate dose the trajectory first forms and then reverses the shortcut, exposing a hysteresis-like asymmetry between acquiring and removing it; and (iii) a critical intervention window -- applying the penalty before onset arrests shortcut formation, whereas the same penalty applied after consolidation is markedly less effective. Together these results recast visual-shortcut collapse not as a binary defect but as a controllable, time-dependent, and asymmetric process, with direct implications for when and how strongly to regularize multimodal RLVR.

Problem

• visual shortcut
• multimodal RLVR
• reinforcement learning
• vision-language models
• grounding penalty

Innovation

• visual shortcut
• reinforcement learning with verifiable rewards
• grounding penalty
• multimodal learning
• hysteresis dynamics