Reinforcing Multimodal Reasoning Against Visual Degradation

2026-05-09

AI Summary

This work addresses the significant performance degradation of multimodal large language models under real-world visual corruptions—such as blur, compression artifacts, and low resolution—and the susceptibility of existing reinforcement learning fine-tuning methods to reward contamination. To mitigate these issues, the authors propose ROMA, a novel framework featuring a dual forward-pass mechanism that evaluates degraded inputs using trajectories derived from clean counterparts, thereby avoiding unreliable generation paths. ROMA further incorporates a worst-case token-level KL divergence penalty to preserve distributional consistency, an auxiliary policy gradient loss weighted by advantages computed on clean images to ensure reward reliability, and correctness-conditioned regularization that applies constraints only on successful trajectories. Experiments on Qwen3-VL 4B/8B demonstrate that ROMA improves robustness by 2.4% and 2.3% on seen and unseen degradations across seven benchmarks, respectively, without compromising performance on clean inputs.

Abstract

Reinforcement Learning has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.

Problem

• visual degradation
• multimodal reasoning
• robustness
• multimodal large language models
• reinforcement learning

Innovation

• Multimodal Reasoning
• Visual Degradation Robustness
• Reinforcement Learning Fine-tuning
• Teacher Forcing
• Distributional Consistency