SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

2026-07-15

AI Summary

Existing multimodal reinforcement learning methods rely on intervention types rather than actual outcomes for supervision, making it difficult to ensure that models reason based on visual evidence. This work proposes SIVA-RL, the first framework to decouple intervention construction from supervision assignment. By integrating PatchSwap local interventions, a frozen auditing policy, and a soft routing weight mechanism, SIVA-RL achieves sample-level, outcome-driven dynamic alignment: it adaptively weights sensitivity and invariance objectives based on post-intervention reward changes, while incorporating token alignment and distance constraints. The framework is compatible with both GRPO and DAPO training paradigms. Evaluated across nine benchmarks, SIVA-RL consistently outperforms existing approaches, achieving an 8.79 percentage point improvement on vision-dependent tasks and up to a 14.9% relative performance gain overall.

Abstract

Reinforcement learning with verifiable rewards (RLVR) drives multimodal reasoning, but answer-level correctness does not guarantee that a vision-language model grounds its predictions in visual evidence. Existing visual-intervention methods contrast policy behavior on original and modified images, yet assign supervision by the type of intervention rather than its observed effect. This assumption fails: identical operators produce heterogeneous outcomes across samples. We propose SIVA-RL, a Sensitivity-Invariance Visual Alignment framework that replaces operator-conditioned regularization with sample-wise, outcome-conditioned supervision. SIVA-RL constructs localized interventions through token-aligned, distance-constrained within-image PatchSwap. A frozen audit policy then scores each clean-intervention pair, and the observed reward drop becomes soft routing weights. Large-drop pairs drive sensitivity alignment, low-drop pairs drive clean-anchored invariance alignment, and ambiguous pairs are down-weighted. This design decouples intervention construction from supervision assignment and is compatible with both GRPO and DAPO backbones. Across nine multimodal reasoning benchmarks spanning mathematical, logical, and vision-dependent tasks, SIVA-RL improves 3B and 7B models over matched RL baselines in every setting. It yields an 8.79 percentage-point gain on vision-dependent reasoning and up to 14.9% relative overall improvement across all four GRPO- and DAPO-based configurations.

Problem

• multimodal reinforcement learning
• visual grounding
• visual intervention
• outcome-conditioned supervision
• sensitivity-invariance alignment

Innovation

• Sensitivity-Invariance Alignment
• Visual Intervention
• Outcome-Conditioned Supervision
• Multimodal Reinforcement Learning
• PatchSwap