SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning

2026-07-12

AI Summary

Current multimodal large language models lack effective self-verification and self-correction mechanisms during inference, limiting their ability to improve accuracy without external supervision. This work proposes SVR-R1, a novel framework that integrates binary self-verification (Yes/No) at inference time with reinforcement learning, enabling the model to autonomously decide—across multiple reasoning rounds—whether to reconsider its answer and to construct reward signals based on its final output. Leveraging the GRPO algorithm and asynchronous multi-round rollouts, SVR-R1 achieves end-to-end training without requiring auxiliary discriminators or external annotations. Experiments demonstrate that SVR-R1 significantly outperforms standard GRPO baselines on vision-language reasoning benchmarks, yielding substantial gains in accuracy while reducing verification rounds during later training stages, thereby validating the model’s capacity to internalize confident responding and self-correction.

Abstract

We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning. For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No). A 'No' triggers a second-chance rethink; a 'Yes,' or a turn cap, finalizes the output for computing the outcome-based reward. SVR-R1 is implemented with GRPO and an asynchronous multi-turn rollout framework and needs no external supervision or auxiliary critics. We evaluate SVR-R1 on vision-language reasoning benchmarks and show that it improves accuracy by a large margin over strong standard GRPO baselines. Training dynamics show decreasing reliance on verification-fewer verification turns, yet higher test accuracy-indicating that the gap between verification and generation narrows as the policy internalizes self-correction and chooses the most confident answer via our framework. SVR-R1 bridges the less explored intersection of inference-time self-refinement and RL training for VLMs, offering a simple yet effective recipe for bootstrapping multimodal reasoning. We will open-source \textbf{SVR-R1} to facilitate future research in VLMs.

Problem

• multimodal reasoning
• self-verification
• reinforcement learning
• vision-language models
• self-correction

Innovation

• Self-verification
• Multi-modal reasoning
• Reinforcement Learning
• Vision-Language Models
• GRPO