Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

2026-09-07

AI Summary

研究通过强化学习方法训练多模态推理代理自我验证和过滤检索证据，减少对外部验证的依赖，提高工具使用效率。

Abstract

Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5{,}000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.

Problem

• Multimodal Reasoning
• Reinforcement Learning
• Tool Use
• Verification Signals

Innovation

• Reinforcement Learning
• Self-Verification
• Multimodal Reasoning
• Search-Aware Penalty
• Query-Diversity Reward