Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

2026-06-29

AI Summary

This work addresses the susceptibility of vision-language models to linguistic priors in reinforcement learning, which often leads to fluent yet visually ungrounded and unfaithful reasoning. To mitigate this issue, the authors propose a Faithful Warm-Start strategy: first, they construct FaithfulQA, a dataset that explicitly models the causal relationship between visual inputs and language outputs, and employ a vision-language model as a judge to filter out unfaithful samples; then, prior to reinforcement learning, they conduct a visual faithfulness pre-training phase to steer the model toward causally consistent multimodal reasoning. This approach substantially reduces hallucinated, vision-unsupported reasoning, improves answer accuracy, and stabilizes the reinforcement learning training process.

Abstract

Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.

Problem

• Vision-Language Models
• Reinforcement Learning
• Visual Grounding
• Reasoning Faithfulness
• Multimodal Reasoning

Innovation

• Faithful Warm-Start
• Vision-Language Models
• Reinforcement Learning
• Visual Grounding
• Causal Consistency