Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning

2026-09-09

AI Summary

为解决视觉-语言模型在无标签数据上自动生成学习信号不可靠的问题，提出TTIQ框架，通过利用图像-问题依赖性提高模型适应能力。

Abstract

Test-time reinforcement learning can adapt vision-language models (VLMs) to unlabeled target data, but its effectiveness is fundamentally limited by the reliability of self-generated learning signals. To assess the reliability of consensus-based learning signals, we analyze VLM test-time reinforcement learning across diverse VQA datasets and model sizes, revealing two limitations. First, gains from consensus-based test-time training largely come from answer normalization rather than content correction. Second, many initial VLM responses are incorrect due to the model's limited ability to jointly use the image and the question; consensus rewards derived from these outputs may preserve the resulting grounding errors rather than correct them. Motivated by these, we propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation. TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants, using the resulting token-level likelihood changes to estimate dependence on each input. It combines image and question dependence with calibrated confidence to construct a response-level reward that favors jointly grounded responses, and uses the token-level signals to assign greater positive policy credit to tokens supported by both inputs. This design favors responses that are jointly grounded in the image and the question and sufficiently confident, rather than merely popular. Experiments across eight VQA datasets and multiple VLM sizes show that TTIQ achieves the best average performance at every model scale. It further generalizes across VLM families, while models trained on one dataset improve performance on unseen datasets without further training.

Problem

• test-time reinforcement learning
• vision-language models
• consensus-based learning signals
• answer normalization
• content correction

Innovation

• Test-time Reinforcement Learning
• Image-Question Dependence
• Visual-Language Models
• Token-Level Likelihood Changes
• Jointly Grounded Responses