ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

2026-06-26

AI Summary

This work addresses the challenge of knowledge-based visual question answering (KB-VQA), which requires integrating visual understanding with external knowledge. Existing approaches typically follow a fixed retrieve-then-generate pipeline and lack adaptivity during reasoning. To overcome this limitation, the authors propose ProMSA—a progressive multimodal search agent that dynamically selects among image retrieval, text retrieval, and halting actions through an iterative decision process. The method incorporates retrieval deduplication and tool-call budgeting to enable efficient reasoning. ProMSA is trained via rejection-sampled supervised fine-tuning combined with TN-GSPO, a sequence-level reinforcement learning objective that jointly optimizes reasoning depth and answer length. Experiments on E-VQA and InfoSeek demonstrate substantial improvements over strong RAG and agent-based baselines, achieving higher retrieval quality and end-to-end VQA accuracy.

Abstract

Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during reasoning. We propose ProMSA, a progressive multimodal search agent for KB-VQA. Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.

Problem

• Knowledge-based Visual Question Answering
• adaptive retrieval
• fixed pipeline
• reasoning
• external knowledge

Innovation

• progressive multimodal agent
• adaptive retrieval
• tool-call budget
• TN-GSPO
• knowledge-based VQA