MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG

2026-07-08

AI Summary

Existing multimodal retrieval-augmented generation (mRAG) approaches are prone to introducing factually incorrect distractors when the knowledge base contains numerous visually similar entities, and their reliance on fixed candidate sets often leads to error propagation. To address these limitations, this work proposes MMAgent-R², an agent-based mRAG framework equipped with visual re-ranking and an active rejection mechanism. The framework enhances retrieval accuracy through fine-grained direct image comparison, dynamically expands the candidate set when confidence is low, and integrates an internal verification module to jointly optimize retrieval, validation, and generation. Trained with a stepwise verification reward function and GRPO optimization, MMAgent-R² achieves state-of-the-art performance on InfoSeek, E-VQA, and MMhops benchmarks, demonstrating particularly strong capabilities in challenging retrieval scenarios and multi-image, multi-hop reasoning tasks.

Abstract

Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions. Existing multimodal Retrieval Augmented Generation (mRAG) methods rely on global visual features to match candidate entities, yet when the knowledge base contains numerous visually similar entities, the retriever struggles to distinguish them, populating the candidate set with visually similar but factually mismatched distractors. Since subsequent processing steps such as noise filtering are also confined to this fixed candidate set, errors from failed retrieval inevitably propagate to the final answer. To address these challenges, we propose MMAgent-R$^2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism. Visual reranking directly compares query and candidate images, capturing discriminative details beyond textual descriptions to precisely identify the target entity among similar candidates; active rejection discards unreliable results and retrieves additional candidates when no confident match is found, moving beyond the fixed candidate pool. We design a composite reward function with step-level verification rewards and achieve joint optimization of external retrieval, internal verification, and answer generation via GRPO training. Experiments on InfoSeek, E-VQA, and MMhops demonstrate that \ours{} achieves state-of-the-art performance, with particularly notable advantages in challenging retrieval scenarios and complex multi-image multi-hop reasoning tasks.

Problem

• Knowledge-based Visual Question Answering
• multimodal Retrieval Augmented Generation
• visually similar entities
• retrieval errors
• candidate set

Innovation

• visual reranking
• active rejection
• agentic mRAG
• GRPO training
• multimodal retrieval