DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents

2026-04-21

AI Summary

This work addresses the premature termination of multimodal search agents caused by terminal rewards that fail to distinguish between exploratory trajectories and redundant contextual interference. To mitigate this, the authors propose an advantage estimation mechanism based on whole-trajectory structural similarity, coupled with a differentiated Gaussian reward scheme that dynamically adjusts interaction tolerance to encourage diverse reasoning path lengths. The approach is integrated within a multimodal tool-calling framework and refined through multi-round reinforcement learning to support deep reasoning. Evaluated on a newly curated dataset comprising 3,602 multi-step question-answer pairs, the method achieves state-of-the-art performance, outperforming MMSearch-R1 by 8.4% on the FVQA-test benchmark.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Agentic multimodal models have garnered significant attention for their ability to leverage external tools to tackle complex tasks. However, it is observed that such agents often meet premature interaction collapse, caused by two primary reasons: 1) the terminal reward often appending on the last token prevents the advantage from distinguishing trajectories with exploratory behavior; 2) excessively redundant context hinders the agent from absorbing useful feedback. To address these issues, we propose the Deepening Reasoning MMSearchAgent, the framework leverages the structural proximity to derive advantage signals from the whole rollout trajectories in an entire batch, such that trajectories of different lengths are further encouraged to be generated, even when containing the same correct answer. Additionally, differentiated gaussian rewards are employed to dynamically calibrate interaction tolerance, thereby ensuring information reliability and reduce redundancy. To support multi-turn interaction training, we have constructed a multi-step deep-reasoning dataset including 3602 high-quality QA pair with at least 3 reasonning steps. Extensive experiments demonstrate that our method achieves state-of-the-art performance, outperforming the MMSearch-R1 by 8.4$\%$ on FVQA-test.

Problem

• multimodal search agents
• premature interaction collapse
• terminal reward
• redundant context
• trajectory differentiation

Innovation

• Deepening Reasoning
• Multimodal Search Agent
• Advantage Signal from Trajectories
• Differentiated Gaussian Rewards
• Multi-step Reasoning Dataset