Hybrid Retriever Evolution for Multimodal Document Reasoning Agents

2026-06-28

AI Summary

Existing multimodal document understanding systems rely on fixed retrieval pipelines and struggle to dynamically coordinate heterogeneous retrievers for step-by-step reasoning. This work proposes a failure-driven evolutionary framework in which a meta-agent analyzes erroneous reasoning trajectories, diagnoses underlying issues, and iteratively rewrites instructions for task agents, thereby modeling retrieval coordination as a learnable reasoning component. The approach enables, for the first time, the evolution of dynamic cross-modal and cross-page retrieval strategies coupled with evidence fusion, integrating lexical, semantic, and multimodal retrievers through multi-agent collaboration, failure analysis, and adaptive routing to enhance reasoning capabilities. Evaluated on MMLongBench-Doc and DocBench, the method achieves gains of up to 19.6 points over strong baselines, significantly outperforming recent systems such as MACT and MDocAgent.

Abstract

Different retrievers, including lexical, semantic, and multimodal approaches, provide highly complementary strengths for multimodal document understanding, yet most systems combine them through fixed pipelines that cannot adapt to the demands of individual reasoning steps. In this work, we ask whether retrieval orchestration itself can be learned as part of the reasoning process. We introduce a failure-driven evolution framework in which a meta-agent autonomously discovers how a tool-using task agent should coordinate diverse retrievers during multi-step document question answering. The meta-agent analyzes incorrect reasoning trajectories, actively probes the same tool environment to diagnose root causes, and iteratively rewrites the task agent's instructions, turning retrieval from a fixed front-end stage into an adaptive, step-wise reasoning decision. The evolved agent learns when to invoke each retriever, how to combine them, and how to compose evidence across modalities and pages. On MMLongBench-Doc and DocBench, the evolved agent achieves gains of up to +19.6 points over the unevolved baseline and consistently outperforms recent systems including MACT, MDocAgent, and SimpleDoc. Detailed retrieval analyses confirm that these improvements arise from adaptive routing and evidence composition rather than reliance on any hard coded retrieval mode, and evolution dynamics reveal a progressive shift from narrow lexical behavior to rich multi-tool coordination. These findings establish autonomous multi-agent coordination as a promising paradigm for multimodal document reasoning.

Problem

• multimodal document reasoning
• retriever orchestration
• adaptive retrieval
• multi-step question answering
• tool coordination

Innovation

• retrieval orchestration
• failure-driven evolution
• multimodal reasoning
• meta-agent
• adaptive retrieval