SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulation

2026-07-07

AI Summary

This work addresses the challenges faced by multimodal search agents in multi-hop reasoning—namely, metadata loss, non-reproducible environments, and sparse rewards—stemming from fragmented training data, environments, and reward signals. To overcome these issues, the authors propose a unified framework that constructs a reproducible simulated search world grounded in a typed knowledge graph. Central to this framework is the Perception-Knowledge Chain (PKC), which samples constrained multi-hop paths at the intersection of visual and knowledge modalities while preserving step-level entity metadata as self-contained reward anchors. Furthermore, they introduce Hop-Anchored Policy Optimization (HaPO), a training strategy that obviates the need for auxiliary process-based reward models. Evaluated across six knowledge-intensive multimodal benchmarks, the method achieves state-of-the-art performance among open-source agents, with SearchEyes-27B outperforming the strongest baseline by an average of 6.2 points.

Abstract

Training multimodal search agents to perform multi-hop reasoning remains challenging due to a fundamental structural disconnect: existing pipelines construct training data, search environments, and reward signals independently, causing synthesized structural metadata to be discarded, environments to rely on irreproducible external engines, and RL rewards to remain sparse at the trajectory level. We present \textbf{SearchEyes}, which uses a typed knowledge graph as the backbone of a \emph{simulated search world} that unifies all three components. We propose \textbf{Perception-Knowledge Chains (PKC)} to sample constrained multi-hop paths over the visual-knowledge intersection of Wikidata5M, retaining hop-level entity metadata that simultaneously defines a self-contained search world and step-level reward anchors. We further propose \textbf{Hop-Anchored Policy Optimization (HaPO)}, which reuses these anchors for step-level credit assignment without a separately trained process reward model. Experiments on six multimodal knowledge-intensive benchmarks show that SearchEyes achieves state-of-the-art performance among open-source multimodal search agents, with SearchEyes-27B improving over the strongest open-source baseline by 6.2 points on average.%

Problem

• multimodal search
• multi-hop reasoning
• reward sparsity
• search environment
• structural metadata

Innovation

• multimodal search
• simulated search world
• Perception-Knowledge Chains
• Hop-Anchored Policy Optimization
• knowledge graph