Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus

2026-06-13

AI Summary

Existing evaluations of deep research agents are largely confined to monolingual settings, failing to capture their true capabilities when query and supporting evidence languages mismatch. This work proposes XBCP, the first cross-lingual web-browsing evaluation benchmark, which systematically disentangles cross-lingual from multilingual scenarios by fixing English question-answer pairs while varying the language of supporting documents. The study comprehensively assesses agent performance in retrieval and reasoning using multiple metrics—including answer accuracy, evidence recall, calibration, citation fidelity, and oracle retrieval—combined with both sparse and dense multilingual retrievers. Results reveal a significant performance drop even with strong dense retrievers under cross-lingual evidence conditions; notably, answer accuracy remains low even when all gold evidence is provided, exposing fundamental challenges in evidence integration under linguistic mismatch.

Abstract

Deep research agents are increasingly evaluated on their ability to search for evidence, reason over retrieved sources, and produce grounded answers. Existing browsing benchmarks, however, largely assume that the user's query and the supporting evidence are written in the same language, leaving open whether agentic search systems can operate when relevant evidence appears in another language. We introduce XBCP (Cross-lingual BrowseComp-Plus), a controlled benchmark that preserves the English question-and-answer space of BrowseComp-Plus but varies the languages of the supporting documents. XBCP instantiates two complementary settings: in the cross-lingual setting, each query is paired with evidence in a single assigned language. In the multilingual setting, the full evidence corpus is distributed equally and randomly across 12 languages spanning high-resource and low-resource regimes. We evaluate four deep research agents using sparse and dense multilingual retrievers, measuring answer accuracy, evidence recall, search behavior, calibration, citation fidelity, and oracle retrieval. Results reveal substantial degradation when evidence is translated. Even strong, dense retrievers lose evidence recall, and agents become less calibrated and cite evidence less reliably. Notably, accuracy remains lower even when all gold evidence is supplied directly. These findings suggest that cross-lingual deep research exposes both retrieval failures and an independent, agent-side difficulty in integrating language-mismatched evidence.

Problem

• cross-lingual
• deep research agents
• multilingual retrieval
• evidence integration
• language mismatch

Innovation

• cross-lingual retrieval
• deep research agents
• multilingual benchmarks
• evidence integration
• language mismatch