SeekerGym: A Benchmark for Reliable Information Seeking

2026-04-18

AI Summary

Current AI-powered information retrieval systems lack mechanisms to ensure informational completeness, rendering users susceptible to implicit biases. This work proposes SeekerGym, a novel benchmark that systematically evaluates the completeness of relevant passage retrieval by AI agents within Wikipedia and machine learning survey literature. The benchmark leverages document structure to formulate retrieval tasks and integrates both completeness metrics and an uncertainty quantification mechanism for missing information. Experimental results reveal that state-of-the-art models retrieve only 42.5% and 29.2% of relevant content in these domains, respectively, underscoring the significant challenge posed by incomplete retrieval and highlighting substantial room for improvement.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Despite their substantial successes, AI agents continue to face fundamental challenges in terms of trustworthiness. Consider deep research agents, tasked with searching for information relevant to a given topic-while AI agents can perform effective information retrieval, there is little guarantee regarding the completeness of this information. Gaps in retrieved information can leave biases that mislead users even if the information they are given is correct and relevant. We introduce SeekerGym, a benchmark designed to evaluate the completeness of information retrieved by AI agents. In addition, SeekerGym also measures how well agents quantify their uncertainty in the completeness of their information; if an agent fails to retrieve all relevant information, it is useful for it to at least quantify how much might be missing. At a high level, each task in SeekerGym is a document (e.g., a Wikipedia article), and the AI agent must issue queries to retrieve passages from that document. Intuitively, the document comprehensively covers a topic, so the ability to retrieve its sections directly measures completeness of information retrieval. In addition to Wikipedia, we also consider machine learning survey papers, where the goal is to retrieve relevant sections of a survey paper. We benchmark several models and algorithms; the best approaches retrieve 42.5% of passages on Wikipedia and 29.2% on ML Surveys, leaving substantial room for improvement.

Problem

• information seeking
• completeness
• AI agents
• uncertainty quantification
• retrieval benchmark

Innovation

• information completeness
• uncertainty quantification
• AI agent benchmarking
• reliable information seeking
• retrieval evaluation