Breaking the Evaluation Paradox: Evaluating High-Entropy Search with Computationally Irreducible Constraints

2026-06-21

AI Summary

Existing evaluation methods struggle to effectively measure large language models’ systematic exploration capabilities in high-entropy exhaustive search tasks, as human annotations cannot cover the complete ground truth and may erroneously penalize models that surpass annotators. This work proposes VERITAS, a novel evaluation framework that introduces a paradigm grounded in computationally irreducible constraints. By leveraging a hash-based verification mechanism and automated test-case generation, VERITAS constructs an unbounded set of verifiable, sparse, and difficulty-controllable test instances. This approach ensures tasks resist heuristic optimization while remaining efficiently verifiable, thereby overcoming the limitations of traditional methods that rely on incomplete human labels or LLM-based judges. The framework enables precise assessment of high-entropy search proficiency and supports scalable generation of training data, significantly enhancing model performance on complex exploration tasks.

Abstract

Evaluating the exhaustive search capabilities of large language models (LLMs) is plagued by a fundamental paradox: verifying completeness requires complete ground truth, yet high-entropy enumeration tasks make such ground truth impossible for humans to create. This causes benchmarks to systematically penalize models for outperforming their human annotators. Despite rapid progress in web-search and deep research agents -- which now issue hundreds of queries, traverse diverse sites, and synthesize long reports -- evaluation still largely relies on partially annotated answer sets, LLM-based judges, or single-answer questions that avoid genuinely exhaustive search scenarios. We break this paradox by shifting the evaluation paradigm from simulating a messy reality to constructing computationally pure challenges. We introduce VERITAS (Verifiable Traversal Assessment for Search), a framework built on the principle of computationally irreducible constraints. By introducing novel, non-optimizable constraints, we create verifiable, sparse-answer search tasks that are computationally equivalent to exhaustive enumeration. These constraints are easy to verify but impossible for LLMs or search engines to optimize, forcing agents to genuinely traverse the entire search space. VERITAS can automatically generate a virtually infinite number of test cases with perfect ground truth and precise difficulty control, with marginal instance cost dominated by hash computations. This provides not only a robust benchmark for evaluating systematic exploration under uncertainty but also a scalable method for generating training data to improve these crucial, yet underdeveloped, capabilities.

Problem

• evaluation paradox
• high-entropy search
• computationally irreducible constraints
• exhaustive enumeration
• ground truth

Innovation

• computationally irreducible constraints
• exhaustive search evaluation
• VERITAS
• high-entropy enumeration
• verifiable sparse-answer tasks