AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning

2026-06-23

AI Summary

This work addresses the challenges large language models face in real-world workplace settings when processing massive, disorganized document collections—specifically, difficulties in evidence retrieval, inconsistencies in terminology, units, and temporal references, and the need for cross-document reasoning. To this end, the authors propose a document reasoning benchmark that emphasizes archival grounding, agent-driven active exploration, and broad cross-domain coverage. Leveraging an agent-based task synthesis pipeline, text obfuscation for anonymization, and rigorous difficulty filtering, they construct a high-quality evaluation suite comprising 362 questions spanning eight domains, built upon 9,664 documents totaling 372 million tokens. Empirical evaluation reveals that even the strongest current models achieve only 59.4% accuracy, underscoring both the inherent difficulty of the task and the effectiveness of the proposed benchmark.

Abstract

Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer. Existing benchmarks address only parts of this setting and none jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage. We introduce Agora, a benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens, far exceeding any model's context window, so agents must explore deliberately rather than scan exhaustively. Agora is built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering. Evaluating eight models, we find the task far from solved: even the strongest reaches only 59.4% accuracy, with notable variation across domains.

Problem

• archive-grounded reasoning
• agentic exploration
• document reasoning
• cross-domain coverage
• workplace documents

Innovation

• archive-grounded reasoning
• agentic exploration
• document reasoning benchmark
• cross-domain evaluation
• large language model agents