WILDTRACE: Benchmarking Natural Evidence Trails in Long-Context Reasoning

2026-07-10

AI Summary

Existing benchmarks for long-document reasoning struggle to evaluate models’ ability to integrate naturally scattered evidence within authentic texts and are often confounded by distributional biases. This work proposes WILDTRACE, a high-quality benchmark comprising 481 tasks derived from 214 real-world long documents—including accident reports and literary works. Its key innovation lies in the first systematic characterization of seven geometric structures of intra-source evidence, establishing an evidence-tracing paradigm grounded in each document’s intrinsic causal, temporal, and narrative logic. By integrating Pearl’s causal hierarchy, multi-hop reasoning taxonomies, document structure mining, and rigorous multi-stage human validation—assessing criteria such as clue necessity and answer traceability—the benchmark ensures high question quality and fills a critical gap in evaluating natural evidence integration, offering a more realistic standard for long-context reasoning.

Abstract

Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across distant passages. In an incident report, the operating condition, design flaw, and missed safety check that jointly explain a disaster may appear dozens of sections apart; in a novel, a character's true motive may surface only through scenes far removed from the moment it becomes relevant. This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it. Needle probes, planted facts, and reverse-engineered multi-hop chains embed evidence that may differ from the host text in distribution, placement, or register, making it unclear whether strong performance reflects genuine source reasoning or distributional artifacts. We introduce WILDTRACE, a benchmark of 481 tasks over 214 naturally occurring long-form sources such as technical incident reports and lesser-known literary narratives, where all evidence trails arise from the document's own causal, temporal, and narrative logic. Drawing on Pearl's causal hierarchy and prior multi-hop reasoning typologies, we define seven source-internal evidence geometries that characterize the distinct relational demands of analytical reading in long documents. A source-first construction pipeline mines candidate trails from document structure before writing questions; each item then undergoes multi-stage validation covering clue necessity, answer groundedness, rubric fidelity, contamination resistance and answerability. As models are increasingly entrusted with real-world high-stakes analytical tasks, this gap between accessing information and reasoning over naturally dispersed evidence emerges as a defining challenge for the next stage of long-context research.

Problem

• long-context reasoning
• natural evidence integration
• source-internal reasoning
• evidence dispersion
• analytical reading

Innovation

• long-context reasoning
• natural evidence trails
• source-internal reasoning
• causal hierarchy
• multi-hop QA benchmark