TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval

2026-01-14

AI Summary

Existing temporal question answering benchmarks are largely confined to simple factual queries and struggle to support the integration of complex temporal reasoning with multi-hop retrieval. To address this gap, this work proposes TEMPO, the first multi-domain benchmark that combines deep temporal reasoning—such as trend identification and cross-period comparison—with multi-hop retrieval. TEMPO encompasses 13 domains and 1,730 queries, accompanied by step-by-step retrieval plans and gold documents. The study introduces novel evaluation metrics, including Temporal Coverage@k and Temporal Precision@k, enabling fine-grained assessment of retrieval systems’ temporal reasoning capabilities. Experiments on 12 state-of-the-art systems reveal that even the best-performing model, DiVeR, achieves only 32.0 NDCG@10 and 71.4% Temporal Coverage@10, highlighting the significant challenge of retrieving temporally complete evidence.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data
• Planning, Routing, and Scheduling: Temporal Planning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Existing temporal QA benchmarks focus on simple fact-seeking queries from news corpora, while reasoning-intensive retrieval benchmarks lack temporal grounding. However, real-world information needs often require reasoning about temporal evolution and synthesizing evidence across time periods. We introduce TEMPO, the first benchmark combining temporal reasoning with reasoning-intensive retrieval across 13 domains. TEMPO features: (1) 1,730 complex queries requiring deep temporal reasoning such as tracking changes, identifying trends, or comparing cross-period evidence; (2) step-wise retrieval planning with 3,976 decomposed steps and gold documents mapped to each step for multi-hop evaluation; and (3) novel temporal metrics including Temporal Coverage@k and Temporal Precision@k measuring whether results span required time periods. Evaluation of 12 retrieval systems reveals substantial challenges: the best model (DiVeR) achieves only 32.0 NDCG@10 and 71.4\% Temporal Coverage@10, demonstrating difficulty in retrieving temporally complete evidence. We believe TEMPO provides a challenging benchmark for improving temporal reasoning in retrieval and RAG systems. Our code and data are available at https://github.com/tempo-bench/Tempo. See also our official website: https://tempo-bench.github.io/.

Problem

• temporal reasoning
• reasoning-intensive retrieval
• multi-domain benchmark
• temporal evolution
• evidence synthesis

Innovation

• temporal reasoning
• multi-hop retrieval
• temporal benchmark
• retrieval evaluation metrics
• reasoning-intensive QA