A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

2026-07-06

AI Summary

Current evaluations of large reasoning models (LRMs) predominantly rely on static datasets or synthetic tasks lacking fine-grained control over difficulty, and they typically assess only final answers, thereby failing to accurately measure temporal reasoning capabilities. This work proposes TRACE, a novel framework that, for the first time, formalizes temporal reasoning as a constraint satisfaction problem grounded in Allen’s interval algebra, enabling precise modulation of logical complexity. TRACE further introduces a reasoning-trajectory-based verification mechanism to dynamically generate test instances with calibrated difficulty levels. The resulting TRACEBench benchmark—comprising 1,200 instances—demonstrates a strong negative correlation (r ≈ −0.96) between task difficulty and model performance, reveals a “false guessing” rate as high as 28% in medium-scale models, and diagnoses characteristic failure modes such as degenerative loops in small models and reasoning explosions in large ones.

Abstract

Defining the reasoning boundaries and ensuring the reliability of Large Reasoning Models (LRMs) remains a critical challenge. Current benchmarks primarily rely on static datasets susceptible to data contamination or synthetic tasks lacking fine-grained difficulty control. Furthermore, standard outcome-based evaluations often conceal reasoning flaws by neglecting the reasoning process. To address these limitations, we introduce TRACE, a testing framework that models temporal reasoning as constraint satisfaction problems via Allen's Interval Algebra. This approach enables precise regulation of logical complexity and incorporates a Trace-Based Verification Oracle to validate reasoning faithfulness. Using this framework, we construct TRACEBench, an extensive benchmark comprising 1,200 synthesized test instances across graded difficulty levels. We employ TRACE to evaluate eight widely used LRMs on TRACEBench. The results confirm a strong negative correlation between model performance and our difficulty metric (Pearson's r approximately -0.96), validating the effectiveness of our difficulty control mechanism. Moreover, our trace-based analysis exposes significant discrepancies between reasoning validity and final answers, revealing a high spurious guessing rate of approximately 28% in mid-sized models. In addition, we diagnose scale-dependent failure modes, ranging from Degenerative Loops in small models to Reasoning Explosion in advanced architectures. TRACE thus provides a robust, automated platform for benchmarking the true temporal reasoning capabilities of LRMs.

Problem

• Temporal Reasoning
• Large Reasoning Models
• Benchmarking
• Difficulty Control
• Reasoning Faithfulness

Innovation

• Temporal Reasoning
• Constraint Satisfaction
• Trace-Based Verification
• Difficulty-Controlled Benchmarking
• Large Reasoning Models