On the Temporal Question-Answering Capabilities of Large Language Models Over Anonymized Data

2025-04-10

AI Summary

This study investigates the generalization and reasoning capabilities of large language models (LLMs) on anonymized, structured/semi-structured time-series data—domains absent from their training corpora—where existing benchmarks lack coverage. Method: We introduce RATA, the first dedicated benchmark comprising 17 natural-language time-series reasoning tasks, and propose a temporal capability evaluation framework for anonymized data. Our approach integrates Tree-of-Thought reasoning, reflexive self-reflection, executable code generation, and hierarchical task decomposition into a unified inference architecture. Contribution/Results: Experiments reveal that vanilla LLMs exhibit poor robustness and limited generalization on unseen time-series scenarios. In contrast, our integrated architecture achieves substantial gains in both accuracy and inference stability. This work establishes an empirically grounded, scalable technical paradigm and benchmark for trustworthy time-series AI in privacy-sensitive applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The applicability of Large Language Models (LLMs) in temporal reasoning tasks over data that is not present during training is still a field that remains to be explored. In this paper we work on this topic, focusing on structured and semi-structured anonymized data. We not only develop a direct LLM pipeline, but also compare various methodologies and conduct an in-depth analysis. We identified and examined seventeen common temporal reasoning tasks in natural language, focusing on their algorithmic components. To assess LLM performance, we created the extit{Reasoning and Answering Temporal Ability} dataset (RATA), featuring semi-structured anonymized data to ensure reliance on reasoning rather than on prior knowledge. We compared several methodologies, involving SoTA techniques such as Tree-of-Thought, self-reflexion and code execution, tuned specifically for this scenario. Our results suggest that achieving scalable and reliable solutions requires more than just standalone LLMs, highlighting the need for integrated approaches.

Problem

• Assessing LLMs' temporal reasoning on anonymized untrained data
• Comparing methodologies for temporal question-answering tasks
• Evaluating integrated approaches for scalable reliable solutions

Innovation

• Developed direct LLM pipeline for temporal reasoning
• Created RATA dataset with anonymized semi-structured data
• Compared SoTA techniques like Tree-of-Thought, self-reflexion