TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale

2026-04-11

AI Summary

Existing time series reasoning benchmarks predominantly rely on manually curated datasets with limited domain coverage, hindering a comprehensive evaluation of large language models’ (LLMs) true comprehension capabilities. This work proposes a scalable evaluation framework that, for the first time, leverages LLM agents combined with template-based mechanisms to automatically generate diverse questions from real-world data in domains such as healthcare, finance, and meteorology. The generated tasks span five core reasoning abilities: pattern recognition, noise interpretation, similarity analysis, anomaly detection, and causal inference. This approach overcomes the constraints of traditional manual construction, achieving data diversity comparable to human-crafted benchmarks while revealing significant deficiencies in current LLMs regarding abstract reasoning and cross-domain generalization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have shown promising performance in time series modeling tasks, but do they truly understand time series data? While multiple benchmarks have been proposed to answer this fundamental question, most are manually curated and focus on narrow domains or specific skill sets. To address this limitation, we propose scalable methods for creating comprehensive time series reasoning benchmarks that combine the flexibility of templates with the creativity of LLM agents. We first develop TimeSeriesExam, a multiple-choice benchmark using synthetic time series to evaluate LLMs across five core reasoning categories: pattern recognitionnoise understandingsimilarity analysisanomaly detection, and causality. Then, with TimeSeriesExamAgent, we scale our approach by automatically generating benchmarks from real-world datasets spanning healthcare, finance and weather domains. Through multi-dimensional quality evaluation, we demonstrate that our automatically generated benchmarks achieve diversity comparable to manually curated alternatives. However, our experiments reveal that LLM performance remains limited in both abstract time series reasoning and domain-specific applications, highlighting ongoing challenges in enabling effective time series understanding in these models. TimeSeriesExamAgent is available at https://github.com/magwiazda/TimeSeriesExamAgent.

Problem

• time series reasoning
• large language models
• benchmarking
• synthetic data
• model evaluation

Innovation

• time series reasoning
• benchmark generation
• LLM agents
• synthetic data
• scalable evaluation