TSAQA: Time Series Analysis Question And Answering Benchmark

2026-01-30

AI Summary

Existing time series question answering (QA) benchmarks are largely confined to forecasting and anomaly detection, offering limited capacity to comprehensively evaluate a model’s temporal reasoning capabilities. To address this gap, this work proposes TSAQA—a unified, multi-task time series QA benchmark encompassing six task categories: anomaly detection, classification, feature characterization, comparison, data transformation, and temporal relationship analysis. TSAQA comprises 210,000 structured QA pairs spanning 13 domains and introduces novel QA formats including true/false, multiple-choice, and puzzle-style questions. The benchmark supports both zero-shot and instruction-tuning evaluation paradigms and is compatible with standard large language model (LLM) testing pipelines. Experimental results reveal that even the strongest commercial model, Gemini-2.5-Flash, achieves only a 65.08 average score under zero-shot settings, while fine-tuned open-source models like LLaMA-3.1-8B still exhibit substantial room for improvement, underscoring the inherent challenges of temporal understanding for current LLMs.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Time-Series/Data Streams
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web question answering
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Time series data are integral to critical applications across domains such as finance, healthcare, transportation, and environmental science. While recent work has begun to explore multi-task time series question answering (QA), current benchmarks remain limited to forecasting and anomaly detection tasks. We introduce TSAQA, a novel unified benchmark designed to broaden task coverage and evaluate diverse temporal analysis capabilities. TSAQA integrates six diverse tasks under a single framework ranging from conventional analysis, including anomaly detection and classification, to advanced analysis, such as characterization, comparison, data transformation, and temporal relationship analysis. Spanning 210k samples across 13 domains, the dataset employs diverse formats, including true-or-false (TF), multiple-choice (MC), and a novel puzzling (PZ), to comprehensively assess time series analysis. Zero-shot evaluation demonstrates that these tasks are challenging for current Large Language Models (LLMs): the best-performing commercial LLM, Gemini-2.5-Flash, achieves an average score of only 65.08. Although instruction tuning boosts open-source performance: the best-performing open-source model, LLaMA-3.1-8B, shows significant room for improvement, highlighting the complexity of temporal analysis for LLMs.

Problem

• time series analysis
• question answering
• benchmark
• temporal reasoning
• large language models

Innovation

• time series question answering
• unified benchmark
• temporal analysis
• large language models
• zero-shot evaluation