MONSTER: Monash Scalable Time Series Evaluation Repository

2025-02-21

AI Summary

Existing time-series classification (TSC) benchmarks—such as UCR/UEA—are small-scale (median ~250 samples), leading to an overemphasis on small-sample accuracy while neglecting scalability and real-world deployment constraints. Method: We introduce the first standardized, large-scale TSC benchmark, encompassing real-world multivariate time-series datasets ranging from thousands to millions of instances, with support for distributed data loading, memory-aware preprocessing, and cross-scale performance evaluation. Contribution/Results: This benchmark is the first to systematically incorporate scalability as a core evaluation dimension, shifting the paradigm from “low-variance, small-sample optimization” to “efficient large-scale learning.” Empirical analysis reveals significant performance degradation and scalability bottlenecks in state-of-the-art models under big-data regimes, establishing a unified baseline and providing both theoretical insights and practical guidance for scalable TSC research and deployment.

Technology Category

• Machine Learning: Scalability of ML Systems
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Computer Vision: Large Vision Models

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Responsible Web: Ethical and legal aspects of web-scale data analysis, uses and collection practices

Abstract

We introduce MONSTER-the MONash Scalable Time Series Evaluation Repository-a collection of large datasets for time series classification. The field of time series classification has benefitted from common benchmarks set by the UCR and UEA time series classification repositories. However, the datasets in these benchmarks are small, with median sizes of 217 and 255 examples, respectively. In consequence they favour a narrow subspace of models that are optimised to achieve low classification error on a wide variety of smaller datasets, that is, models that minimise variance, and give little weight to computational issues such as scalability. Our hope is to diversify the field by introducing benchmarks using larger datasets. We believe that there is enormous potential for new progress in the field by engaging with the theoretical and practical challenges of learning effectively from larger quantities of data.

Problem

• Addresses lack of large time series datasets
• Expands beyond small dataset benchmarks
• Focuses on scalable learning challenges

Innovation

• Scalable time series datasets
• Larger benchmarks diversification
• Focus on computational scalability