🤖 AI Summary
This study addresses the lack of systematic evaluation comparing data-driven models, such as Long Short-Term Memory (LSTM) networks, with model-based approaches in structured time series classification tasks. The authors construct a controlled evaluation framework using Monte Carlo simulations to compare LSTM against an Expectation-Maximization (EM)-based classifier within linear Gaussian state-space models, across varying task difficulties, sequence lengths, and training set sizes. The theoretical performance upper bound is established by the likelihood ratio test derived from Kalman filter outputs under known model parameters. For the first time in a structured setting, this work quantifies the performance gap between these paradigms, revealing that LSTM exhibits performance saturation when classification relies solely on differences in measurement noise, failing to surpass the theoretical optimum even with increased data or longer sequences. In contrast, the EM-based classifier consistently approaches the upper bound when model assumptions hold.
📝 Abstract
The rapid adoption of deep learning has increasingly led to data-driven models replacing classical model-based algorithms, even in domains governed by well-understood physical laws. While data-driven models, such as long short-term memory (LSTM) networks, have become a popular choice for time-series analysis, their performance relative to model-based approaches in structured environments is rarely evaluated objectively. This paper presents a performance evaluation framework comparing an LSTM classifier against a model-based expectation maximization (EM) classifier for binary time-series classification. The evaluation is conducted on two scalar linear Gaussian state space models differing only in their noise statistics, where the Kalman filter likelihood ratio test with true parameters serves as a reference for the best achievable classification performance.Through Monte Carlo simulations, the classifiers are evaluated across three axes: task difficulty, controlled by the separation in process or measurement noise between the two models; sequence length; and training dataset size. The results show that the EM classifier, which exploits the known model structure, performs strongly when the data conform to the assumed model class. The LSTM classifier requires a larger separation in noise statistics to achieve reliable classification, and its performance saturates below the reference classifier when the models differ only in measurement noise, regardless of sequence length or training dataset size.