Are Data Embeddings effective in time series forecasting?

2025-05-27

AI Summary

Is the necessity of data embedding layers in time-series forecasting overestimated? This study systematically challenges the prevailing assumption through ablation experiments across 15 state-of-the-art models and 4 benchmark datasets. Contrary to conventional wisdom, we find that embedding layers—commonly regarded as essential—consistently degrade both accuracy and efficiency in most advanced architectures. Removing them reduces MAE/MSE significantly (exceeding typical performance gaps between leading models) and accelerates inference by 23–41% on average. We introduce a standardized evaluation framework enabling cross-model and cross-dataset comparison, supported by fully reproducible, open-source implementations. Our core contribution is a paradigm-shifting critique of the “embedding-by-default” design principle, empirically demonstrating the absence of marginal gain from embeddings in most forecasting scenarios. These findings provide both theoretical grounding and practical guidance for developing lightweight, high-efficiency time-series models.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Time series forecasting plays a crucial role in many real-world applications, and numerous complex forecasting models have been proposed in recent years. Despite their architectural innovations, most state-of-the-art models report only marginal improvements -- typically just a few thousandths in standard error metrics. These models often incorporate complex data embedding layers to transform raw inputs into higher-dimensional representations to enhance accuracy. But are data embedding techniques actually effective in time series forecasting? Through extensive ablation studies across fifteen state-of-the-art models and four benchmark datasets, we find that removing data embedding layers from many state-of-the-art models does not degrade forecasting performance. In many cases, it improves both accuracy and computational efficiency. The gains from removing embedding layers often exceed the performance differences typically reported between competing models. Code available at: https://github.com/neuripsdataembedidng/DataEmbedding

Problem

• Evaluating effectiveness of data embeddings in time series forecasting
• Assessing impact of removing embedding layers on model performance
• Comparing accuracy and efficiency gains without embedding layers

Innovation

• Removing data embedding layers improves performance
• Data embeddings often unnecessary in forecasting
• Simpler models enhance accuracy and efficiency