JustDense: Just using Dense instead of Sequence Mixer for Time Series analysis

2025-08-04

AI Summary

Mainstream time-series models rely on complex sequence mixers—such as self-attention or MLP-Mixers—yet their necessity remains未经 systematically validated. Method: We propose MatrixMixer, a unifying framework that abstracts diverse sequence mixers as learnable mixing matrices, and for the first time systematically replaces them with fully connected layers (i.e., dense weight matrices), conducting large-scale empirical evaluation across 29 benchmark tasks. Contribution/Results: Our experiments demonstrate that the simplified fully connected architecture matches or surpasses state-of-the-art methods across five canonical time-series tasks—including forecasting, classification, imputation, anomaly detection, and regression—thereby challenging the implicit assumption that deeper or more complex architectures inherently yield superior performance. This work establishes a new paradigm for efficient, interpretable time-series modeling, showing that sophisticated sequence-mixing mechanisms are not strictly necessary for strong empirical performance.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Sequence and channel mixers, the core mechanism in sequence models, have become the de facto standard in time series analysis (TSA). However, recent studies have questioned the necessity of complex sequence mixers, such as attention mechanisms, demonstrating that simpler architectures can achieve comparable or even superior performance. This suggests that the benefits attributed to complex sequencemixers might instead emerge from other architectural or optimization factors. Based on this observation, we pose a central question: Are common sequence mixers necessary for time-series analysis? Therefore, we propose JustDense, an empirical study that systematically replaces sequence mixers in various well-established TSA models with dense layers. Grounded in the MatrixMixer framework, JustDense treats any sequence mixer as a mixing matrix and replaces it with a dense layer. This substitution isolates the mixing operation, enabling a clear theoretical foundation for understanding its role. Therefore, we conducted extensive experiments on 29 benchmarks covering five representative TSA tasks using seven state-of-the-art TSA models to address our research question. The results show that replacing sequence mixers with dense layers yields comparable or even superior performance. In the cases where dedicated sequence mixers still offer benefits, JustDense challenges the assumption that "deeper and more complex architectures are inherently better" in TSA.

Problem

• Investigating necessity of complex sequence mixers in time series analysis
• Proposing JustDense to replace sequence mixers with dense layers
• Challenging deeper complex architectures assumption in time series models

Innovation

• Replace sequence mixers with dense layers
• MatrixMixer framework for mixing operations
• Simpler architectures match complex performance