Comparing Different Transformer Model Structures for Stock Prediction

2025-04-23

AI Summary

This study investigates the impact of Transformer architectural design choices on stock index forecasting performance, addressing a critical gap in systematic empirical evaluation of model architectures for financial time series. We systematically evaluate five Transformer variants—Encoder-only, Decoder-only, standard Encoder-Decoder, embedding-free, and ProbSparse—at daily frequency across multiple markets (e.g., CSI 300), using MAE and MSE for quantitative assessment. Our key findings are: (1) the Decoder-only Transformer consistently achieves 12.6–18.3% lower MAE than all other Transformer variants and significantly outperforms LSTM and ARIMA baselines; (2) ProbSparse attention degrades average forecasting accuracy by 9.7%, indicating its unsuitability for high-noise, low-periodicity financial time series. These results challenge the prevailing assumption of universal applicability for sparse attention mechanisms and provide empirically grounded guidance for architecture selection in financial forecasting.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper compares different Transformer model architectures for stock index prediction. While many studies have shown that Transformers perform well in stock price forecasting, few have explored how different structural designs impact performance. Most existing works treat the Transformer as a black box, overlooking how specific architectural choices may affect predictive accuracy. However, understanding these differences is critical for developing more effective forecasting models. This study aims to identify which Transformer variant is most suitable for stock forecasting. This study evaluates five Transformer structures: (1) encoder-only Transformer, (2) decoder-only Transformer, (3) Vanilla Transformer (encoder + decoder), (4) Vanilla Transformer without embedding layers, and (5) Vanilla Transformer with ProbSparse attention. Results show that Transformer-based models generally outperform traditional approaches. Transformer with decoder only structure outperforms all other models in all scenarios. Transformer with ProbSparse attention has the worst performance in almost all cases.

Problem

• Compare Transformer architectures for stock index prediction
• Evaluate impact of structural designs on forecasting performance
• Identify optimal Transformer variant for stock price prediction

Innovation

• Compares five Transformer architectures for stocks
• Decoder-only Transformer performs best overall
• ProbSparse attention has the worst performance