PLUTUS: A Well Pre-trained Large Unified Transformer can Unveil Financial Time Series Regularities

2024-08-19

arXiv.org

AI Summary

Financial time series exhibit high noise, nonlinearity, and nonstationarity, limiting the effectiveness of general-purpose pretrained models. To address this, we propose FinTS-BERT—the first open-source, billion-parameter foundational model specifically designed for financial time series. Our method introduces a reversible embedding module and a novel dual-dimensional attention mechanism that jointly captures cross-variable and temporal dependencies, enabling large-scale pretraining on trillion-scale observations via distributed training. The model integrates contrastive learning with masked autoencoding objectives to enhance robustness to noise and improve deep pattern extraction. Evaluated across price forecasting, volatility modeling, and anomaly detection, FinTS-BERT achieves state-of-the-art performance, significantly outperforming existing approaches. Moreover, it demonstrates strong cross-market and cross-asset transferability. This work establishes a new benchmark for financial time series modeling.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Financial time series modeling is crucial for understanding and predicting market behaviors but faces challenges such as non-linearity, non-stationarity, and high noise levels. Traditional models struggle to capture complex patterns due to these issues, compounded by limitations in computational resources and model capacity. Inspired by the success of large language models in NLP, we introduce $ extbf{PLUTUS}$, a $ extbf{P}$re-trained $ extbf{L}$arge $ extbf{U}$nified $ extbf{T}$ransformer-based model that $ extbf{U}$nveils regularities in financial time $ extbf{S}$eries. PLUTUS uses an invertible embedding module with contrastive learning and autoencoder techniques to create an approximate one-to-one mapping between raw data and patch embeddings. TimeFormer, an attention based architecture, forms the core of PLUTUS, effectively modeling high-noise time series. We incorporate a novel attention mechanisms to capture features across both variable and temporal dimensions. PLUTUS is pre-trained on an unprecedented dataset of 100 billion observations, designed to thrive in noisy financial environments. To our knowledge, PLUTUS is the first open-source, large-scale, pre-trained financial time series model with over one billion parameters. It achieves state-of-the-art performance in various tasks, demonstrating strong transferability and establishing a robust foundational model for finance. Our research provides technical guidance for pre-training financial time series data, setting a new standard in the field.

Problem

• Modeling financial time series with inherent stochasticity
• Addressing low signal-to-noise ratios in financial systems
• Developing foundation models for financial domain adaptation

Innovation

• Pre-trained transformer model for financial time series
• Invertible embedding module to mitigate noise effects
• Rigorous theoretical explanation with billion-scale validation