LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters

2023-08-16

AI Summary

To address core challenges in multivariate time series forecasting—namely, large language models’ (LLMs) difficulty in processing non-textual data, capturing multi-scale temporal dynamics, and generalizing under few-shot settings—this paper proposes a two-stage fine-tuning framework. In Stage I, temporal alignment embeddings and a dual-level sliding-window aggregation mechanism explicitly bridge semantic and structural discrepancies between LLMs and time series. In Stage II, position encoding adaptation and task-specific supervised fine-tuning optimize the model for downstream forecasting. This work achieves the first structured alignment of LLMs within the time series representation space. Evaluated on seven benchmark datasets, our method consistently outperforms from-scratch baselines and ranks first under few-shot conditions. Ablation studies validate the efficacy of each component, and further analysis confirms that pretrained weights critically underpin generalization capability.

Abstract

Multivariate time-series forecasting is vital in various domains, e.g., economic planning and weather prediction. Deep train-from-scratch models have exhibited effective performance yet require large amounts of data, which limits real-world applicability. Recently, researchers have leveraged the representation learning transferability of pre-trained Large Language Models (LLMs) to handle limited non-linguistic datasets effectively. However, incorporating LLMs with time-series data presents challenges of limited adaptation due to different compositions between time-series and linguistic data, and the inability to process multi-scale temporal information. To tackle these challenges, we propose LLM4TS, a framework for time-series forecasting with pre-trained LLMs. LLM4TS consists of a two-stage fine-tuning strategy: the time-series alignment stage to align LLMs with the nuances of time-series data, and the forecasting fine-tuning stage for downstream time-series forecasting tasks. Furthermore, our framework features a novel two-level aggregation method that integrates multi-scale temporal data within pre-trained LLMs, enhancing their ability to interpret time-specific information. In experiments across 7 time-series forecasting datasets, LLM4TS is superior to existing state-of-the-art methods compared with trained-from-scratch models in full-shot scenarios, and also achieves the highest rank in few-shot scenarios. In addition, evaluations compared with different unsupervised representation learning approaches highlight LLM4TS's effectiveness with representation learning in forecasting tasks. Ablation studies further validate each component's contribution to LLM4TS and underscore the essential role of utilizing LLM's pre-trained weights for optimal performance. The code is available at https://github.com/blacksnail789521/LLM4TS.

Problem

• Adapting LLMs for time-series forecasting
• Addressing multi-scale temporal data challenges
• Enhancing data efficiency in forecasting tasks

Innovation

• Two-stage fine-tuning strategy
• Multi-scale temporal data integration
• Pre-trained LLMs for forecasting