Parameter-Efficient Adaptation of Pretrained Language Models for Time-Series Forecasting

2026-09-14

AI Summary

研究通过参数高效的迁移学习框架，将预训练语言模型应用于时间序列预测，解决了跨模态迁移问题。方法包括直接将时间序列片段投影到GPT-2的嵌入空间，并分析了不同设计选择的影响。

Abstract

We study the adaptation of pretrained language models to univariate time-series forecasting through a parameter-efficient transfer learning framework, with the goal of understanding which design choices drive effective cross-modal transfer. While language models operate on discrete textual tokens, time series consist of continuous numerical observations with temporal dependencies. To bridge this modality gap, we project fixed-length time-series patches directly into the embedding space of a pretrained GPT-2 backbone, bypassing textual tokenization and treating the Transformer as a generic sequence encoder. Through controlled ablation studies on seven benchmark datasets spanning energy, weather, traffic, and finance, we analyze the effects of (i)~representation strategy (continuous embeddings versus textual serialisation), (ii)~adaptation regime (frozen backbone versus partial or full fine-tuning), (iii)~architectural components such as adapters, pooling strategies, and prediction heads, and (iv)~input context length. Continuous patch-based embeddings consistently outperform textual prompting and randomly initialised backbones. The adapted pipeline attains MASE within the range of specialised forecasting architectures while updating less than 1\% of total model parameters. Results further indicate that freezing the pretrained backbone and training lightweight projection and adapter modules provides a favourable accuracy--efficiency trade-off with stable behaviour across varying context lengths.

Problem

• pretrained language models
• time-series forecasting
• cross-modal transfer
• parameter-efficient adaptation
• continuous numerical observations

Innovation

• parameter-efficient transfer learning
• continuous patch-based embeddings
• pretrained language models for time-series
• frozen backbone with adapters