From Values to Tokens: An LLM-Driven Framework for Context-aware Time Series Forecasting via Symbolic Discretization

2025-08-07

AI Summary

Time-series forecasting faces challenges in effectively fusing historical numeric sequences with unstructured textual context—particularly in energy, healthcare, and finance domains. To address this, we propose TokenCast, the first framework to leverage large language models (LLMs) as cross-modal bridges: it symbolically discretizes continuous time series into semantically aligned temporal tokens, then jointly embeds them with textual context into a unified semantic space for end-to-end multimodal modeling. Our approach comprises a learnable discrete tokenizer, LLM embedding freezing and targeted fine-tuning, autoregressive generative training, and inverse decoding for reconstruction. Evaluated on multiple real-world benchmark datasets featuring authentic textual context, TokenCast consistently outperforms state-of-the-art methods, demonstrating strong generalization and practical efficacy. This work establishes a novel paradigm for multimodal time-series modeling by unifying numeric and linguistic modalities through LLM-based semantic alignment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Time series forecasting plays a vital role in supporting decision-making across a wide range of critical applications, including energy, healthcare, and finance. Despite recent advances, forecasting accuracy remains limited due to the challenge of integrating historical numerical sequences with contextual features, which often comprise unstructured textual data. To address this challenge, we propose TokenCast, an LLM-driven framework that leverages language-based symbolic representations as a unified intermediary for context-aware time series forecasting. Specifically, TokenCast employs a discrete tokenizer to transform continuous numerical sequences into temporal tokens, enabling structural alignment with language-based inputs. To bridge the semantic gap between modalities, both temporal and contextual tokens are embedded into a shared representation space via a pre-trained large language model (LLM), further optimized with autoregressive generative objectives. Building upon this unified semantic space, the aligned LLM is subsequently fine-tuned in a supervised manner to predict future temporal tokens, which are then decoded back into the original numerical space. Extensive experiments on diverse real-world datasets enriched with contextual features demonstrate the effectiveness and generalizability of TokenCast.

Problem

• Integrating historical numerical sequences with contextual unstructured text data
• Bridging semantic gap between time series and language modalities for forecasting
• Enabling context-aware forecasting using symbolic discretization and LLM alignment

Innovation

• Symbolic discretization for time series tokenization
• Shared embedding space via pre-trained LLM
• Autoregressive generative token prediction