QKCV Attention: Enhancing Time Series Forecasting with Static Categorical Embeddings for Both Lightweight and Pre-trained Foundation Models

2025-10-21

AI Summary

To address the underutilization of static categorical information in time series forecasting, this paper proposes the QKCV attention mechanism: it extends the standard QKV framework by incorporating a static categorical embedding $C$, enabling explicit modeling of category-specific dynamic patterns. The mechanism is architecture-agnostic—compatible with Transformer, Informer, PatchTST, TFT, and others—and supports efficient transfer learning via fine-tuning only the categorical embedding $C$, drastically reducing computational overhead. Experiments across multiple real-world datasets demonstrate that QKCV significantly improves prediction accuracy in univariate time series forecasting. Notably, it delivers consistent performance gains in both lightweight models and fine-tuning scenarios of pretrained foundation models, achieving an optimal balance between accuracy and efficiency. QKCV establishes a general, scalable paradigm for categorical-aware time series modeling.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Object Detection & Categorization
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

In real-world time series forecasting tasks, category information plays a pivotal role in capturing inherent data patterns. This paper introduces QKCV (Query-Key-Category-Value) attention, an extension of the traditional QKV framework that incorporates a static categorical embedding C to emphasize category-specific information. As a versatile plug-in module, QKCV enhances the forecasting accuracy of attention-based models (e.g., Vanilla Transformer, Informer, PatchTST, TFT) across diverse real-world datasets. Furthermore, QKCV demonstrates remarkable adaptability in fine-tuning univariate time series foundation model by solely updating the static embedding C while preserving pretrained weights, thereby reducing computational overhead and achieving superior fine-tuning performance.

Problem

• Incorporates static categorical embeddings to enhance time series forecasting accuracy
• Extends QKV attention framework with category-specific information for diverse datasets
• Enables efficient foundation model fine-tuning by updating only categorical embeddings

Innovation

• QKCV attention integrates static categorical embeddings
• Enhances forecasting accuracy across attention-based models
• Fine-tunes foundation models by updating static embeddings only