Self-Gating Attention for Efficient Time Series Forecasting

2026-07-02

AI Summary

This work addresses the computational inefficiency of standard self-attention mechanisms in time series forecasting, where quadratic complexity hinders scalable inference. The authors propose a plug-and-play Self-Gated Attention (SGA) mechanism that exploits redundancy in attention maps by decomposing attention scores into a shared pattern and an input-dependent residual component. By leveraging a learnable shared matrix, SGA eliminates the need for query-key projections, thereby achieving linear time and memory complexity. Evaluated on nine real-world datasets, SGA significantly improves inference efficiency while maintaining predictive accuracy comparable to state-of-the-art methods.

Abstract

Transformer architectures have shown strong potential in time series forecasting, where multi-head self-attention is widely used to capture temporal dependencies across historical timestamps. However, standard self-attention has quadratic time and memory complexity with respect to the look-back length. This cost may limit its use in resource-constrained or high-throughput forecasting systems, where fast and memory-efficient inference is important. Through qualitative and quantitative analyses, we observe that self-attention maps in time series forecasting often contain redundant patterns across different timestamps. This phenomenon can be related to the repeated temporal patterns and relatively stable temporal correlations in many real-world time series. Motivated by this observation, we propose Self-Gating Attention (SGA), a plug-and-play attention mechanism that represents the attention score with a shared learnable matrix and an input-dependent residual component. The shared matrix captures common attention patterns, while the residual component captures input-dependent variations. In this way, SGA avoids the query and key projections used in standard attention score computation, leading to linear time and score-matrix memory complexity with respect to the look-back length. We integrate SGA into several forecasting backbones and compare it with standard self-attention and lightweight attention variants on nine publicly available real-world datasets covering electricity, finance, weather, medical monitoring, human activity, and climate records. The results show that SGA improves inference efficiency on public benchmarks while maintaining competitive forecasting performance against state-of-the-art attention mechanisms. These benchmark results provide deployment-oriented evidence.

Problem

• time series forecasting
• self-attention
• computational efficiency
• memory complexity
• quadratic complexity

Innovation

• Self-Gating Attention
• linear complexity
• time series forecasting
• efficient attention
• plug-and-play mechanism