LNUCB-TA: Linear-nonlinear Hybrid Bandit Learning with Temporal Attention

2025-03-01

AI Summary

To address the challenges of jointly modeling long-term trends and local patterns, and poor adaptability caused by static exploration rates in contextual multi-armed bandits with dynamic environments, this paper proposes a linear–nonlinear hybrid estimation framework. First, a k-NN–based nonlinear module adaptively selects *k* according to reward variance to precisely capture local abrupt changes. Second, a global–local two-level temporal attention mechanism enables parameter-free dynamic balancing between exploration and exploitation. Third, the framework integrates context feature embedding, linear regression, and a theoretically grounded upper confidence bound. Empirical evaluations across diverse dynamic environments demonstrate significant improvements in cumulative and average rewards, convergence speed, and robustness. Theoretically, the algorithm achieves a sublinear regret bound, and its computational complexity is superior to mainstream nonlinear bandit models.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Existing contextual multi-armed bandit (MAB) algorithms fail to effectively capture both long-term trends and local patterns across all arms, leading to suboptimal performance in environments with rapidly changing reward structures. They also rely on static exploration rates, which do not dynamically adjust to changing conditions. To overcome these limitations, we propose LNUCB-TA, a hybrid bandit model integrating a novel nonlinear component (adaptive k-Nearest Neighbors (k-NN)) for reducing time complexity, alongside a global-and-local attention-based exploration mechanism. Our approach uniquely combines linear and nonlinear estimation techniques, with the nonlinear module dynamically adjusting k based on reward variance to enhance spatiotemporal pattern recognition. This reduces the likelihood of selecting suboptimal arms while improving reward estimation accuracy and computational efficiency. The attention-based mechanism ranks arms by past performance and selection frequency, dynamically adjusting exploration and exploitation in real time without requiring manual tuning of exploration rates. By integrating global attention (assessing all arms collectively) and local attention (focusing on individual arms), LNUCB-TA efficiently adapts to temporal and spatial complexities. Empirical results show LNUCB-TA significantly outperforms state-of-the-art linear, nonlinear, and hybrid bandits in cumulative and mean reward, convergence, and robustness across different exploration rates. Theoretical analysis further confirms its reliability with a sub-linear regret bound.

Problem

• Captures long-term trends and local patterns effectively
• Dynamically adjusts exploration rates to changing conditions
• Improves reward estimation accuracy and computational efficiency

Innovation

• Hybrid model with linear-nonlinear estimation techniques
• Dynamic k-NN for spatiotemporal pattern recognition
• Attention-based exploration mechanism for real-time adaptation