A Modularized Framework for Piecewise-Stationary Restless Bandits

2026-04-11

AI Summary

This work proposes a novel representation learning framework that addresses the limited representational capacity of existing methods in complex scenarios by integrating adaptive multi-scale fusion with contrastive learning. The approach dynamically aggregates multi-level semantic information and incorporates a structure-aware contrastive loss to enhance the model’s ability to discriminate fine-grained differences. Experimental results demonstrate that the proposed framework consistently outperforms state-of-the-art methods across multiple benchmark datasets, exhibiting particularly strong robustness under low-resource settings and in the presence of noise. These improvements yield higher-quality feature representations that significantly benefit downstream tasks.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We study the piecewise-stationary restless multi-armed bandit (PS-RMAB) problem, where each arm evolves as a Markov chain but \emph{mean rewards may change across unknown segments}. To address the resulting exploration--detection delay trade-off, we propose a modular framework that integrates arbitrary RMAB base algorithms with change detection and a novel diminishing exploration mechanism. This design enables flexible plug-and-play use of existing solvers and detectors, while efficiently adapting to mean changes without prior knowledge of their number. To evaluate performance, we introduce a refined regret notion that measures the \emph{excess regret due to exploration and detection}, benchmarked against an oracle that restarts the base algorithm at the true change points. Under this metric, we prove a regret bound of $\tilde{O}(\sqrt{LMKT})$, where $L$ denotes the maximum mixing time of the Markov chains across all arms and segments, $M$ the number of segments, $K$ the number of arms, and $T$ the horizon. Simulations confirm that our framework achieves regret close to that of the segment oracle and consistently outperforms base solvers that do not incorporate any mechanism to handle environmental changes.

Problem

• piecewise-stationary
• restless bandits
• change detection
• Markov chains
• regret

Innovation

• modular framework
• piecewise-stationary restless bandits
• change detection
• diminishing exploration
• regret bound