Problem
Research questions and friction points this paper is trying to address.
Meta-reinforcement learning
Edge Caching
Base Stations
Gradient Variance
Wireless Networks
Fast-Convergent Meta-RL via Gradient-Clustered BS Sampling for Edge Caching
🤖 AI Summary
本文针对无线边缘缓存网络中RL缓存代理训练慢的问题，提出了一种基于梯度聚类的元强化学习框架，加速收敛并降低元梯度估计的方差。
📝 Abstract
Wireless edge caching networks typically consist of many independent Base Stations (BSs), each facing its own request rate and content popularity profile. Training a Reinforcement Learning (RL) caching agent from scratch at every BS forces each agent to relearn, through slow trial and error, a decision problem that is structurally identical across the network. Meta-reinforcement learning removes this redundancy by learning a shared initialization that adapts to any BS in a few local updates; however, meta-training itself becomes the bottleneck at scale: the meta-gradient must be estimated from a small subset of BSs at each meta-iteration, and sampling this subset uniformly at random yields a high-variance estimate, an issue existing meta-RL caching frameworks leave unaddressed. This paper proposes a meta-reinforcement learning framework for caching across independent, non-overlapping BSs that directly targets this bottleneck. Each BS runs a local Proximal Policy Optimization (PPO) agent, formulated as a Semi-Markov Decision Process (SMDP) over content popularity, size, lifetime, and importance, while a shared meta-policy is learned via a Model-Agnostic Meta-Learning (MAML)-style loop. To scale meta-training and accelerate convergence, we introduce gradient-based clustering, which groups BSs by local gradient similarity and draws from every cluster, in proportion to its size, at each meta-iteration. We prove, via an Analysis of Variance (ANOVA)-style decomposition of gradient variance, that this strategy yields a strictly lower-variance meta-gradient estimator than uniform random sampling under BS heterogeneity.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Gradient-Clustered BS Sampling
Meta-Reinforcement Learning
Edge Caching
Variance Reduction
Proximal Policy Optimization
🔎 Similar Papers
No similar papers found.
