Collaborative Yet Personalized Policy Training: Single-Timescale Federated Actor-Critic

2026-05-14

AI Summary

This work addresses the challenge of coordinating personalized policies under environmental heterogeneity and Markovian sampling in federated reinforcement learning. The authors propose a single-timescale federated Actor-Critic framework wherein agents share a common linear subspace representation while maintaining individualized policy and critic heads, with all three components updated jointly. For the first time under Markovian sampling, the method establishes finite-time convergence guarantees for federated personalized policies and demonstrates a linear speedup with respect to the number of agents \(K\). Leveraging techniques such as projected subspace updates, QR decomposition, and careful analysis of Markovian noise perturbations, the critic estimation error and policy gradient norm converge at rates of \(\widetilde{O}(1/((1-\gamma)^4\sqrt{TK}))\) and \(\widetilde{O}(1/((1-\gamma)^6\sqrt{TK}))\), respectively. Experiments on the Hopper-v5 benchmark with heterogeneous environments show superior performance over Single PPO and FedAvg PPO, along with strong downstream transferability.

Abstract

Despite the popularity of the actor-critic method and the practical needs of collaborative policy training, existing works typically either overlook environmental heterogeneity or give up personalization altogether by training a single shared policy across all agents. We consider a federated actor-critic framework in which agents share a common linear subspace representation while maintaining personalized local policy components, and agents iteratively estimate the common subspace, local critic heads, and local policies (i.e., actors). Under canonical single-timescale updates with Markovian sampling, we establish finite-time convergence via a novel joint linear approximation framework. Specifically, we show that the critic error converges to zero at the rate of $\tilde{\mathcal{O}}(1/((1-γ)^4\sqrt{TK}))$, and the policy gradient norm converges to zero at the rate of $\tilde{\mathcal{O}}(1/((1-γ)^6\sqrt{TK}))$, where $T$ is the number of rounds, $K$ is the number of agents, and $γ\in (0,1)$ is the discount factor. These results demonstrate linear speedup with respect to the number of agents $K$, despite heterogeneous Markovian trajectories under distinct transition kernels and coupled learning dynamics. To address these challenges, we develop a new perturbation analysis for the projected subspace updates and QR decomposition steps, together with conditional mixing arguments for heterogeneous Markovian noise. Furthermore, to handle the additional complications induced by policy updates and temporal dependence, we establish fine-grained characterizations of the discrepancies between function evaluations under Markovian sampling and under temporally frozen policies. Experiments instantiate the framework within PPO on federated \texttt{Hopper-v5} action-map heterogeneity, showing gains over Single PPO and FedAvg PPO and downstream transfer from the learned shared trunk.

Problem

• federated reinforcement learning
• environmental heterogeneity
• personalized policy
• actor-critic
• collaborative training

Innovation

• Federated Reinforcement Learning
• Actor-Critic
• Personalized Policy
• Single-Timescale Convergence
• Markovian Sampling