Q-exponential Family for Policy Optimization

2024-08-14

arXiv.org

AI Summary

To address the limited robustness and generalization of Gaussian policies in continuous control, this work introduces the q-exponential family—specifically q-Gaussian and Student’s t distributions—into the actor-critic framework for heavy-tailed or light-tailed policy parameterization. We derive the q-policy gradient under Tsallis advantage weighting and design a numerically stable training scheme, enhancing both flexibility and robustness in policy optimization. Empirical evaluation across multiple offline RL benchmarks demonstrates that our approach consistently outperforms Gaussian policies: Student’s t policies exhibit superior training stability, while q-Gaussian policies achieve state-of-the-art performance in AWAC. This work establishes a novel paradigm for policy modeling in deep reinforcement learning and reveals the critical role of heavy-tailed priors in improving generalization.

Technology Category

• Game Theory and Economic Paradigms: Adversarial Learning
• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Policy optimization methods benefit from a simple and tractable policy parametrization, usually the Gaussian for continuous action spaces. In this paper, we consider a broader policy family that remains tractable: the $q$-exponential family. This family of policies is flexible, allowing the specification of both heavy-tailed policies ($q>1$) and light-tailed policies ($q<1$). This paper examines the interplay between $q$-exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian. In particular, we find the Student's t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed $q$-Gaussian for Tsallis Advantage Weighted Actor-Critic consistently performs well in offline benchmark problems. Our code is available at url{https://github.com/lingweizhu/qexp}.

Problem

• q-exponential family
• reinforcement learning
• continuous action space

Innovation

• q-exponential family
• long-tailed distribution
• Tsallis advantage weighted actor-critic algorithm