Behavior-Consistent Deep Reinforcement Learning

2026-05-20

AI Summary

This work addresses the high variability in policies across different training runs of reinforcement learning, which leads to unstable performance and hinders real-world deployment. It formally introduces the problem of behavior-consistent reinforcement learning and proposes a maximum-entropy framework that enhances consistency by constraining the similarity of policy distributions. The key innovation is the Q-Expectile Divergence (QED), a single-run proxy metric that enables adaptive scheduling of state-dependent temperature parameters. Theoretically, the temperature is shown to be proportional to the divergence of Q-functions, effectively controlling the KL divergence between policies. Evaluated on 18 continuous control tasks, the method reduces cross-run policy divergence by two orders of magnitude, substantially decreases return variance, and achieves this with negligible performance loss and only a minor cost in sample efficiency.

Abstract

Reinforcement learning (RL) often exhibits high variance across training runs, leading to unreliable performance and posing a major challenge to deployment in real-world domains. In this work, we address the challenge of cross-run policy divergence by formalizing the problem of behavior-consistent RL, where the objective is to obtain policies that are both high-performing and distributionally similar across training runs. Our key observation is that maximum-entropy RL provides a direct mechanism for controlling behavioral divergence by anchoring runs to a common (uniform) prior. We prove that, for Boltzmann policies, choosing the temperature proportional to $Q$-function disagreement bounds the pairwise KL divergence between the induced policies. However, we also show that naïvely increasing entropy might impair policy optimization while amplifying off-policy error. Building upon these observations, we propose $Q$-value Expectile Disagreement (QED), a state-dependent temperature schedule that uses double-critic disagreement as a single-run proxy for cross-run disagreement. Empirically, we demonstrate that across 18 continuous-control tasks, QED reduces across-run divergence by two orders of magnitude without sacrificing performance, resulting in a considerable reduction in return variance at modest sample-efficiency costs.

Problem

• behavior-consistent reinforcement learning
• cross-run policy divergence
• policy variance
• distributional similarity
• reinforcement learning reliability

Innovation

• behavior-consistent reinforcement learning
• maximum-entropy RL
• Q-value Expectile Disagreement
• temperature scheduling
• policy divergence