Regularized Reward-Punishment Reinforcement Learning

2026-06-26

AI Summary

This work addresses the challenge in traditional reinforcement learning where reward and punishment mechanisms are typically optimized independently, hindering effective coordination. To overcome this limitation, the authors propose the KL-Coupled Policy Regularization (KCPR) framework, which enables direct interaction between reward and punishment signals at the policy level for the first time. The approach employs mutual companion policies as soft priors, jointly optimizing policy updates and value propagation. It introduces a KL-regularized Bellman operator, the klDMP deep learning algorithm, and a dual replay buffer mechanism. Evaluated on grid-world environments and Gazebo-based robotic navigation tasks, klDMP demonstrates significantly improved learning stability and safety while maintaining task performance comparable to established methods such as DQN, SQL, and softDMP.

Abstract

We propose KL-Coupled Policy Regularization (KCPR), a policy coordination framework for Reward-Punishment Reinforcement Learning (RPRL). Based on KCPR, we derive KL-Coupled Soft Optimality (KCSO) and develop its deep realization, klDMP. Unlike existing RPRL approaches that optimize reward-seeking and punishment-related policies largely independently, KCPR enables direct interactions between companion policies by treating each as a dynamically learned prior for the other. KCSO yields coupled soft-optimal policies and KL-regularized Bellman operators, allowing reward and punishment information to jointly influence value propagation. To improve learning stability, we introduce a companion-prior softening mechanism and evaluate separate replay-buffer designs for balancing reward- and punishment-related experience. Experiments in grid-world and Gazebo robotic navigation tasks demonstrate that klDMP improves safety and learning stability while maintaining competitive task performance compared with DQN, SQL and softDMP. These results suggest that policy-level coordination provides an effective mechanism for integrating multiple behavioral objectives and may serve as a useful design principle for reinforcement learning systems with interacting motivational processes.

Problem

• Reward-Punishment Reinforcement Learning
• Policy Coordination
• KL Regularization
• Soft Optimality
• Multi-objective Reinforcement Learning

Innovation

• KL-Coupled Policy Regularization
• Reward-Punishment Reinforcement Learning
• Soft Optimality
• Policy Coordination
• KL-Regularized Bellman Operator