Steering Generative Reinforcement Learning into Stable Robotic Controller

2026-06-15

AI Summary

This work addresses the challenge of achieving stable and precise control in high-dimensional robotic systems using generative reinforcement learning policies, which often suffer from inherent stochasticity. To this end, the paper introduces SteerGenPO, a novel framework that decouples exploration and execution in latent space for the first time: during training, diverse exploratory behaviors are generated via diffusion or flow models, while at deployment, deterministic latent actions are employed to ensure consistent and reliable control. By incorporating a latent-space steering mechanism, SteerGenPO preserves exploration diversity while significantly enhancing execution stability. Empirical evaluations across six benchmark tasks in Isaac Lab and real-world locomotion tasks on the Unitree G1 robot demonstrate that SteerGenPO consistently outperforms both classical and state-of-the-art generative RL methods, achieving superior control accuracy and more dependable response to high-level commands.

Abstract

Diffusion and flow-based generative policies provide a powerful policy class for reinforcement learning by inducing rich stochastic exploration through iterative action generation. However, the stochasticity of diffusion policies is not suitable for stable and precise control in high-dimensional robotic systems, where small action variations can accumulate into inconsistent motion and reduced robustness. To address this issue, we propose SteerGenPO, a latent-space reinforcement learning framework that steers a trained generative policy into a robust deterministic robotic controller. The key idea is to replace stochastic latent sampling of the trained generative policy with a learned latent actor that predicts a state-dependent latent input for the generative policies. This separates exploration and control: stochastic generative sampling provides diverse action proposals during policy learning, while deterministic latent steering provides stable and adaptive control at deployment. We evaluate SteerGenPO on six Isaac Lab benchmarks and a Unitree G1 locomotion task. The results show SteerGenPO improves over both classical RL and generative RL baselines, while its deterministic latent steering produces more stable inference-time behaviors and more reliable command responses.

Problem

• diffusion policies
• robotic control
• stochasticity
• stability
• reinforcement learning

Innovation

• generative reinforcement learning
• latent-space steering
• deterministic control
• diffusion policies
• robotic policy