Safe Execution of RL Policies Via Acceleration-Based CBF-QP Constraint Enforcement for Real-World Robotic Deployments

2026-07-15

AI Summary

This work addresses the safety risks posed by out-of-distribution states when deploying reinforcement learning policies on real robots. The authors propose Acc-CBF-QP, a runtime safety filter that enforces policy actions to remain within a safe set without altering the training process. By integrating acceleration-based Control Barrier Functions (CBFs) with Quadratic Programming (QP), the method innovatively unifies position, velocity, torque, and collision constraints through TorqueTask and Forward Dynamics Task formulations, enabling principled trade-offs between safety and performance. Experimental results demonstrate that the approach reduces constraint violation rates by 92% on the Unitree H1 humanoid robot—from 10.04 to 0.80 violations per second—and completely eliminates violations on the Kinova Gen3 robotic arm, all while preserving the original task performance.

Abstract

Reinforcement Learning (RL) has demonstrated remarkable capabilities for solving complex robotic control problems, but its lack of safety guarantees severely limits deployment on hardware. In particular, as legged robots and manipulators often operate near safety-critical boundaries, out-of-distribution states can lead to failure upon deployment. To address this, we introduce Acc-CBF-QP, an acceleration-based Quadratic Program (QP) safety filter using Control Barrier Functions (CBFs) that constrains any RL policy onto a safe set at runtime without modifying training. The method applies to unconstrained and Safe-RL policies, and enforces joint position, velocity, torque, and collision constraints within a unified optimization framework. A key contribution is the formulation of RL+QP tasks that regulate deviation from the RL command when constraints would otherwise be violated. We introduce a TorqueTask, minimizing torque deviation, and a Forward Dynamics Task, minimizing induced acceleration deviation, thus providing principled control over safety-performance trade-offs. Experiments on a 7-DoF Kinova Gen3 manipulator and a 19-DoF Unitree H1 humanoid, both in simulation and on hardware, highlight substantial reductions in constraint violations. On the real H1 hardware, a Safe-RL policy alone yielded 10.04 violations/s, which were reduced by 92% to 0.80 violations/s when augmented with Acc-CBF-QP. On the Kinova Gen3, Acc-CBF-QP fully eliminated violations. Nominal task performance of the RL objective is preserved in violation-free regimes. Under aggressive velocity commands on H1, Acc-CBF-QP improves execution by preventing constraint-induced shutdowns, yielding longer survival times. The full pipeline is open-source.

Problem

• Reinforcement Learning
• Safety Guarantees
• Real-World Deployment
• Constraint Violations
• Robotic Control

Innovation

• Control Barrier Functions
• Quadratic Programming
• Reinforcement Learning Safety
• Acceleration-based Constraints
• Real-world Robotics