Safe Online Learning via Smooth Safety-Structured Policy Composition

2026-06-30

AI Summary

This work addresses the challenge of online reinforcement learning under strict safety constraints while maintaining smooth learning dynamics. To this end, the authors propose AutoSafe, a novel architecture that seamlessly integrates structured safety monitoring and intervention mechanisms directly into the policy action generation process. By leveraging a risk-aware behavior-switching mechanism, AutoSafe achieves a continuous and adaptive trade-off between performance optimization and safety assurance. The approach combines structured safe policy composition with an online learning framework for continuous control, effectively circumventing the discontinuities typically introduced by conventional intervention strategies. Empirical results demonstrate that AutoSafe simultaneously attains high safety constraint satisfaction rates and smooth learning dynamics across multiple continuous control benchmarks, with successful real-world deployment validated on a physical inverted pendulum system.

Abstract

Safe online reinforcement learning requires policies to respect safety constraints while maintaining smooth optimization dynamics. Existing approaches typically rely on either strict safety enforcement via action interventions, which introduce discontinuities in system interaction and learning, or soft safety constraint formulations, which preserve smooth learning but provide limited safety assurance. We propose AutoSafe, a safety-aware policy architecture that integrates structured safety monitoring and intervention directly into the action generation process. This design enables smooth, risk-dependent transitions between performance-driven and safety-preserving behaviors, resulting in continuous online interaction and learning dynamics. Empirical results across a suite of continuous-control benchmarks demonstrate strong safety enforcement without sacrificing learning smoothness. We further validate AutoSafe on a physical cart-pole system, highlighting its practical effectiveness for safe online learning in the real world.

Problem

• safe reinforcement learning
• online learning
• smooth optimization
• safety constraints
• policy composition

Innovation

• safe reinforcement learning
• policy composition
• smooth safety enforcement
• online learning
• safety-aware architecture