Impedance Primitive-augmented Hierarchical Reinforcement Learning for Sequential Tasks

2025-08-27

AI Summary

To address low sample efficiency, poor task composability, and insufficient stiffness adaptability in robotic sequential contact tasks, this paper proposes a hierarchical reinforcement learning framework integrating impedance behavior primitives. Methodologically, we design variable-stiffness behavior primitives embedded with adaptive stiffness modulation and affordance-driven exploration, coupled with a dynamic impedance controller enabling end-to-end training. Crucially, we innovatively couple affordance perception with impedance parameterization to enhance environmental compliance and exploration efficiency. Experiments on contact-intensive tasks—including block grasping, door opening, and object pushing—demonstrate that our framework significantly improves learning speed (2.3× average acceleration) and task success rates (96.5% in simulation; 89.2% on real hardware), while enabling cross-task policy transfer and efficient sim-to-real deployment.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

This paper presents an Impedance Primitive-augmented hierarchical reinforcement learning framework for efficient robotic manipulation in sequential contact tasks. We leverage this hierarchical structure to sequentially execute behavior primitives with variable stiffness control capabilities for contact tasks. Our proposed approach relies on three key components: an action space enabling variable stiffness control, an adaptive stiffness controller for dynamic stiffness adjustments during primitive execution, and affordance coupling for efficient exploration while encouraging compliance. Through comprehensive training and evaluation, our framework learns efficient stiffness control capabilities and demonstrates improvements in learning efficiency, compositionality in primitive selection, and success rates compared to the state-of-the-art. The training environments include block lifting, door opening, object pushing, and surface cleaning. Real world evaluations further confirm the framework's sim2real capability. This work lays the foundation for more adaptive and versatile robotic manipulation systems, with potential applications in more complex contact-based tasks.

Problem

• Develops hierarchical reinforcement learning for robotic sequential tasks
• Enables variable stiffness control in contact-based manipulation
• Improves learning efficiency and success rates in manipulation

Innovation

• Hierarchical reinforcement learning with impedance primitives
• Variable stiffness control for robotic manipulation
• Adaptive stiffness controller with affordance coupling