On the Fly Adaptation of Behavior Tree-Based Policies through Reinforcement Learning

2025-03-08

AI Summary

Local task variations—such as minor workpiece misalignments and dimensional discrepancies—in flexible manufacturing often cause robot control policies to fail due to insufficient generalization. Method: This paper proposes a hierarchical adaptive framework that embeds reinforcement learning into behavior trees (RL-BT), achieving the first deep integration of hierarchical reinforcement learning (HRL) with behavior trees. The framework preserves BT’s modularity, reactivity, and interpretability while enabling online, sample-efficient dynamic task generalization. It employs sim-to-real co-training on a Franka Emika Panda 7-DoF manipulator for obstacle avoidance and rotational manipulation tasks. Contribution/Results: Experimental results demonstrate rapid convergence, high robustness, and minimal online interaction requirements for policy adaptation. The method significantly enhances robotic behavioral flexibility and adaptability in dynamic, unstructured environments.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

With the rising demand for flexible manufacturing, robots are increasingly expected to operate in dynamic environments where local -- such as slight offsets or size differences in workpieces -- are common. We propose to address the problem of adapting robot behaviors to these task variations with a sample-efficient hierarchical reinforcement learning approach adapting Behavior Tree (BT)-based policies. We maintain the core BT properties as an interpretable, modular framework for structuring reactive behaviors, but extend their use beyond static tasks by inherently accommodating local task variations. To show the efficiency and effectiveness of our approach, we conduct experiments both in simulation and on a Franka Emika Panda 7-DoF, with the manipulator adapting to different obstacle avoidance and pivoting tasks.

Problem

• Adapting robot behaviors to dynamic manufacturing environments
• Using reinforcement learning for sample-efficient behavior adaptation
• Extending Behavior Tree policies to handle local task variations

Innovation

• Hierarchical reinforcement learning adapts Behavior Tree policies.
• Behavior Trees extended for dynamic, local task variations.
• Experiments validate adaptability in obstacle and pivoting tasks.