Optimistic Reinforcement Learning-Based Skill Insertions for Task and Motion Planning

2025-10-15

AI Summary

To address the challenges of modeling probabilistic actions and integrating symbolic planning with reinforcement learning (RL) in robot long-horizon task and motion planning (TAMP), this paper proposes a hierarchical planning framework. First, it employs a data-driven approach to encode deep RL skills as interpretable logical primitives, enabling direct invocation by a symbolic planner. Second, it introduces an optimistic policy search–based plan refinement mechanism to dynamically mitigate execution uncertainty. This work achieves, for the first time, tight coupling between RL-based skills and TAMP within a unified logical framework. Experiments on manipulation tasks involving multiple sources of uncertainty demonstrate that the method improves planning success rate by 32.5% and reduces average planning time by 41.7%, outperforming both classical TAMP and hierarchical RL baselines.

Technology Category

• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Multiagent Planning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Task and motion planning (TAMP) for robotics manipulation necessitates long-horizon reasoning involving versatile actions and skills. While deterministic actions can be crafted by sampling or optimizing with certain constraints, planning actions with uncertainty, i.e., probabilistic actions, remains a challenge for TAMP. On the contrary, Reinforcement Learning (RL) excels in acquiring versatile, yet short-horizon, manipulation skills that are robust with uncertainties. In this letter, we design a method that integrates RL skills into TAMP pipelines. Besides the policy, a RL skill is defined with data-driven logical components that enable the skill to be deployed by symbolic planning. A plan refinement sub-routine is designed to further tackle the inevitable effect uncertainties. In the experiments, we compare our method with baseline hierarchical planning from both TAMP and RL fields and illustrate the strength of the method. The results show that by embedding RL skills, we extend the capability of TAMP to domains with probabilistic skills, and improve the planning efficiency compared to the previous methods.

Problem

• Integrating RL skills into TAMP for robotic manipulation
• Addressing uncertainty in probabilistic actions for planning
• Improving planning efficiency with data-driven logical components

Innovation

• Integrates RL skills into TAMP pipelines
• Defines skills with data-driven logical components
• Uses plan refinement to handle effect uncertainties