AdaptPNP: Integrating Prehensile and Non-Prehensile Skills for Adaptive Robotic Manipulation

2025-11-14

AI Summary

This work addresses the challenge of unifying planning for both grasping (P) and non-grasping (NP) manipulation tasks in complex, dynamic environments. We propose AdaptPNP, a novel framework that integrates vision-language models (VLMs) with object-centric intermediate representations and digital twin technology to achieve high-precision object pose estimation and real-time scene modeling. Leveraging VLM-driven semantic-geometric joint reasoning, AdaptPNP autonomously determines P/NP action types, selects appropriate primitive skills, and generates online-replannable, multi-step hybrid policies. A control module maps high-level plans to low-level execution commands and enables closed-loop adaptation via real-time sensory feedback. Extensive experiments in simulation and on physical robotic platforms demonstrate that AdaptPNP significantly improves task success rates and cross-environment generalization. The framework establishes a scalable, end-to-end paradigm for general-purpose dexterous manipulation.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Non-prehensile (NP) manipulation, in which robots alter object states without forming stable grasps (for example, pushing, poking, or sliding), significantly broadens robotic manipulation capabilities when grasping is infeasible or insufficient. However, enabling a unified framework that generalizes across different tasks, objects, and environments while seamlessly integrating non-prehensile and prehensile (P) actions remains challenging: robots must determine when to invoke NP skills, select the appropriate primitive for each context, and compose P and NP strategies into robust, multi-step plans. We introduce ApaptPNP, a vision-language model (VLM)-empowered task and motion planning framework that systematically selects and combines P and NP skills to accomplish diverse manipulation objectives. Our approach leverages a VLM to interpret visual scene observations and textual task descriptions, generating a high-level plan skeleton that prescribes the sequence and coordination of P and NP actions. A digital-twin based object-centric intermediate layer predicts desired object poses, enabling proactive mental rehearsal of manipulation sequences. Finally, a control module synthesizes low-level robot commands, with continuous execution feedback enabling online task plan refinement and adaptive replanning through the VLM. We evaluate ApaptPNP across representative P&NP hybrid manipulation tasks in both simulation and real-world environments. These results underscore the potential of hybrid P&NP manipulation as a crucial step toward general-purpose, human-level robotic manipulation capabilities. Project Website: https://sites.google.com/view/adaptpnp/home

Problem

• Integrating prehensile and non-prehensile manipulation skills in robotics
• Creating unified framework that generalizes across tasks and environments
• Systematically selecting and combining grasping and non-grasping actions

Innovation

• VLM interprets scenes and tasks for planning
• Digital twin predicts poses for mental rehearsal
• Feedback enables adaptive replanning during execution