Make a Donut: Hierarchical EMD-Space Planning for Zero-Shot Deformable Manipulation with Tools

2023-11-05

AI Summary

This work addresses the challenge of zero-shot robotic manipulation of deformable objects (e.g., dough) without human demonstrations. Methodologically, it introduces a hierarchical planning framework that leverages large language models (LLMs) to generate multi-stage tool-use strategies and point-cloud-based sub-goals; employs a differentiable physical loss defined in Earth Mover Distance (EMD) space; and integrates a differentiable physics engine (DiffPhysics) with closed-loop model predictive control (MPC) for precise deformation control. Its key contribution is the first LLM–physics co-designed, demonstration-free hierarchical paradigm, enabling long-horizon task generalization and real-robot deployment. Experiments demonstrate significant performance gains over prior methods on unseen complex tasks—including dough kneading and donut shaping—and successful transfer to a physical robotic arm platform.

Abstract

Deformable object manipulation stands as one of the most captivating yet formidable challenges in robotics. While previous techniques have predominantly relied on learning latent dynamics through demonstrations, typically represented as either particles or images, there exists a pertinent limitation: acquiring suitable demonstrations, especially for long-horizon tasks, can be elusive. Moreover, basing learning entirely on demonstrations can hamper the model's ability to generalize beyond the demonstrated tasks. In this work, we introduce a demonstration-free hierarchical planning approach capable of tackling intricate long-horizon tasks without necessitating any training. We employ large language models (LLMs) to articulate a high-level, stage-by-stage plan corresponding to a specified task. For every individual stage, the LLM provides both the tool's name and the Python code to craft intermediate subgoal point clouds. With the tool and subgoal for a particular stage at our disposal, we present a granular closed-loop model predictive control strategy. This leverages Differentiable Physics with Point-to-Point correspondence (DiffPhysics-P2P) loss in the earth mover distance (EMD) space, applied iteratively. Experimental findings affirm that our technique surpasses multiple benchmarks in dough manipulation, spanning both short and long horizons. Remarkably, our model demonstrates robust generalization capabilities to novel and previously unencountered complex tasks without any preliminary demonstrations. We further substantiate our approach with experimental trials on real-world robotic platforms. Our project page: https://qq456cvb.github.io/projects/donut.

Problem

• Robotics
• Tool Usage
• Shape Manipulation

Innovation

• Language Model Assisted Planning
• Deformable Object Manipulation
• Unseen Complex Task Handling