General-purpose Clothes Manipulation with Semantic Keypoints

2024-08-15

arXiv.org

AI Summary

Current household robots exhibit limited clothing manipulation capabilities due to the high-dimensional deformability of fabrics, often restricting them to single-task execution (e.g., folding only). To address this, we propose CLASP—a generalizable clothing manipulation framework that (1) formally defines semantic keypoints on garments (e.g., “left sleeve”) and unifies semantic planning with geometric control; (2) introduces an LLM-driven hierarchical learning architecture enabling zero-shot transfer across unseen garment types (T-shirts, towels, pants) and manipulation tasks (folding, flattening, hanging); and (3) leverages simulation pretraining followed by real-robot co-finetuning. Experiments demonstrate that CLASP significantly outperforms baselines in simulation and achieves direct deployment on physical robots, successfully handling over ten common garment categories with an average task success rate exceeding 85%. To our knowledge, CLASP is the first clothing manipulation solution for domestic service robots that simultaneously delivers strong generalization, multi-task versatility, and plug-and-play deployability.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Clothes manipulation is a critical skill for household robots. Recent advancements have been made in task-specific clothes manipulation, such as folding, flattening, and hanging. However, due to clothes' complex geometries and deformability, creating a general-purpose robot system that can manipulate a diverse range of clothes in many ways remains challenging. Since clothes are typically designed with specific structures, we propose identifying these specific features like ``left sleeve'' as semantic keypoints. Semantic keypoints can provide semantic cues for task planning and geometric cues for low-level action generation. With this insight, we develop a hierarchical learning framework using the large language model (LLM) for general-purpose CLothes mAnipulation with Semantic keyPoints (CLASP). Extensive simulation experiments show that CLASP outperforms baseline methods on both seen and unseen tasks across various clothes manipulation tasks. Real-world experiments show that CLASP can be directly deployed in the real world and applied to a wide variety of clothes.

Problem

• General-purpose clothes manipulation for household robots
• Overcoming high-dimensional geometry challenges in deformable fabric
• Bridging task planning and action execution via semantic keypoints

Innovation

• Uses semantic keypoints for clothes representation
• Bridges LLM planning with action execution
• Works across diverse clothes and tasks