A Dataset and Benchmark for Robotic Cloth Unfolding Grasp Selection: The ICRA 2024 Cloth Competition

2025-08-22

AI Summary

The robotics community lacks standardized benchmarks and shared datasets for cloth manipulation, particularly for aerial cloth unfolding. Method: This work introduces the first benchmark for grasp pose selection in aerial cloth unfolding, validated through head-to-head real-world evaluation in the ICRA 2024 Cloth Manipulation Competition. We collect 679 real-robot unfolding demonstrations across 34 garment categories and propose a joint metric combining grasp success rate and cloth coverage area. The dataset, benchmark code, and evaluation protocol are publicly released. Contributions/Results: (1) First open, reproducible, and method-agnostic benchmark enabling cross-algorithm comparison in cloth manipulation; (2) Empirical evidence that handcrafted methods remain competitive in real-world settings; (3) Establishment of an out-of-lab evaluation paradigm; (4) Substantial improvement in benchmarking consistency and method validation capability within the field.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Robotic cloth manipulation suffers from a lack of standardized benchmarks and shared datasets for evaluating and comparing different approaches. To address this, we created a benchmark and organized the ICRA 2024 Cloth Competition, a unique head-to-head evaluation focused on grasp pose selection for in-air robotic cloth unfolding. Eleven diverse teams participated in the competition, utilizing our publicly released dataset of real-world robotic cloth unfolding attempts and a variety of methods to design their unfolding approaches. Afterwards, we also expanded our dataset with 176 competition evaluation trials, resulting in a dataset of 679 unfolding demonstrations across 34 garments. Analysis of the competition results revealed insights about the trade-off between grasp success and coverage, the surprisingly strong achievements of hand-engineered methods and a significant discrepancy between competition performance and prior work, underscoring the importance of independent, out-of-the-lab evaluation in robotic cloth manipulation. The associated dataset is a valuable resource for developing and evaluating grasp selection methods, particularly for learning-based approaches. We hope that our benchmark, dataset and competition results can serve as a foundation for future benchmarks and drive further progress in data-driven robotic cloth manipulation. The dataset and benchmarking code are available at https://airo.ugent.be/cloth_competition.

Problem

• Addressing lack of standardized benchmarks for robotic cloth manipulation
• Evaluating grasp pose selection methods for in-air cloth unfolding
• Providing dataset for developing learning-based cloth manipulation approaches

Innovation

• Created benchmark for robotic cloth unfolding
• Organized ICRA 2024 competition with teams
• Expanded dataset with 679 unfolding demonstrations