Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement

2026-06-23

AI Summary

This work addresses the challenge of robust cloth manipulation from a single RGB image by proposing a simulator-in-the-loop optimization framework that requires no real-world training data. The approach integrates visual perception, synthetic-data-driven state reconstruction, and model-based planning to jointly optimize action trajectories online. Key innovations include the first successful application of simulator-in-the-loop optimization to real-world cloth manipulation, a learnable real-to-sim mapping module, and a planning strategy that combines sparse-grid simulation with prior-guided Model Predictive Path Integral (MPPI) control. Real-robot experiments demonstrate that the method significantly outperforms baseline approaches, achieving high task success rates and exceptional robustness to disturbances and variations in initial cloth configurations.

Abstract

Simulator-in-the-loop optimization offers a promising inference-time mechanism for robot manipulation. It uses a physical simulator as a backend rollout engine to evaluate candidate trajectories in parallel and refine nominal actions online, a paradigm proven effective in rigid-body manipulation where state and contact are relatively tractable. We bring this paradigm to real-world cloth manipulation from a single RGB input through three pillars. (i) We design a scalable synthetic-data generation and inference-time rollout pipeline built on FLASH, a deformable-object simulator that provides a practical balance among physical fidelity, numerical stability, and rollout efficiency. (ii) We develop a real-to-sim module, trained purely on synthetic data, that maps a single RGB observation to simulation-compatible cloth state by fusing pretrained visual features with learnable canonical tokens. (iii) We perform online planning by coupling a sparse-mesh rollout backend with prior-guided MPPI, anchored at an offline-distilled policy trajectory, preserving manipulation-relevant deformation and contact while enabling sufficient parallel rollout batches. Real-robot experiments show higher success rates and stronger robustness than baseline methods.

Problem

• cloth manipulation
• simulator-in-the-loop
• deformable objects
• RGB-based perception
• robotic manipulation

Innovation

• simulator-in-the-loop
• cloth manipulation
• real-to-sim
• deformable object simulation
• online planning