SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation

2026-06-13

AI Summary

This work addresses the challenge of sim-to-real transfer for RGB vision–driven manipulation of deformable objects, eliminating the need for real-world fine-tuning or task-specific calibration. The authors propose a zero-shot visuomotor policy framework grounded in high-fidelity simulation, achieving effective real-world performance with only 200 simulated demonstrations per task. Key components include a measurement-informed high-fidelity simulator, a single-image asset generation pipeline, a topology-aware trajectory synthesizer, and an ISP-aware photometric adaptation protocol. Evaluated across five deformable object manipulation tasks, the method achieves an average success rate of 91%—exceeding 80% for plastic bag handling and reaching 100% for silk grasping—significantly outperforming baselines that rely on real-world data, while reducing trajectory execution cost by two orders of magnitude.

Abstract

RGB sim-to-real for deformable manipulation has remained largely unsolved without real-world fine-tuning. We present SimWeaver, which trains zero-shot RGB VLA policies on 200 simulated demonstrations per task, reaching above 80% per-task and 91% average real-world success across 5 diverse deformable tasks including plastic-bag manipulation, without teleoperation or per-task calibration. SimWeaver combines a reliable measurement-backed simulator (SimWeaver-Sim) with an extensible asset framework supporting single-image generation(SimWeaver-Asset), a deterministic topology-aware trajectory synthesizer (SimWeaver-Syn), and a sim-to-real protocol with ISP-aware photometric augmentation (SimWeaver-Real). On silk grasping, the sim-trained policy reaches 100% under visual distribution shifts where real-data baselines drop to 9-70%, at two orders of magnitude lower per-trajectory cost. We will release SimWeaver and a representative asset subset. Project page: https://simweaver.github.io/

Problem

• sim-to-real
• deformable manipulation
• zero-shot
• RGB vision
• robotic policy

Innovation

• zero-shot sim-to-real
• deformable manipulation
• photometric augmentation
• topology-aware trajectory synthesis
• VLA policy