Zero-Shot Sim-to-Real Contact-Rich Assembly via Proprioception-Anchored Cross-Modal Pretraining

2026-09-07

AI Summary

为解决接触丰富装配任务中的模拟到现实迁移问题，提出了一种基于本体感觉锚定的跨模态预训练方法PACE，提高了策略在实际硬件上的执行成功率。

Abstract

Contact-rich assembly remains challenging because it requires submillimeter spatial accuracy and reliable interpretation of forces during sustained contact. Although simulation-based reinforcement learning offers a scalable training paradigm, discrepancies in visual observations, contact dynamics, and force/torque (F/T) measurements often limit policy transfer. We observe that proprioception is comparatively consistent across domains because calibrated joint positions and consistently computed joint velocities align closely between simulation and hardware. Based on this observation, we present PACE (Proprioception-Anchored Cross-Modal Encoder), which supervises temporal visual and F/T representations by predicting proprioceptive state transitions. Static domain-specific factors, including lighting, texture, and sensor bias, contain little information about joint motion; the proposed objective therefore encourages the encoder to suppress these factors while retaining task-relevant motion cues. Policies trained on frozen PACE features are deployed on hardware without real-world fine-tuning or object-pose tracking. Across four contact-rich assembly tasks, PACE attains an average real-world success rate of 93.3\% and only a 2.7-percentage-point sim-to-real drop, meanwhile remaining robust to perturbations that substantially degrade pose-based and learned-fusion baselines.

Problem

• contact-rich assembly
• submillimeter spatial accuracy
• reliable interpretation of forces
• simulation-based reinforcement learning
• policy transfer

Innovation

• PACE
• Proprioception-Anchored Cross-Modal Encoder
• zero-shot sim-to-real transfer
• contact-rich assembly
• supervised learning