Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-Shot Real-World Deployment

2026-03-30

AI Summary

This work addresses the challenge in tactile reinforcement learning where high-fidelity physics simulation and computational efficiency are difficult to balance, hindering effective sim-to-real transfer. To this end, the authors propose Tac2Real, a lightweight visuo-tactile simulation framework that integrates a preconditioned nonlinear conjugate gradient incremental potential contact (PNCG-IPC) solver with a multi-node, multi-GPU high-throughput parallel architecture, enabling high-speed online training. Additionally, they introduce TacAlign, a strategy that systematically aligns both structured and stochastic domain shifts between simulation and reality. Evaluated on real-world peg-in-hole tasks, the approach achieves, for the first time, efficient and robust zero-shot transfer of tactile policies, significantly improving success rates and demonstrating its effectiveness and real-time capability in contact-rich manipulation scenarios.

Abstract

Visuotactile sensors are indispensable for contact-rich robotic manipulation tasks. However, policy learning with tactile feedback in simulation, especially for online reinforcement learning (RL), remains a critical challenge, as it demands a delicate balance between physics fidelity and computational efficiency. To address this challenge, we present Tac2Real, a lightweight visuotactile simulation framework designed to enable efficient online RL training. Tac2Real integrates the Preconditioned Nonlinear Conjugate Gradient Incremental Potential Contact (PNCG-IPC) method with a multi-node, multi-GPU high-throughput parallel simulation architecture, which can generate marker displacement fields at interactive rates. Meanwhile, we propose a systematic approach, TacAlign, to narrow both structured and stochastic sources of domain gap, ensuring a reliable zero-shot sim-to-real transfer. We further evaluate Tac2Real on the contact-rich peg insertion task. The zero-shot transfer results achieve a high success rate in the real-world scenario, verifying the effectiveness and robustness of our framework. The project page is: https://ningyurichard.github.io/tac2real-project-page/

Problem

• visuotactile simulation
• online reinforcement learning
• sim-to-real transfer
• contact-rich manipulation
• domain gap

Innovation

• visuotactile simulation
• online reinforcement learning
• zero-shot sim-to-real transfer
• PNCG-IPC
• domain gap alignment