Tactile Sim2Real without Tactile Simulation via Bottlenecked Latent Reconstruction

2026-08-16

AI Summary

This study addresses the challenge of low simulation fidelity caused by tactile sensor diversity, which hinders sim-to-real transfer. We propose SBLR, a framework enabling sensor-agnostic transfer without requiring specific sensor simulation. By constructing a universal oracle sensor with bottleneck latent reconstruction and employing rectified flow transformation networks to align real and simulated latent spaces, SBLR facilitates robust generalization. The method utilizes a two-stage training strategy with unpaired data learning to achieve zero-shot deployment in tasks such as peg insertion. Hardware experiments demonstrate success rates of 85%–97.5%, outperforming physics-based simulation baselines by 7.5%–15%. These results effectively validate the framework’s generalization capability and practical utility in overcoming sensor heterogeneity for tactile sim-to-real applications.

Abstract

Robot sensor designs, particularly tactile sensors, are highly diverse and evolve rapidly. Modeling each sensor in simulation demands substantial domain expertise and computational approximations can degrade the fidelity of the simulated signals. We propose Sim2Real via Bottlenecked Latent Reconstruction (SBLR), a framework that avoids sensor-specific simulation entirely by (1) training policies on a simulator-native oracle sensor that is easy to construct without modeling any particular sensor (e.g. we use a point-cloud and finger-tip forces as a tactile oracle), and (2) aligning real sensor latent embeddings to those of the oracle sensor at inference time. Policy training proceeds in two-stage: the policy first learns from the oracle sensor latents, then a bottlenecked latent reconstruction adapts it to the information loss expected when using the real sensor instead of the oracle. The alignment between oracle and real sensor is learned from unpaired random-play data collected in both simulation and the real world, using rectified-flow-based transformation networks trained on nearest-neighbor pseudo-pairs. Simulation experiments on three contact-rich tasks show that SBLR matches or approaches the performance of an oracle with direct access to tactile simulation. Hardware experiments on Peg Insertion and Gear Meshing with GelSight Mini and DIGIT sensors demonstrate 85-97.5% zero-shot success without requiring any sensor-specific modeling or calibration, outperforming a physics-based tactile simulation baseline by 7.5-15%.

Problem

• Tactile Sim2Real
• Sensor Modeling
• Simulation Fidelity
• Domain Adaptation

Innovation

• Tactile Sim2Real
• Bottlenecked Latent Reconstruction
• Oracle Sensor
• Unpaired Data Alignment
• Rectified Flow