TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer

2026-06-17

AI Summary

Existing tactile simulators struggle to accurately replicate the complex deformations and transduction mechanisms of real sensors, limiting sim-to-real transfer performance. This work proposes a multimodal representation learning framework that maps heterogeneous tactile signals—such as simulated penetration depth and real capacitive readings—into a shared latent space using modality-specific encoders. The model is trained with self-reconstruction, cross-reconstruction, and contrastive alignment losses, enabling zero-shot transfer without requiring high-fidelity simulation of raw sensory signals. By integrating multiphysics simulation to enrich embedding informativeness and leveraging a Warp-accelerated penalty-based contact model for computational efficiency, the approach achieves a 16.7% reduction in force prediction error and a 45.8% decrease in shape reconstruction error. It further demonstrates successful zero-shot cross-modal transfer across multiple downstream tasks and includes an open-sourced, efficient tactile simulation module.

Abstract

Tactile sensing provides direct measurements of contact interactions that are essential for robotic manipulation. However, current simulators lack the fidelity to faithfully model the complex deformation and transduction mechanics of tactile sensors, severely hindering sim-to-real transfer in robot learning pipelines. To address this challenge, we propose a multi-modal representation learning framework that aligns heterogeneous tactile modalities within a shared latent space, eliminating the need for accurate raw-signal simulation while preserving relevant contact information. Our approach employs modality-specific encoders to project diverse tactile observations, such as simulated penetration depth and real-world capacitance, into a common embedding space. The model is trained using self- and cross-reconstruction objectives alongside contrastive alignment, encouraging modality-invariant yet information-rich representations. We evaluate the learned embeddings on indenter shape identification, force prediction, and geometric reconstruction tasks, training exclusively in simulation and testing directly on real sensor measurements. Our results demonstrate zero-shot sim-to-real transfer across physically dissimilar representations. Furthermore, incorporating multi-physics simulation modalities yields more informative embeddings that transfer across diverse downstream tasks, demonstrating a 16.7% reduction in force prediction error and a 45.8% reduction in shape reconstruction error. Finally, we release an efficient Warp-based implementation of a penalty-based tactile simulation model for Isaac Lab, enabling scalable tactile data generation.

Problem

• tactile sensing
• sim-to-real transfer
• sensor simulation
• contact interaction
• robotic manipulation

Innovation

• tactile sim-to-real transfer
• shared latent space
• multi-modal representation learning
• physics-enriched embedding
• zero-shot transfer