Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map

2026-02-25

AI Summary

This work addresses the tactile domain gap between simulation and reality that hinders the deployment of vision-based tactile sensors in robotic dexterous manipulation. Existing approaches often sacrifice physical fidelity for computational efficiency or vice versa. To bridge this gap, the authors propose Tacmap, a novel framework that introduces a unified geometric representation based on penetration depth. In simulation, Tacmap generates geometrically consistent depth maps from 3D intersection volumes, while in the real world, it leverages an automated data collection system to learn a mapping from raw tactile images to corresponding ground-truth depth maps. This alignment strategy preserves physical consistency across domains without compromising computational efficiency. Experimental results demonstrate that Tacmap produces deformation maps closely matching real-world measurements, and policies trained using its simulated outputs achieve zero-shot transfer to physical robots without fine-tuning.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Vision-Based Tactile Sensors (VBTS) are essential for achieving dexterous robotic manipulation, yet the tactile sim-to-real gap remains a fundamental bottleneck. Current tactile simulations suffer from a persistent dilemma: simplified geometric projections lack physical authenticity, while high-fidelity Finite Element Methods (FEM) are too computationally prohibitive for large-scale reinforcement learning. In this work, we present Tacmap, a high-fidelity, computationally efficient tactile simulation framework anchored in volumetric penetration depth. Our key insight is to bridge the tactile sim-to-real gap by unifying both domains through a shared deform map representation. Specifically, we compute 3D intersection volumes as depth maps in simulation, while in the real world, we employ an automated data-collection rig to learn a robust mapping from raw tactile images to ground-truth depth maps. By aligning simulation and real-world in this unified geometric space, Tacmap minimizes domain shift while maintaining physical consistency. Quantitative evaluations across diverse contact scenarios demonstrate that Tacmap's deform maps closely mirror real-world measurements. Moreover, we validate the utility of Tacmap through an in-hand rotation task, where a policy trained exclusively in simulation achieves zero-shot transfer to a physical robot.

Problem

• tactile sim-to-real gap
• vision-based tactile sensors
• tactile simulation
• domain shift
• physical authenticity

Innovation

• tactile simulation
• sim-to-real transfer
• penetration depth map
• vision-based tactile sensing
• deform map