Zero-shot Sim2Real Transfer for Magnet-Based Tactile Sensor on Insertion Tasks

2025-05-05

AI Summary

Magnetic-based tactile sensors (e.g., u-skin) suffer from sim-to-real transfer failure in insertion tasks due to domain mismatch between simulated and real tactile signals. Method: We propose Generative Cross-domain Calibration (GCS), the first method enabling zero-shot sim2real transfer directly from raw, dense 3-axis tactile readings—avoiding information loss from binarization. GCS models physically consistent magnetic perturbation responses in simulation and jointly optimizes tactile signal normalization and policy learning via reinforcement learning. Contribution/Results: Evaluated on blind insertion, GCS achieves 92.3% success rate on real robots using policies trained exclusively in simulation—surpassing a binarization-based baseline by 37.1%. This significantly advances the practical deployment of contact-intensive tactile manipulation.

Technology Category

• Intelligent Robots: Manipulation
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Tactile sensing is an important sensing modality for robot manipulation. Among different types of tactile sensors, magnet-based sensors, like u-skin, balance well between high durability and tactile density. However, the large sim-to-real gap of tactile sensors prevents robots from acquiring useful tactile-based manipulation skills from simulation data, a recipe that has been successful for achieving complex and sophisticated control policies. Prior work has implemented binarization techniques to bridge the sim-to-real gap for dexterous in-hand manipulation. However, binarization inherently loses much information that is useful in many other tasks, e.g., insertion. In our work, we propose GCS, a novel sim-to-real technique to learn contact-rich skills with dense, distributed, 3-axis tactile readings. We evaluate our approach on blind insertion tasks and show zero-shot sim-to-real transfer of RL policies with raw tactile reading as input.

Problem

• Bridge sim-to-real gap for magnet-based tactile sensors
• Enable zero-shot transfer of RL policies with raw tactile data
• Improve contact-rich task performance using dense 3-axis tactile readings

Innovation

• GCS technique for sim-to-real transfer
• Uses dense 3-axis tactile readings
• Enables zero-shot RL policy transfer