SimShear: Sim-to-Real Shear-based Tactile Servoing

2025-08-28

AI Summary

Modeling shear dynamics in interactions with dynamic objects is computationally expensive and hinders real-time tactile simulation. Method: This paper proposes a shear-free tactile control framework, centered on shPix2pix—a shear-conditioned U-Net generative adversarial network that synthesizes high-fidelity tactile images with physically realistic shear deformations from shear-agnostic simulations. The approach integrates vision–tactile sensing, shear-encoded latent vectors, and tactile servoing control. Contribution/Results: To our knowledge, this is the first method enabling shear-aware sim-to-real transfer in rigid-body simulation without explicit shear modeling. Evaluated on a low-cost dual-arm robotic platform, the system achieves stable tactile tracking and cooperative lifting with contact errors within 1–2 mm, demonstrating effectiveness and robustness. It breaks the conventional paradigm reliant on complex, physics-based shear modeling, enabling efficient and scalable tactile interaction control.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Low Level & Physics-based Vision
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: The sharing economy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We present SimShear, a sim-to-real pipeline for tactile control that enables the use of shear information without explicitly modeling shear dynamics in simulation. Shear, arising from lateral movements across contact surfaces, is critical for tasks involving dynamic object interactions but remains challenging to simulate. To address this, we introduce shPix2pix, a shear-conditioned U-Net GAN that transforms simulated tactile images absent of shear, together with a vector encoding shear information, into realistic equivalents with shear deformations. This method outperforms baseline pix2pix approaches in simulating tactile images and in pose/shear prediction. We apply SimShear to two control tasks using a pair of low-cost desktop robotic arms equipped with a vision-based tactile sensor: (i) a tactile tracking task, where a follower arm tracks a surface moved by a leader arm, and (ii) a collaborative co-lifting task, where both arms jointly hold an object while the leader follows a prescribed trajectory. Our method maintains contact errors within 1 to 2 mm across varied trajectories where shear sensing is essential, validating the feasibility of sim-to-real shear modeling with rigid-body simulators and opening new directions for simulation in tactile robotics.

Problem

• Simulating shear dynamics in tactile robotics without explicit modeling
• Transforming shear-free simulated images into realistic tactile equivalents
• Enabling precise tactile control tasks requiring shear sensing

Innovation

• Sim-to-real pipeline using shear-conditioned GAN
• Transforms simulated images with shear vector
• Enables tactile control without explicit shear modeling