Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation

2026-05-27

AI Summary

This work addresses the loss of rich contact details in sim-to-real transfer due to oversimplified tactile representations by proposing a physics-informed center-of-pressure (CoP) tactile representation. This approach preserves dense contact information while enhancing transfer robustness and enables sensor orientation calibration without requiring ground-truth force measurements through differentiable dynamics. Integrated with reinforcement learning and multi-fingered dexterous hand control, the system achieves zero-shot sim-to-real transfer on vision-deprived tasks such as plug insertion and ball balancing. It significantly outperforms binary contact and raw tactile baselines, with the learned policy implicitly encoding physical properties of objects, including mass.

Abstract

A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data. Sim-to-real reinforcement learning offers a scalable alternative, but the simulation-reality gap prevents information-dense modalities like touch from being effectively used. Existing sim-to-real methods often mitigate this gap by simplifying tactile data into coarse low-dimensional features -- sacrificing the richness required for complex manipulation. In this work, we introduce Center-of-Pressure (CoP), an effective tactile representation grounded in physical principles that preserves dense contact information while maintaining robustness for sim-to-real transfer. To support this representation, we propose a sensor calibration scheme based on differentiable dynamics, enabling the estimation of taxel orientations without requiring ground-truth force measurements. We evaluate CoP on two blind, challenging contact-rich manipulation tasks: peg-in-hole insertion and ball balancing. Across both tasks, policies conditioned on CoP achieve zero-shot sim-to-real transfer on a multi-fingered hand, and outperform both coarse binary-contact and raw-taxel baselines. Analysis of learned policy states further suggests that CoP-conditioned policies encode task-relevant physical properties, such as object mass, as an emergent byproduct of control.

Problem

• dexterous manipulation
• sim-to-real
• tactile sensing
• contact-rich tasks
• simulation-reality gap

Innovation

• Center-of-Pressure
• sim-to-real transfer
• dexterous manipulation
• tactile representation
• differentiable dynamics