PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation

2026-03-04

AI Summary

This work addresses the challenges of scarce high-quality tactile strategy data in real-world settings and the difficulty of transferring dexterous manipulation skills from simulation to reality. To overcome these issues, the authors propose the Privileged Tactile Latent Distillation (PTLD) method, which leverages privileged sensors in the real world to collect tactile policy data and employs knowledge distillation to train a state estimator that relies solely on tactile inputs. This estimator enhances proprioceptive policies learned in simulation, enabling efficient sim-to-real transfer without requiring explicit tactile simulation. PTLD represents the first integration of privileged learning with tactile latent distillation for dexterous manipulation. Experimental results demonstrate a 182% performance improvement on in-hand rotation tasks and a 57% increase in success rate for tactile-driven in-hand reorientation tasks.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Tactile dexterous manipulation is essential to automating complex household tasks, yet learning effective control policies remains a challenge. While recent work has relied on imitation learning, obtaining high quality demonstrations for multi-fingered hands via robot teleoperation or kinesthetic teaching is prohibitive. Alternatively, with reinforcement we can learn skills in simulation, but fast and realistic simulation of tactile observations is challenging. To bridge this gap, we introduce PTLD: sim-to-real Privileged Tactile Latent Distillation, a novel approach to learning tactile manipulation skills without requiring tactile simulation. Instead of simulating tactile sensors or relying purely on proprioceptive policies to transfer zero-shot sim-to-real, our key idea is to leverage privileged sensors in the real world to collect real-world tactile policy data. This data is then used to distill a robust state estimator that operates on tactile input. We demonstrate from our experiments that PTLD can be used to improve proprioceptive manipulation policies trained in simulation significantly by incorporating tactile sensing. On the benchmark in-hand rotation task, PTLD achieves a 182% improvement over a proprioception only policy. We also show that PTLD enables learning the challenging task of tactile in-hand reorientation where we see a 57% improvement in the number of goals reached over using proprioception alone. Website: https://akashsharma02.github.io/ptld-website/.

Problem

• tactile manipulation
• sim-to-real transfer
• dexterous manipulation
• reinforcement learning
• policy distillation

Innovation

• tactile manipulation
• sim-to-real transfer
• privileged distillation
• dexterous manipulation
• latent state estimation