TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objects

2026-03-14

AI Summary

Manipulating transparent objects with dexterity is challenging due to self-occlusion, depth noise, and missing visual information. To address these issues, this work proposes TransDex, the first approach to introduce Transformer-based self-supervised point cloud reconstruction pretraining into transparent object manipulation. By reconstructing complete 3D structures and incorporating a hierarchical visuo-tactile perception encoder with multi-round attention mechanisms, TransDex enables differentiated motion prediction for both robotic arms and dexterous hands. The method innovatively integrates fine-grained hierarchical perception with adaptive multimodal feature fusion, significantly enhancing policy generalization. Real-world robotic experiments demonstrate that TransDex outperforms existing baselines on transparent object manipulation tasks, validating the effectiveness of its individual components and the overall framework’s superiority.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

Dexterous manipulation enables complex tasks but suffers from self-occlusion, severe depth noise, and depth information loss when manipulating transparent objects. To solve this problem, this paper proposes TransDex, a 3D visuo-tactile fusion motor policy based on point cloud reconstruction pre-training. Specifically, we first propose a self-supervised point cloud reconstruction pre-training approach based on Transformer. This method accurately recovers the 3D structure of objects from interactive point clouds of dexterous hands, even when random noise and large-scale masking are added. Building on this, TransDex is constructed in which perceptual encoding adopts a fine-grained hierarchical scheme and multi-round attention mechanisms adaptively fuse features of the robotic arm and dexterous hand to enable differentiated motion prediction. Results from transparent object manipulation experiments conducted on a real robotic system demonstrate that TransDex outperforms existing baseline methods. Further analysis validates the generalization capabilities of TransDex and the effectiveness of its individual components.

Problem

• dexterous manipulation
• transparent objects
• depth noise
• self-occlusion
• depth information loss

Innovation

• point cloud reconstruction
• visuo-tactile fusion
• dexterous manipulation
• transparent objects
• self-supervised pre-training