NeuralTouch: Neural Descriptors for Precise Sim-to-Real Tactile Robot Control

2025-10-23

AI Summary

Visual imperfections—such as calibration errors and sparse or incomplete point clouds—induce pose estimation inaccuracies during robotic grasping, while existing tactile methods rely on predefined contact geometries, limiting generalization. Method: This paper proposes NeuralTouch, the first framework that jointly couples neural descriptor fields (NDFs) with tactile feedback to implicitly model contact geometry and guide a deep reinforcement learning policy—eliminating the need for explicit contact-type definitions. It integrates vision–tactile multimodal perception and enables zero-shot sim-to-real transfer. Contribution/Results: Evaluated on fine manipulation tasks—including peg-in-hole insertion and bottle-cap opening—NeuralTouch significantly improves grasping accuracy and robustness under sensory uncertainty. It outperforms state-of-the-art baselines, demonstrating superior generalization across diverse contact configurations and real-world deployment without task-specific adaptation.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Grasping accuracy is a critical prerequisite for precise object manipulation, often requiring careful alignment between the robot hand and object. Neural Descriptor Fields (NDF) offer a promising vision-based method to generate grasping poses that generalize across object categories. However, NDF alone can produce inaccurate poses due to imperfect camera calibration, incomplete point clouds, and object variability. Meanwhile, tactile sensing enables more precise contact, but existing approaches typically learn policies limited to simple, predefined contact geometries. In this work, we introduce NeuralTouch, a multimodal framework that integrates NDF and tactile sensing to enable accurate, generalizable grasping through gentle physical interaction. Our approach leverages NDF to implicitly represent the target contact geometry, from which a deep reinforcement learning (RL) policy is trained to refine the grasp using tactile feedback. This policy is conditioned on the neural descriptors and does not require explicit specification of contact types. We validate NeuralTouch through ablation studies in simulation and zero-shot transfer to real-world manipulation tasks--such as peg-out-in-hole and bottle lid opening--without additional fine-tuning. Results show that NeuralTouch significantly improves grasping accuracy and robustness over baseline methods, offering a general framework for precise, contact-rich robotic manipulation.

Problem

• Enhancing grasping accuracy through multimodal sensor fusion
• Overcoming inaccurate poses from vision-only methods
• Enabling precise contact-rich manipulation without predefined geometries

Innovation

• Integrates vision-based NDF with tactile sensing
• Uses deep RL policy conditioned on neural descriptors
• Enables zero-shot transfer to real-world tasks