ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation

2025-06-13

AI Summary

This work addresses the challenge of jointly estimating object pose and external contact points during dexterous manipulation. We propose an object-centric neural implicit representation that fuses high-resolution tactile sensing with vision. Our method explicitly models object geometry as a signed distance field (SDF) and parameterizes tactile shear force distributions as a Neural Shear Field, enabling explicit 3D spatial registration of contact locations. A multimodal feature fusion architecture is designed to support sim-to-real zero-shot transfer. Experiments in both simulation and on real robotic hardware demonstrate that our approach achieves high-accuracy pose estimation and external contact localization under partial observability and sensory noise. It significantly improves robustness and cross-domain generalization for contact-rich dexterous manipulation tasks.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Mastering dexterous, contact-rich object manipulation demands precise estimation of both in-hand object poses and external contact locations$unicode{x2013}$tasks particularly challenging due to partial and noisy observations. We present ViTaSCOPE: Visuo-Tactile Simultaneous Contact and Object Pose Estimation, an object-centric neural implicit representation that fuses vision and high-resolution tactile feedback. By representing objects as signed distance fields and distributed tactile feedback as neural shear fields, ViTaSCOPE accurately localizes objects and registers extrinsic contacts onto their 3D geometry as contact fields. Our method enables seamless reasoning over complementary visuo-tactile cues by leveraging simulation for scalable training and zero-shot transfers to the real-world by bridging the sim-to-real gap. We evaluate our method through comprehensive simulated and real-world experiments, demonstrating its capabilities in dexterous manipulation scenarios.

Problem

• Estimating in-hand object poses and external contact locations accurately
• Fusing vision and tactile feedback for object manipulation
• Bridging sim-to-real gap for zero-shot transfer

Innovation

• Fuses vision and high-resolution tactile feedback
• Represents objects as signed distance fields
• Bridges sim-to-real gap for zero-shot transfer