V-HOP: Visuo-Haptic 6D Object Pose Tracking

2025-02-24

AI Summary

Existing visuo-haptic 6D pose estimation methods suffer from poor generalization to real-world scenarios, temporal inconsistency across frames, and limited adaptability to diverse grippers and tactile sensors. This paper proposes a real-time, operation-oriented framework for 6D object pose tracking. First, we design a unified tactile representation compatible with heterogeneous grippers and both taxel-based and vision-based tactile sensors. Second, we introduce the first end-to-end visuo-haptic Transformer tracker, enabling sequence-level coherent pose estimation and cross-actuator generalization. Third, we integrate multimodal feature alignment with closed-loop motion planning. Experiments demonstrate that our method significantly outperforms state-of-the-art purely visual trackers in real-world settings. It maintains high accuracy across unseen grippers, novel objects, and heterogeneous tactile sensor modalities, and successfully enables high-precision closed-loop manipulation tasks.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Humans naturally integrate vision and haptics for robust object perception during manipulation. The loss of either modality significantly degrades performance. Inspired by this multisensory integration, prior object pose estimation research has attempted to combine visual and haptic/tactile feedback. Although these works demonstrate improvements in controlled environments or synthetic datasets, they often underperform vision-only approaches in real-world settings due to poor generalization across diverse grippers, sensor layouts, or sim-to-real environments. Furthermore, they typically estimate the object pose for each frame independently, resulting in less coherent tracking over sequences in real-world deployments. To address these limitations, we introduce a novel unified haptic representation that effectively handles multiple gripper embodiments. Building on this representation, we introduce a new visuo-haptic transformer-based object pose tracker that seamlessly integrates visual and haptic input. We validate our framework in our dataset and the Feelsight dataset, demonstrating significant performance improvement on challenging sequences. Notably, our method achieves superior generalization and robustness across novel embodiments, objects, and sensor types (both taxel-based and vision-based tactile sensors). In real-world experiments, we demonstrate that our approach outperforms state-of-the-art visual trackers by a large margin. We further show that we can achieve precise manipulation tasks by incorporating our real-time object tracking result into motion plans, underscoring the advantages of visuo-haptic perception. Our model and dataset will be made open source upon acceptance of the paper. Project website: https://lhy.xyz/projects/v-hop/

Problem

• Enhance object pose tracking with visuo-haptic integration.
• Improve generalization across diverse grippers and sensors.
• Achieve robust real-time manipulation using unified perception.

Innovation

• Unified haptic representation for multiple grippers
• Visuo-haptic transformer-based pose tracker
• Real-time object tracking for precise manipulation