NormalFlow: Fast, Robust, and Accurate Contact-Based Object 6DoF Pose Tracking With Vision-Based Tactile Sensors

2024-12-12

IEEE Robotics and Automation Letters

AI Summary

This work addresses key challenges in 6-degree-of-freedom (6DoF) object pose tracking using vision-based tactile sensors: poor robustness, strong reliance on surface texture, and insufficient real-time performance. We propose a geometric consistency optimization method grounded in surface normal flow (NormalFlow) modeling—the first to integrate normal flow into tactile tracking frameworks. By minimizing inter-frame discrepancies in estimated tactile surface normals, our approach infers contact motion without requiring texture priors, enabling reliable tracking on textureless or weakly textured surfaces. The method jointly refines high-fidelity normal estimation and contact motion modeling. In a 360° bead-rolling experiment, it achieves a rotation error of only 2.5°, significantly outperforming existing baselines. Moreover, it attains state-of-the-art tactile-driven 3D reconstruction accuracy while maintaining real-time operation, high precision, and long-term stability.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Security and Privacy: Tracking, profiling, and countermeasures against them
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Tactile sensing is crucial for robots aiming to achieve human-level dexterity. Among tactile-dependent skills, tactile-based object tracking serves as the cornerstone for many tasks, including manipulation, in-hand manipulation, and 3D reconstruction. In this work, we introduce NormalFlow, a fast, robust, and real-time tactile-based 6DoF tracking algorithm. Leveraging the precise surface normal estimation of vision-based tactile sensors, NormalFlow determines object movements by minimizing discrepancies between the tactile-derived surface normals. Our results show that NormalFlow consistently outperforms competitive baselines and can track low-texture objects like table surfaces. For long-horizon tracking, we demonstrate when rolling the sensor around a bead for 360 degrees, NormalFlow maintains a rotational tracking error of 2.5 degrees. Additionally, we present state-of-the-art tactile-based 3D reconstruction results, showcasing the high accuracy of NormalFlow. We believe NormalFlow unlocks new possibilities for high-precision perception and manipulation tasks that involve interacting with objects using hands.

Problem

• Develops a fast, robust 6DoF object tracking algorithm.
• Enables precise tactile-based object movement tracking.
• Improves 3D reconstruction accuracy using tactile sensors.

Innovation

• Real-time 6DoF tracking using tactile sensors
• Minimizes discrepancies with tactile-derived surface normals
• Achieves high accuracy in 3D reconstruction tasks