Haptic-ACT: Bridging Human Intuition with Compliant Robotic Manipulation via Immersive VR

2024-09-18

arXiv.org

AI Summary

High-cost robot teleoperation demonstrations and challenges in modeling compliant manipulation motivate this work. We propose Haptic-ACT, a tactile-enhanced VR teleoperation framework for imitation learning. Our method introduces an immersive VR teleoperation system capable of real-time force feedback acquisition, and pioneers a novel architecture that integrates tactile signal-driven action segmentation with a Transformer backbone to explicitly model contact dynamics and temporal action dependencies. The framework unifies MuJoCo simulation, haptic rendering, and deployment on physical robots. Experiments demonstrate that Haptic-ACT significantly improves compliant manipulation performance over baseline ACT in both simulation and real-robot settings: demonstrator fingertip contact forces decrease by 32.7%, and the system successfully completes 50 fine-grained pick-and-place tasks. These results validate the critical role of tactile guidance in acquiring high-fidelity demonstration data for robust imitation learning.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Robotic manipulation is essential for the widespread adoption of robots in industrial and home settings and has long been a focus within the robotics community. Advances in artificial intelligence have introduced promising learning-based methods to address this challenge, with imitation learning emerging as particularly effective. However, efficiently acquiring high-quality demonstrations remains a challenge. In this work, we introduce an immersive VR-based teleoperation setup designed to collect demonstrations from a remote human user. We also propose an imitation learning framework called Haptic Action Chunking with Transformers (Haptic-ACT). To evaluate the platform, we conducted a pick-and-place task and collected 50 demonstration episodes. Results indicate that the immersive VR platform significantly reduces demonstrator fingertip forces compared to systems without haptic feedback, enabling more delicate manipulation. Additionally, evaluations of the Haptic-ACT framework in both the MuJoCo simulator and on a real robot demonstrate its effectiveness in teaching robots more compliant manipulation compared to the original ACT. Additional materials are available at https://sites.google.com/view/hapticact.

Problem

• Efficiently acquiring high-quality robotic manipulation demonstrations.
• Reducing demonstrator fingertip forces using immersive VR with haptic feedback.
• Teaching robots compliant manipulation via the Haptic-ACT framework.

Innovation

• Immersive VR teleoperation for demonstration collection
• Haptic-ACT framework for compliant robotic manipulation
• Reduced fingertip forces with haptic feedback