Train Robots in a JIF: Joint Inverse and Forward Dynamics with Human and Robot Demonstrations

2025-03-15

AI Summary

Robot skill learning is hindered by reliance on large quantities of costly robot demonstration data—especially for tasks requiring tactile feedback. Method: We propose a self-supervised pretraining framework leveraging multimodal human demonstrations (vision + touch), introducing the first unified architecture jointly modeling inverse and forward dynamics to learn task-specific latent state representations and enable efficient transfer from human demonstrations to robot policies. The method supports multimodal inputs and requires only a small number of robot demonstrations for high-performance fine-tuning. Contribution/Results: Our approach significantly improves data efficiency, reducing dependence on expensive robot teleoperation data. Experiments demonstrate substantial gains in sample efficiency and generalization across diverse manipulation tasks. It establishes a scalable paradigm for human-in-the-loop robotic learning, enabling robust policy adaptation with minimal robot-specific supervision. This work bridges the gap between rich human sensory demonstrations and practical robot deployment, advancing the frontier of imitation learning and embodied AI.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Pre-training on large datasets of robot demonstrations is a powerful technique for learning diverse manipulation skills but is often limited by the high cost and complexity of collecting robot-centric data, especially for tasks requiring tactile feedback. This work addresses these challenges by introducing a novel method for pre-training with multi-modal human demonstrations. Our approach jointly learns inverse and forward dynamics to extract latent state representations, towards learning manipulation specific representations. This enables efficient fine-tuning with only a small number of robot demonstrations, significantly improving data efficiency. Furthermore, our method allows for the use of multi-modal data, such as combination of vision and touch for manipulation. By leveraging latent dynamics modeling and tactile sensing, this approach paves the way for scalable robot manipulation learning based on human demonstrations.

Problem

• High cost and complexity of robot data collection
• Limited tactile feedback in robot-centric datasets
• Need for scalable manipulation learning from human demonstrations

Innovation

• Joint inverse and forward dynamics learning
• Multi-modal human demonstrations pre-training
• Latent dynamics modeling with tactile sensing