HIL: Hybrid Imitation Learning of Diverse Parkour Skills from Videos

2025-05-19

AI Summary

Existing data-driven approaches struggle to enable physics-based simulated characters to perform complex, coherent human behaviors naturally in novel environments—particularly exhibiting limited cross-scene generalization and multi-skill coordination. To address this, we propose the first hybrid framework integrating motion tracking with adversarial imitation learning, enabling self-supervised acquisition of diverse parkour skills directly from Internet videos and yielding a unified, environment-agnostic controller. Our method employs parallel multi-task training, agent-centric scene representation, motion-capture-guided skill reproduction, and GAN-based adversarial imitation learning. Experiments demonstrate that the resulting controller significantly improves motion naturalness and skill diversity, achieving higher task completion rates than current state-of-the-art methods across multiple challenging parkour scenarios.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Motion & Tracking
• Multiagent Systems: Adversarial Agents

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Recent data-driven methods leveraging deep reinforcement learning have been an effective paradigm for developing controllers that enable physically simulated characters to produce natural human-like behaviors. However, these data-driven methods often struggle to adapt to novel environments and compose diverse skills coherently to perform more complex tasks. To address these challenges, we propose a hybrid imitation learning (HIL) framework that combines motion tracking, for precise skill replication, with adversarial imitation learning, to enhance adaptability and skill composition. This hybrid learning framework is implemented through parallel multi-task environments and a unified observation space, featuring an agent-centric scene representation to facilitate effective learning from the hybrid parallel environments. Our framework trains a unified controller on parkour data sourced from Internet videos, enabling a simulated character to traverse through new environments using diverse and life-like parkour skills. Evaluations across challenging parkour environments demonstrate that our method improves motion quality, increases skill diversity, and achieves competitive task completion compared to previous learning-based methods.

Problem

• Adapting to novel environments with diverse skills
• Combining precise skill replication and adaptability
• Enhancing motion quality and skill diversity

Innovation

• Hybrid imitation learning combines tracking and adversarial imitation
• Parallel multi-task environments enhance adaptability and skill composition
• Agent-centric scene representation facilitates effective learning