RoboCopilot: Human-in-the-loop Interactive Imitation Learning for Robot Manipulation

2025-03-10

AI Summary

Current dual-arm robotic skill learning faces two key bottlenecks: low data efficiency in passive imitation learning and the absence of an interactive interface supporting active human instruction. To address these, we propose an active imitation learning system featuring a novel compliance-enabled bidirectional teleoperation architecture, enabling real-time, smooth, and dynamic control authority transfer between human instructors and autonomous policies during complex bimanual tasks—such as bottle opening and collaborative assembly. The system integrates human-in-the-loop imitation learning, sim-to-real policy transfer, and real-time strategy arbitration. Evaluated on both simulation and physical platforms, it achieves a 42% improvement in task success rate and a 67% reduction in human intervention, significantly enhancing teaching efficiency and cross-task generalization capability.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Interaction Techniques and Devices

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Learning from human demonstration is an effective approach for learning complex manipulation skills. However, existing approaches heavily focus on learning from passive human demonstration data for its simplicity in data collection. Interactive human teaching has appealing theoretical and practical properties, but they are not well supported by existing human-robot interfaces. This paper proposes a novel system that enables seamless control switching between human and an autonomous policy for bi-manual manipulation tasks, enabling more efficient learning of new tasks. This is achieved through a compliant, bilateral teleoperation system. Through simulation and hardware experiments, we demonstrate the value of our system in an interactive human teaching for learning complex bi-manual manipulation skills.

Problem

• Enables seamless human-autonomous control switching
• Improves learning of bi-manual manipulation tasks
• Supports interactive human teaching via teleoperation

Innovation

• Seamless human-autonomous control switching
• Compliant bilateral teleoperation system
• Interactive human teaching for bi-manual tasks