Autonomous Human-Robot Interaction via Operator Imitation

2025-04-03

AI Summary

Remote robotic teleoperation heavily relies on operator expertise and social intuition, limiting accessibility and scalability. Method: This paper proposes an autonomous human–robot interaction framework based on operator behavior cloning. We introduce the first unified multimodal Transformer architecture that jointly models continuous motion control and discrete interaction commands—including affective expressions—by integrating diffusion modeling and classification. Additionally, we incorporate real-time pose awareness and cross-platform zero-shot transfer to enable deployment across heterogeneous robotic platforms. Results: Experiments in both simulation and real-world settings demonstrate expert-level teleoperation performance. Users accurately recognize the robot’s affective states, confirming enhanced interaction naturalness. The framework significantly improves generalization across tasks and platforms while reducing dependence on operator experience, thereby advancing accessible, socially intelligent teleoperation systems.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

Teleoperated robotic characters can perform expressive interactions with humans, relying on the operators' experience and social intuition. In this work, we propose to create autonomous interactive robots, by training a model to imitate operator data. Our model is trained on a dataset of human-robot interactions, where an expert operator is asked to vary the interactions and mood of the robot, while the operator commands as well as the pose of the human and robot are recorded. Our approach learns to predict continuous operator commands through a diffusion process and discrete commands through a classifier, all unified within a single transformer architecture. We evaluate the resulting model in simulation and with a user study on the real system. We show that our method enables simple autonomous human-robot interactions that are comparable to the expert-operator baseline, and that users can recognize the different robot moods as generated by our model. Finally, we demonstrate a zero-shot transfer of our model onto a different robotic platform with the same operator interface.

Problem

• Autonomous human-robot interaction via operator imitation
• Predict operator commands using diffusion and classifier
• Zero-shot transfer to different robotic platforms

Innovation

• Imitation learning from expert operator data
• Unified transformer with diffusion and classifier
• Zero-shot transfer to different robotic platform