Generation of Real-time Robotic Emotional Expressions Learning from Human Demonstration in Mixed Reality

2025-08-12

AI Summary

This study addresses the challenge of enabling robots to generate realistic, diverse, and responsive emotional expressions in real-time during human–robot interaction. We propose a novel method integrating mixed reality (MR) with flow-matching generative modeling: MR-based first-person teleoperation by human experts, combined with motion capture, is used to record facial, head, and upper-body emotional behaviors, which are then mapped onto robotic actuators (e.g., eyes, ears, neck, arms); a conditional flow-matching model synthesizes coherent, multimodal emotional behaviors in real time, guided by both affective state and dynamic environmental cues (e.g., moving objects). To our knowledge, this is the first work to jointly leverage MR-enabled demonstration learning and flow-matching generation, significantly improving expressiveness and responsiveness. Preliminary experiments validate the approach’s effectiveness in terms of real-time performance, behavioral diversity, and contextual adaptability.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Motion & Tracking

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Expressive behaviors in robots are critical for effectively conveying their emotional states during interactions with humans. In this work, we present a framework that autonomously generates realistic and diverse robotic emotional expressions based on expert human demonstrations captured in Mixed Reality (MR). Our system enables experts to teleoperate a virtual robot from a first-person perspective, capturing their facial expressions, head movements, and upper-body gestures, and mapping these behaviors onto corresponding robotic components including eyes, ears, neck, and arms. Leveraging a flow-matching-based generative process, our model learns to produce coherent and varied behaviors in real-time in response to moving objects, conditioned explicitly on given emotional states. A preliminary test validated the effectiveness of our approach for generating autonomous expressions.

Problem

• Generate real-time robotic emotional expressions from human demonstrations
• Map human facial and body gestures to robotic components
• Produce coherent behaviors conditioned on emotional states

Innovation

• Real-time robotic emotional expressions generation
• Learning from human MR demonstrations
• Flow-matching-based generative process