LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music

2026-06-30

AI Summary

This work proposes a multimodal interaction framework that overcomes the limitations of traditional human-robot interaction, which often relies on predefined commands and lacks naturalness and expressiveness. For the first time, a large language model is leveraged to fuse semantic speech, deictic gestures, and musical beat cues through context-aware reasoning, generating coherent and expressive motion sequences for a quadruped robot. The system integrates modules for speech transcription, gesture recognition, and beat detection, employing structured prompt templates to guide the large language model and executing the resulting action queue in real time via ROS. Experimental results demonstrate that this approach significantly enhances the naturalness, flexibility, and creativity of human-robot interaction.

Abstract

The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats. Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection. These processed inputs are contextualized using prompt templates and fed into a LLM. The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions. This sequence is dispatched to an action queue for execution on a quadruped robot over ROS. The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music. This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner.

Problem

• human-robot interaction
• multimodal input
• action synthesis
• natural communication
• expressive robotics

Innovation

• Large Language Models
• Multimodal Interaction
• Robotic Action Synthesis
• Human-Robot Interaction
• Gesture and Speech Fusion