Co-policy: Responsive Human-Robot Co-Creation for Musical Performances

2026-06-18

AI Summary

This work addresses the challenge of effectively bridging semantic understanding and real-time executable actions in robot-augmented musical co-creation. The authors propose Co-policy, a novel framework that decouples semantic intent anchoring, constrained musical variation, and visuomotor execution for the first time. Their approach integrates a fine-tuned Qwen-VL planner (F-Qwen) for high-level semantic planning with a Gaussian Mixture Policy (GMP) for low-latency, multimodal action generation, enabling complementary responses under both musical and physical constraints. Real-world experiments on a carillon platform, complemented by expert evaluations, demonstrate that the proposed method significantly outperforms diffusion-based policies and ablation baselines in intent alignment, execution accuracy, and response frequency, thereby overcoming the limitations of conventional playback-oriented robotic systems.

Abstract

Art has long stood as a pivotal expression of human creativity. Embodied artificial intelligence offers a route for generative models to participate in that creativity through physical action rather than disembodied digital content. In robotic music co-creation, it is challenging to connect semantic musical understanding with real-time and physically executable performance. We present Co-policy, a framework for human-robot musical co-creation that separates semantic intent grounding, constrained musical variation, and visuomotor execution. To ground musical semantics, Co-policy uses pre-inference semantic anchors and a fine-tuned Qwen-vl planner (F-Qwen) to transform speech, live musical seeds, and visual observations into structured co-creation plans. To support low-latency execution, Co-policy introduces a Gaussian-Mixture Visuomotor Policy (GMP), implemented as a conditional mixture-density policy that maps target notes and visual context to multimodal robot actions in a single forward pass. Unlike robotic playback systems that merely reproduce user-specified notes, Co-policy generates complementary musical responses under both musical and physical constraints. Real-robot chime experiments, ablations, and expert evaluation show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines, supporting physically grounded action generation as a key requirement for embodied human-AI co-creation.

Problem

• human-robot co-creation
• musical semantics
• real-time execution
• embodied AI
• visuomotor policy

Innovation

• Co-policy
• visuomotor policy
• embodied AI
• musical co-creation
• semantic grounding