A Design Space for Live Music Agents

2026-02-04

AI Summary

This study addresses the fragmented progress and limited collaboration in real-time musical agent research, which stems from the absence of a unified interdisciplinary framework. By integrating perspectives from human-computer interaction, artificial intelligence, and computer music, this work presents the first systematic analysis of 184 real-time musical intelligent systems through comprehensive literature and video review. It constructs a multidimensional design space encompassing usage scenarios, interaction paradigms, technical implementations, and ecosystem considerations. The contribution includes a novel structured conceptual framework that elucidates key trends and critical research gaps, alongside the release of an open-access repository. Together, these resources offer both theoretical grounding and practical foundations for advancing human–AI collaborative music creation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Multiagent Systems: Agent/AI Theories and Architectures
• Humans and AI: Intelligent User Interfaces

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Live music provides a uniquely rich setting for studying creativity and interaction due to its spontaneous nature. The pursuit of live music agents--intelligent systems supporting real-time music performance and interaction--has captivated researchers across HCI, AI, and computer music for decades, and recent advancements in AI suggest unprecedented opportunities to evolve their design. However, the interdisciplinary nature of music has led to fragmented development across research communities, hindering effective communication and collaborative progress. In this work, we bring together perspectives from these diverse fields to map the current landscape of live music agents. Based on our analysis of 184 systems across both academic literature and video, we develop a comprehensive design space that categorizes dimensions spanning usage contexts, interactions, technologies, and ecosystems. By highlighting trends and gaps in live music agents, our design space offers researchers, designers, and musicians a structured lens to understand existing systems and shape future directions in real-time human-AI music co-creation. We release our annotated systems as a living artifact at https://live-music-agents.github.io.

Problem

• live music agents
• interdisciplinary fragmentation
• real-time human-AI music co-creation
• design space
• research collaboration

Innovation

• design space
• live music agents
• human-AI co-creation
• real-time interaction
• interdisciplinary synthesis