Spatial Orchestra: Locomotion Music Instruments through Spatial Exploration

2025-10-27

AI Summary

This study addresses the challenge of lowering barriers to musical creation by investigating cross-modal coupling between spatial perception and musical performance. We propose an augmented reality (AR)-based spatialized music interaction system in which users trigger cello sound samples in real time by physically walking toward and touching color-coded virtual note bubbles. The system leverages a standalone AR headset, high-precision spatial tracking, and 3D audio rendering to generate an immersive, body-motion-responsive soundscape—dynamically modulating timbre, pitch, and spatialization based on user position, orientation, and gait rhythm. Its key innovation lies in mapping walking tempo, spatial trajectories, and harmonic progressions in real time, enabling improvisational spatial composition without prior music-theoretic knowledge. A user study demonstrates significant improvements in engagement and expressive freedom, empirically validating the efficacy and generalizability of embodied movement as a driver for auditory art generation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Humans and AI: Interaction Techniques and Devices
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

Spatial Orchestra demonstrates how easy it is to play musical instruments using basic input like natural locomotion, which is accessible to most. Unlike many musical instruments, our work allows individuals of all skill levels to effortlessly create music by walking into virtual bubbles. Our Augmented Reality experience involves interacting with ever-shifting sound bubbles that the user engages with by stepping into color-coded bubbles within the assigned area using a standalone AR headset. Each bubble corresponds to a cello note, and omits sound from the center of the bubble, and lets the user hear and express in spatial audio, effectively transforming participants into musicians. This interactive element enables users to explore the intersection of spatial awareness, musical rhythm that extends to bodily expression through playful movements and dance-like gestures within the bubble-filled environment. This unique experience illuminates the intricate relationship between spatial awareness and the art of musical performance.

Problem

• Creating accessible music instruments through natural locomotion
• Enabling musical expression via spatial interaction with virtual bubbles
• Exploring spatial awareness and bodily movement in music performance

Innovation

• AR bubbles trigger music via natural locomotion
• Color-coded spatial audio bubbles map to cello notes
• Standalone headset enables movement-based musical expression