Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation

2026-06-23

AI Summary

Existing generative music AI systems struggle to support real-time human-AI co-creation due to high inference latency and offline rendering paradigms. This work proposes a streaming consistency distillation framework that transforms text-to-music models into low-latency, responsive interactive instruments without requiring paired audio–latent data. By integrating prompt-driven online trajectory synthesis, a unified latent space, and a music-perception-aware consistency objective that fuses spectral and temporal differences, the method—combined with parameter-efficient fine-tuning—drastically reduces both generation steps and real-time factors while preserving timbral quality, transient fidelity, and rhythmic stability. The resulting system enables highly coherent and seamless human–AI musical co-performance.

Abstract

Interactive music and live performance relies on real-time human expression, but modern generative music AI remains largely absent from this domain due to its prohibitive inference latency and offline rendering paradigm. To provide pioneer musicians with a novel medium for interactive composition, we should fundamentally change these static models into dynamic, playable instruments. In this paper, we propose a framework that bridges this gap. To achieve the low latency required for live interaction without sacrificing structural coherence, we formulate distillation within a streaming autoregressive latent space. Our approach gets rid of the need for expensive paired audio-latent datasets by utilizing prompt-only inputs to synthesize teacher-guided, chunk-wise trajectories on the fly. Because live instruments require high acoustic fidelity, we introduce music-aware consistency objectives, which combine latent, spectral, and temporal-difference losses, to preserve crucial qualities like timbre, transients, and rhythmic stability during accelerated single-step streaming generation. Implemented via parameter-efficient adaptation, our distillation reduces generation steps to achieve a low real-time factor. Crucially, by operating as a continuous autoregressive stream, the system can seamlessly assimilate dynamic human inputs on the fly, allowing users to instantly steer the musical trajectory without interrupting the audio flow. Ultimately, this work recontextualizes generative text-to-music models not as passive prompt-and-wait systems, but as responsive instruments, opening new frontiers for live human-AI musical co-creation.

Problem

• real-time
• interactive music generation
• inference latency
• live performance
• generative AI

Innovation

• streaming consistency distillation
• real-time music generation
• data-free distillation
• music-aware consistency
• interactive AI instruments