HARP 2.0: Expanding Hosted, Asynchronous, Remote Processing for Deep Learning in the DAW

2025-03-04

AI Summary

Music creators face significant challenges in seamlessly integrating deep learning models into digital audio workstations (DAWs). Method: We propose a hosted, asynchronous remote processing framework enabling transparent integration. It comprises a VST/AU-compliant DAW plugin that communicates with Gradio endpoints via a lightweight pyharp Python API, supporting real-time audio/MIDI streaming, MIDI event serialization, and inference of joint audio-MIDI annotation models—all with native in-plugin rendering of interactive UIs and results. Contribution/Results: This work introduces the first DAW-native support for general-purpose MIDI generation and multimodal joint audio-MIDI annotation models. It enables zero-context-switch creative AI workflows and achieves bounded end-to-end latency (<200 ms under typical conditions), while significantly improving plugin stability and cross-platform compatibility (macOS, Windows, Linux). The framework lowers the barrier to adopting AI tools in music production and fosters efficient collaboration between music technology developers and creators.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

HARP 2.0 brings deep learning models to digital audio workstation (DAW) software through hosted, asynchronous, remote processing, allowing users to route audio from a plug-in interface through any compatible Gradio endpoint to perform arbitrary transformations. HARP renders endpoint-defined controls and processed audio in-plugin, meaning users can explore a variety of cutting-edge deep learning models without ever leaving the DAW. In the 2.0 release we introduce support for MIDI-based models and audio/MIDI labeling models, provide a streamlined pyharp Python API for model developers, and implement numerous interface and stability improvements. Through this work, we hope to bridge the gap between model developers and creatives, improving access to deep learning models by seamlessly integrating them into DAW workflows.

Problem

• Integrates deep learning models into DAW software.
• Enables audio and MIDI transformations via remote processing.
• Bridges gap between model developers and creative professionals.

Innovation

• Hosted asynchronous remote processing for DAWs
• Integration of deep learning models in DAWs
• Support for MIDI and audio/MIDI labeling models