Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance

2026-07-13

AI Summary

This work addresses the challenge of zero-shot synthesis of realistic timbres for arbitrary target MIDI sequences using only a single reference audio clip of an unseen instrument and its corresponding MIDI. To this end, the authors propose Anysynth, a neural synthesizer that operates without timbre embeddings and leverages a Diffusion Transformer architecture conditioned directly on raw reference audio and target MIDI through an in-context flow-matching mechanism. Key innovations include eliminating fixed-length embeddings to enable scalable timbre cloning from reference audios of arbitrary duration, and introducing an asymmetric hierarchical classifier-free guidance (CFG) scheme that effectively disentangles MIDI content from timbre control. Experiments demonstrate that Anysynth outperforms existing embedding-based and autoregressive baselines in audio quality, timbre similarity, and melodic consistency, while uniquely exhibiting higher timbre fidelity with longer reference clips.

Abstract

Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/

Problem

• zero-shot instrument cloning
• timbre reconstruction
• reference audio conditioning
• MIDI-to-audio synthesis
• in-context learning

Innovation

• zero-shot instrument cloning
• in-context learning
• embedding-free synthesis
• asymmetric hierarchical guidance
• diffusion transformer