AdaTT: Text-Guided Instrument Timbre Transfer with Target-Adaptive Structural Control

2026-06-14

AI Summary

This work addresses the issue of timbral ambiguity in fine-grained musical instrument timbre transfer, which arises from conflicts between expressive details in the source performance and the intrinsic characteristics of the target timbre. To resolve this, the authors propose AdaTT, a system built upon the ControlNet architecture that incorporates a target-adaptive mechanism. This mechanism dynamically modulates the strength of pitch and loudness controls on a per-frame basis via textual prompts to align with the target instrument’s timbral identity. Additionally, a semi-automated data construction pipeline is introduced to guide the model in discerning which expressive nuances should be preserved or transformed. Experimental results demonstrate that, while preserving the original musical score content, AdaTT significantly enhances both timbral fidelity and perceptual naturalness, outperforming existing approaches.

Abstract

This paper addresses timbral ambiguity in instrument timbre transfer under fine-grained structural conditions. We argue this issue stems from instrument-specific expressive details in these conditions, which conflict with the target timbral properties. For example, imposing a violin's pitch-dominant vibrato contours onto a flute, which naturally exhibits loudness-dominant vibrato, impairs timbral fidelity. We propose AdaTT, a target-adaptive system that ensures high timbral fidelity across diverse timbre transfer scenarios within the ControlNet scheme. It selectively scales the frame-wise influence of pitch and loudness controls via text prompts to match the target instrument's identity. We also present a semi-automatic data construction pipeline to teach the model which expressive details to transform or preserve. Results show AdaTT achieves superior timbral fidelity and naturalness while retaining score-level content. Audio samples are available at https://dabinkim0.github.io/adatt/.

Problem

• timbre transfer
• timbral ambiguity
• expressive details
• structural control
• instrument timbre

Innovation

• timbre transfer
• target-adaptive control
• text-guided synthesis
• expressive detail preservation
• ControlNet