FineCombo-TTS: Collaborative and Precise Controllable Speech Synthesis Using Text Descriptions and Reference Speech

2026-06-17

AI Summary

This work addresses the limitations of existing controllable text-to-speech (TTS) approaches, which struggle to achieve fine-grained control when relying solely on reference audio or textual descriptions, and often suffer from suboptimal performance due to loose coupling in joint methods. To overcome these challenges, we propose FineCombo-TTS, a unified framework that seamlessly integrates reference audio and textual guidance without explicit attribute disentanglement. Our approach leverages a shared acoustic representation and a conditional flow matching (CFM)-based variance predictor to enable flexible and precise manipulation of acoustic attributes. We further introduce FineEdit, a structured paired dataset designed for relative attribute control. Experimental results demonstrate that FineCombo-TTS significantly enhances the flexibility, accuracy, and expressiveness of synthesized speech in controllable TTS tasks.

Abstract

Controllable text-to-speech (TTS) has become a key research focus. However, methods based on either reference speech or text descriptions lack flexibility and precise control, and recent joint approaches remain loosely coupled, with speech modeling timbre and text controlling global style. We propose FineCombo-TTS, a unified framework for speech synthesis grounded in reference speech and guided by text descriptions, enabling flexible and precise control over acoustic attributes. Instead of explicit attribute disentanglement, we learn a unified acoustic representation and introduce a Conditional Flow Matching (CFM)-based Speech Variance Predictor to model fine-grained reference-to-target transformations guided by text descriptions. To support relative attribute control, we construct FineEdit, a structured paired dataset that explicitly encodes source-to-target attribute variations. Experiments demonstrate that our approach achieves flexible, precise, and expressive controllable TTS.

Problem

• controllable TTS
• speech synthesis
• fine-grained control
• acoustic attributes
• reference speech

Innovation

• Controllable TTS
• Conditional Flow Matching
• Reference Speech
• Text-to-Speech Synthesis
• Acoustic Representation