Tutti: Expressive Multi-Singer Synthesis via Structure-Level Timbre Control and Vocal Texture Modeling

2026-02-09

AI Summary

Existing singing synthesis systems are constrained by global timbre control, making it difficult to model dynamic multi-singer arrangements and rich acoustic textures within a single piece. This work proposes Tutti, a unified framework that enables flexible singer scheduling aligned with musical structure through a structure-aware singer prompting mechanism. Furthermore, it introduces a condition-guided variational autoencoder (VAE) to learn complementary acoustic textures by jointly leveraging explicit and implicit acoustic features. By moving beyond the limitations of conventional global timbre settings, the proposed method significantly outperforms existing approaches in both multi-singer scheduling accuracy and the acoustic realism of choral synthesis, establishing a new paradigm for complex polyphonic singing synthesis.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

While existing Singing Voice Synthesis systems achieve high-fidelity solo performances, they are constrained by global timbre control, failing to address dynamic multi-singer arrangement and vocal texture within a single song. To address this, we propose Tutti, a unified framework designed for structured multi-singer generation. Specifically, we introduce a Structure-Aware Singer Prompt to enable flexible singer scheduling evolving with musical structure, and propose Complementary Texture Learning via Condition-Guided VAE to capture implicit acoustic textures (e.g., spatial reverberation and spectral fusion) that are complementary to explicit controls. Experiments demonstrate that Tutti excels in precise multi-singer scheduling and significantly enhances the acoustic realism of choral generation, offering a novel paradigm for complex multi-singer arrangement. Audio samples are available at https://annoauth123-ctrl.github.io/Tutii_Demo/.

Problem

• multi-singer synthesis
• timbre control
• vocal texture
• singing voice synthesis
• choral generation

Innovation

• multi-singer synthesis
• structure-aware singer prompt
• complementary texture learning
• condition-guided VAE
• vocal texture modeling