YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidance

2025-12-04

AI Summary

Singing Voice Synthesis (SVS) has long suffered from heavy reliance on phoneme-level alignments and manual melody annotations, resulting in high annotation costs and poor generalization. This paper proposes a zero-shot SVS framework that requires neither phoneme nor melody annotations—only a reference audio clip is needed to extract melody and synthesize high-fidelity singing for arbitrary lyrics. Our approach introduces three key innovations: (1) an unsupervised melody encoder guided by a teacher model; (2) a Diffusion Transformer architecture integrating implicit alignment with weakly supervised duration modeling; and (3) Flow-GRPO, a multi-objective reinforcement learning algorithm optimizing both audio quality and melody fidelity. Experiments demonstrate substantial improvements over prior work in zero-shot generation and lyric editing tasks, achieving state-of-the-art performance in both objective metrics and subjective MOS scores. Moreover, the framework supports cross-style transfer and high-fidelity melody reproduction.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Singing Voice Synthesis (SVS) remains constrained in practical deployment due to its strong dependence on accurate phoneme-level alignment and manually annotated melody contours, requirements that are resource-intensive and hinder scalability. To overcome these limitations, we propose a melody-driven SVS framework capable of synthesizing arbitrary lyrics following any reference melody, without relying on phoneme-level alignment. Our method builds on a Diffusion Transformer (DiT) architecture, enhanced with a dedicated melody extraction module that derives melody representations directly from reference audio. To ensure robust melody encoding, we employ a teacher model to guide the optimization of the melody extractor, alongside an implicit alignment mechanism that enforces similarity distribution constraints for improved melodic stability and coherence. Additionally, we refine duration modeling using weakly annotated song data and introduce a Flow-GRPO reinforcement learning strategy with a multi-objective reward function to jointly enhance pronunciation clarity and melodic fidelity. Experiments show that our model achieves superior performance over existing approaches in both objective measures and subjective listening tests, especially in zero-shot and lyric adaptation settings, while maintaining high audio quality without manual annotation. This work offers a practical and scalable solution for advancing data-efficient singing voice synthesis. To support reproducibility, we release our inference code and model checkpoints.

Problem

• Eliminates need for phoneme-level alignment in singing synthesis
• Removes dependency on manually annotated melody contours
• Enables zero-shot synthesis with arbitrary lyrics and melodies

Innovation

• Melody extraction from audio without phoneme alignment
• Diffusion Transformer with teacher-guided melody encoding
• Flow-GRPO reinforcement learning for pronunciation and melody