LAPS-Diff: A Diffusion-Based Framework for Singing Voice Synthesis With Language Aware Prosody-Style Guided Learning

2025-07-07

AI Summary

In low-resource Bollywood Hindi singing voice synthesis (SVS), modeling vocal style and capturing language-dependent pitch variations remain challenging. Method: We propose a language-aware and singing-style joint modeling framework based on diffusion models, integrating pretrained language models (word- and phoneme-level embeddings), a style encoder, a pitch extraction module, and multimodal speech representation models (MERT and IndicWav2Vec) to construct hierarchical text- and music-contextualized priors. Contribution/Results: To our knowledge, this is the first work to jointly model linguistic structure and vocal expressivity in low-resource Hindi SVS. Extensive objective and subjective evaluations on a newly curated Hindi singing dataset demonstrate that our method significantly outperforms state-of-the-art baselines in naturalness, style fidelity, and pitch accuracy.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The field of Singing Voice Synthesis (SVS) has seen significant advancements in recent years due to the rapid progress of diffusion-based approaches. However, capturing vocal style, genre-specific pitch inflections, and language-dependent characteristics remains challenging, particularly in low-resource scenarios. To address this, we propose LAPS-Diff, a diffusion model integrated with language-aware embeddings and a vocal-style guided learning mechanism, specifically designed for Bollywood Hindi singing style. We curate a Hindi SVS dataset and leverage pre-trained language models to extract word and phone-level embeddings for an enriched lyrics representation. Additionally, we incorporated a style encoder and a pitch extraction model to compute style and pitch losses, capturing features essential to the naturalness and expressiveness of the synthesized singing, particularly in terms of vocal style and pitch variations. Furthermore, we utilize MERT and IndicWav2Vec models to extract musical and contextual embeddings, serving as conditional priors to refine the acoustic feature generation process further. Based on objective and subjective evaluations, we demonstrate that LAPS-Diff significantly improves the quality of the generated samples compared to the considered state-of-the-art (SOTA) model for our constrained dataset that is typical of the low resource scenario.

Problem

• Challenges in capturing vocal style and pitch inflections in SVS
• Difficulty in handling language-dependent characteristics in low-resource scenarios
• Need for improved naturalness and expressiveness in synthesized singing

Innovation

• Diffusion model with language-aware embeddings
• Vocal-style guided learning mechanism
• Pre-trained models for enriched lyrics representation