Versatile Framework for Song Generation with Prompt-based Control

2025-04-27

AI Summary

Existing song generation methods struggle to simultaneously produce high-fidelity, temporally aligned vocals and accompaniment conditioned on text prompts, and lack support for multi-task music generation. This paper introduces VersBand—the first multi-task controllable song synthesis framework capable of jointly generating lyrics, melody, vocals, and accompaniment. Its core innovations include: (1) a decoupled VocalBand architecture for prompt-driven vocal synthesis and a Band-MoE–enhanced AccompBand for controllable accompaniment generation, enabling end-to-end prompt control and cross-modal alignment; and (2) integration of flow matching, streaming Transformers, and a multi-stage generation paradigm. VersBand achieves significant improvements over state-of-the-art methods across objective metrics—including MCD, F0 RMSE, and alignment score—as well as subjective MOS evaluations. The model, training code, and audio samples are publicly released.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Song generation focuses on producing controllable high-quality songs based on various prompts. However, existing methods struggle to generate vocals and accompaniments with prompt-based control and proper alignment. Additionally, they fall short in supporting various tasks. To address these challenges, we introduce VersBand, a multi-task song generation framework for synthesizing high-quality, aligned songs with prompt-based control. VersBand comprises these primary models: 1) VocalBand, a decoupled model, leverages the flow-matching method for generating singing styles, pitches, and mel-spectrograms, allowing fast, high-quality vocal generation with style control. 2) AccompBand, a flow-based transformer model, incorporates the Band-MOE, selecting suitable experts for enhanced quality, alignment, and control. This model allows for generating controllable, high-quality accompaniments aligned with vocals. 3) Two generation models, LyricBand for lyrics and MelodyBand for melodies, contribute to the comprehensive multi-task song generation system, allowing for extensive control based on multiple prompts. Experimental results demonstrate that VersBand performs better over baseline models across multiple song generation tasks using objective and subjective metrics. Audio samples are available at https://VersBand.github.io.

Problem

• Generate controllable high-quality songs with prompt-based control
• Align vocals and accompaniments properly in song generation
• Support various song generation tasks with a unified framework

Innovation

• Flow-matching method for vocal style and pitch control
• Flow-based transformer with Band-MOE for accompaniment alignment
• Multi-task system integrating lyrics and melody generation