MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

2026-08-12

AI Summary

Existing text-to-audio generation approaches struggle to coherently integrate speech, music, and sound effects due to their decoupled encoder–decoder architectures, often resulting in limited speech intelligibility. This work proposes MiDashengLM-Gen, an end-to-end framework that, for the first time, combines a pretrained large language model with token-wise conditional flow matching to enable autoregressive, variable-length generation of mixed audio scenes. Through end-to-end joint training, the method substantially enhances cross-modal consistency and speech clarity, reducing the word error rate (WER) on the Seed-TTS benchmark from 12.15% to 2.79%—approaching the performance of specialized TTS systems (1.24%)—while maintaining strong multimodal audio generation quality on the MECAT benchmark.

Abstract

Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at https://github.com/xiaomi-research/midashenglm-gen and https://huggingface.co/mispeech/midashenglm-gen, and the demo page is available at https://xingws.github.io/midashenglm-gen-demo/.

Problem

• audio scene generation
• speech intelligibility
• text-to-audio
• multimodal generation
• coherent audio

Innovation

• LLM-driven audio generation
• autoregressive flow matching
• end-to-end text-to-audio
• unified audio scene synthesis
• per-token conditional modeling