ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models

2026-07-10

AI Summary

Existing diffusion models struggle to generate high-quality audio efficiently under low-frequency, highly compressed conditions due to implicit coupling in intermediate representations. This work proposes ReGen, a novel framework that jointly models data and representations through multi-vector fields and enhances the generalization of conditional flow matching via Generalized Flow Matching (GFM). ReGen introduces a hierarchical multi-prompt representation mechanism, enabling high-fidelity speech and audio synthesis at extremely low sampling rates (6.25–12.5 Hz). Built upon a diffusion Transformer, a neural audio codec, and a latent diffusion architecture, ReGenVoice requires only four GPUs for one day of training and achieves a real-time factor (RTF) of 0.08 during inference, significantly improving word error rate (WER) for intelligibility and speaker similarity (SIM).

Abstract

Representation alignment (REPA) has been investigated to accelerate diffusion training, but we observe that regularizing intermediate representations in diffusion Transformers (DiT) may implicitly entangle latents and limit generative capacity. To address this issue, we propose ReGen, a hierarchical multi-prompt representation generation framework that jointly estimates multiple vector fields for both representations and data within a single diffusion model. We further introduce generalized flow matching (GFM) to improve the generalization of conditional flow matching (CFM). We validate ReGen on single-stage waveform diffusion models including neural audio codec and Wave-VAE. ReGen significantly improves waveform generation quality from highly compressed latent representations at 12.5 Hz. We also present ReGenVoice, a latent diffusion model (LDM)-based text-to-speech model that achieves strong speech intelligibility (WER) and speaker similarity (SIM) with a small dataset. Moreover, operating the LDM at 6.25 Hz with rich semantic and acoustic latent representation enables efficient training and sampling, requiring only 1 day of training on 4 GPUs and fast inference with an RTF of 0.08. Audio samples are available at https://regenvoice.github.io/demo/.

Problem

• representation alignment
• latent entanglement
• waveform diffusion
• low-rate latent representation
• generative capacity

Innovation

• ReGen
• hierarchical multi-prompt representation
• generalized flow matching
• waveform diffusion
• latent diffusion model