Repurposing a Speech Classifier for Guided Diffusion-Based Speech Generation

2026-06-18

AI Summary

This work addresses the inefficiency of traditional classifier-guided diffusion models, which require separate training of a classifier and a generative model, leading to redundancy and high computational costs. To overcome this, the authors propose an efficient approach for conditional speech generation that repurposes a pretrained speech classifier as a shared backbone network. By freezing the backbone’s parameters and training only lightweight auxiliary subnetworks, the method enables conditional generation while unifying discriminative modeling and speech synthesis within a single architecture for the first time. The framework integrates noise-conditional classification, log-Mel spectrogram space modeling, and denoising score matching, achieving high-quality speech synthesis with substantially reduced memory and computational overhead.

Abstract

Classifier guidance is a way to control diffusion generation by using a noise-conditioned classifier to steer the sampling process toward a target class. One drawback of classifier guidance is that it requires two separately trained models: a classifier and a diffusion model. We therefore study a more compact alternative in which a conventionally trained speech classifier is repurposed as the backbone for diffusion generation. Starting from a frozen noise-conditioned classifier in log-Mel space, we attach a lightweight subnetwork that reuses intermediate classifier representations and train only this subnetwork under a Denoising Score Matching objective. Our work shows that a pretrained classifier can be repurposed for conditional generation, providing an appealing bridge between discriminative modeling and conditional speech synthesis resulting in high speech quality within a single-backbone model, with reduced memory footprint and computational cost.

Problem

• classifier guidance
• diffusion-based speech generation
• speech classifier repurposing
• conditional speech synthesis
• single-backbone model

Innovation

• classifier guidance
• diffusion-based speech generation
• repurposed classifier
• denoising score matching
• single-backbone model