DDPO-VC: Speaker De-Identification via Diffusion Denoising Policy Optimization

2026-06-13

AI Summary

This work addresses the challenge in speaker de-identification of simultaneously preserving privacy and maintaining speech utility—such as cues indicative of cognitive health—where their inherent correlation undermines conventional disentanglement approaches. To overcome this limitation, the authors propose DDPO-VC, a novel framework that introduces diffusion models to speaker de-identification and incorporates reinforcement learning-based post-training. The generation process is optimized using multi-teacher reward signals guided by both privacy and utility objectives, thereby circumventing the restrictive assumption of independence between these factors. Experimental results on two dementia-related speech benchmarks demonstrate that DDPO-VC not only effectively safeguards speaker identity but also significantly outperforms strong existing baselines in retaining the speech characteristics essential for downstream cognitive assessment.

Abstract

A key challenge of speaker de-identification is the balance between privacy and utility. Many utility variables, such as the cognitive health status of the speaker, are correlated with the privacy variable, such as the speaker identity, violating the independence assumption held by the disentanglement-based approaches, causing leakage of private information and the loss of useful information for downstream tasks. To tackle this challenge, we propose a general framework, DDPO-VC, for speaker de-identification through reinforcement learning-based post-training with diffusion models. Learning from reward signals combining knowledge from privacy-focused and utility-focused teachers, our method outperforms various strong \deid/ methods in both privacy preservation and cognitive utility on two commonly used dementia speech benchmarks. Please check out our code\footnote{\href{https://github.com/cactuswiththoughts/DDPO-VC}{https://github.com/cactuswiththoughts/DDPO-VC}} and demo\footnote{\href{https://cactuswiththoughts.github.io/SpeakerDeID-Demo/}{https://cactuswiththoughts.github.io/SpeakerDeID-Demo/}}.

Problem

• speaker de-identification
• privacy-utility trade-off
• diffusion models
• cognitive health
• voice conversion

Innovation

• Diffusion Models
• Reinforcement Learning
• Speaker De-identification
• Privacy-Utility Trade-off
• Post-training Optimization