RIVET: Robust Idempotent Voice Attribute Editing

2026-06-17

AI Summary

This work addresses the instability of speech attribute editing models caused by label noise or inconsistency in large-scale datasets. To mitigate this issue, the study introduces idempotency constraints—formally defined as \( f(f(x)) = f(x) \)—into the training of end-to-end conditional generative models, serving as an implicit regularization mechanism. By enforcing that repeated application of the editing function yields the same result as a single application, the proposed approach enhances robustness to label noise without requiring explicit noise modeling. Experiments demonstrate significant improvements in editing success rates on both synthetically controlled noisy data and the real-world GLOBE dataset, while simultaneously better preserving speaker identity characteristics compared to existing methods.

Abstract

Voice attribute editing models modify characteristics such as age and gender while preserving speaker identity. In large-scale speech datasets, however, attribute annotations are often noisy or inconsistent, which can cause conditional generative models to produce unstable edits. In this work, we show that idempotency provides an effective mechanism for improving robustness to noisy labels. An idempotent operator is one for which repeated application does not change the result, i.e., f(f(x)) = f(x). Enforcing this property acts as an implicit regularizer that reduces sensitivity to mislabeled examples. We introduce RIVET, a training framework that incorporates an idempotency objective to improve robustness to label noise. We evaluate RIVET under controlled label noise and on the GLOBE dataset with naturally noisy annotations. RIVET improves editing success and better preserves speaker identity than standard training, showing that idempotency improves robustness in voice editing models.

Problem

• voice attribute editing
• label noise
• speaker identity preservation
• noisy annotations
• robustness

Innovation

• idempotency
• voice attribute editing
• label noise robustness
• speaker identity preservation
• RIVET