CAARMA: Class Augmentation with Adversarial Mixup Regularization

2025-03-20

AI Summary

To address the limited class diversity in real speech data for speaker verification—which weakens discriminability and generalization capability of embedding spaces—this paper proposes a class-level embedding-space mixing augmentation method tailored for zero-shot learning. Our approach features three key contributions: (1) the first application of class-level adversarial Mixup directly in the embedding space to generate semantically coherent synthetic classes; (2) an adversarial refinement mechanism that jointly optimizes generator and discriminator networks to ensure synthetic classes are distributionally indistinguishable from real ones; and (3) end-to-end training via coupling with contrastive learning loss. Evaluated on multiple speaker verification and zero-shot speech analysis benchmarks, our method achieves an average 8% performance gain over all baseline models, significantly enhancing intra-class compactness and inter-class separability of learned embeddings.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Speech
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Speaker verification is a typical zero-shot learning task, where inference of unseen classes is performed by comparing embeddings of test instances to known examples. The models performing inference must hence naturally generate embeddings that cluster same-class instances compactly, while maintaining separation across classes. In order to learn to do so, they are typically trained on a large number of classes (speakers), often using specialized losses. However real-world speaker datasets often lack the class diversity needed to effectively learn this in a generalizable manner. We introduce CAARMA, a class augmentation framework that addresses this problem by generating synthetic classes through data mixing in the embedding space, expanding the number of training classes. To ensure the authenticity of the synthetic classes we adopt a novel adversarial refinement mechanism that minimizes categorical distinctions between synthetic and real classes. We evaluate CAARMA on multiple speaker verification tasks, as well as other representative zero-shot comparison-based speech analysis tasks and obtain consistent improvements: our framework demonstrates a significant improvement of 8% over all baseline models. Code for CAARMA will be released.

Problem

• Enhancing speaker verification with synthetic class augmentation
• Improving generalization via adversarial mixup regularization
• Addressing limited class diversity in zero-shot learning tasks

Innovation

• Generates synthetic classes via embedding space mixing
• Uses adversarial refinement for class authenticity
• Improves speaker verification performance by 8%