Clustering-based hard negative sampling for supervised contrastive speaker verification

2025-07-23

AI Summary

In speaker verification, supervised contrastive learning suffers from limited discriminative power due to insufficient hard negative mining. To address this, we propose a dynamic hard negative sampling method based on embedding clustering: first, intra-speaker embeddings are clustered to refine intra-class structure; then, highly challenging negative pairs are dynamically constructed according to inter-cluster distances, optimizing the hardness distribution of negative samples within each batch. This is the first work to incorporate clustering into supervised contrastive learning for negative sample selection—requiring no additional annotations or complex modules and enabling end-to-end integration with lightweight backbone networks. On VoxCeleb1, our approach achieves up to 18% relative reduction in both EER and minDCF, significantly outperforming state-of-the-art classification- and contrastive-learning-based methods, particularly in distinguishing acoustically similar speakers.

Technology Category

• Machine Learning: Clustering
• Natural Language Processing: Speech
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

In speaker verification, contrastive learning is gaining popularity as an alternative to the traditionally used classification-based approaches. Contrastive methods can benefit from an effective use of hard negative pairs, which are different-class samples particularly challenging for a verification model due to their similarity. In this paper, we propose CHNS - a clustering-based hard negative sampling method, dedicated for supervised contrastive speaker representation learning. Our approach clusters embeddings of similar speakers, and adjusts batch composition to obtain an optimal ratio of hard and easy negatives during contrastive loss calculation. Experimental evaluation shows that CHNS outperforms a baseline supervised contrastive approach with and without loss-based hard negative sampling, as well as a state-of-the-art classification-based approach to speaker verification by as much as 18 % relative EER and minDCF on the VoxCeleb dataset using two lightweight model architectures.

Problem

• Improving speaker verification via hard negative sampling
• Optimizing batch composition for contrastive learning
• Enhancing supervised speaker representation learning accuracy

Innovation

• Clustering-based hard negative sampling method
• Adjusts batch composition for optimal negatives
• Outperforms baseline and classification-based approaches