Where Do Self-Supervised Speech Models Become Unfair?

2026-04-20

AI Summary

This study addresses fairness disparities in self-supervised speech models across speaker subgroups, whose underlying causes remain poorly understood. By probing layer-wise embeddings on speaker identification (SID) and automatic speech recognition (ASR) tasks, the work reveals, for the first time, opposing patterns of bias propagation: SID bias intensifies with network depth, whereas ASR bias emerges prominently in shallow layers and exhibits a negative correlation with overall error rates. The findings demonstrate that subgroup biases become entrenched during pretraining and are largely resistant to mitigation through fine-tuning, suggesting that effective fairness interventions must be implemented at the pretraining stage rather than as downstream adjustments.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Speech encoder models are known to model members of some speaker groups (SGs) better than others. However, there has been little work in establishing why this occurs on a technological level. To our knowledge, we present the first layerwise fairness analysis of pretrained self-supervised speech encoder models (S3Ms), probing each embedding layer for speaker identification (SID) automatic speech recognition (ASR). We find S3Ms produce embeddings biased against certain SGs for both tasks, starting at the very first latent layers. Furthermore, we find opposite patterns of layerwise bias for SID vs ASR for all models in our study: SID bias is minimized in layers that minimize overall SID error; on the other hand, ASR bias is maximized in layers that minimize overall ASR error. The inverse bias/error relationship for ASR is unaffected when probing S3Ms that are finetuned for ASR, suggesting SG-level bias is established during pretraining and is difficult to remove.

Problem

• self-supervised speech models
• fairness
• speaker groups
• bias
• layerwise analysis

Innovation

• self-supervised speech models
• layerwise fairness analysis
• speaker group bias
• automatic speech recognition
• speaker identification