Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models

2026-04-24

AI Summary

This work addresses the performance disparities of automatic speech recognition systems across speaker groups by investigating the fine-grained sources of unfairness in phoneme embeddings. We propose a novel framework that attributes group-level unfairness to two distinct types of errors in phoneme modeling: stochastic error (high variance) and systematic bias. These components are disentangled using group-specific phoneme classification probes. By integrating variance and bias metrics with domain augmentation and adversarial training, we analyze the embedding properties of self-supervised speech models. Our experiments reveal that stochastic error exerts a substantially greater impact on group fairness than systematic bias, and that existing fairness-aware fine-tuning strategies struggle to effectively mitigate this issue or alter the benefits derived from probe training.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Modern automatic speech recognition (ASR) systems have been observed to function better for certain speaker groups (SGs) than others, despite recent gains in overall performance. One potential impediment to progress towards fairer ASR is a more nuanced understanding of the types of modeling errors that speech encoder models make, and in particular the difference between the structure of embeddings for high-performance and low-performance SGs. This paper proposes a framework typifying two types of error that can occur in modeling phonemes in ASR systems: random error/high variance in phoneme embedding, vs systematic error/embedding bias. We find that training phoneme classification probes only on a single, typically disadvantaged SG, sometimes improves performance for that SG, which is evidence for the existence of SG-level bias in phoneme embeddings. On the other hand, we find that speakers and SGs with higher levels of phoneme variance are the same as those with worse phoneme prediction accuracy. We conclude that both types of error are present in phoneme embeddings and both are candidate causes for SG-level unfairness in ASR, though random error is likely a greater hindrance to fairness than systematic error. Furthermore, we find that finetuning encoder models using a fairness-enhancing algorithm (domain enhancing and adversarial training) changes neither the benefits of in-domain phoneme classification probe training, nor measured levels of random embedding error.

Problem

• demographic unfairness
• phoneme-level embeddings
• self-supervised speech recognition
• speaker group bias
• embedding variance

Innovation

• phoneme-level embeddings
• demographic unfairness
• systematic bias
• random error
• self-supervised speech recognition