The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

2026-03-15

AI Summary

This study investigates implicit biases in SpeechLLMs when processing spoken inputs varying in accent and gender presentation. Using voice cloning to control linguistic content, the authors systematically manipulate speaker accent (e.g., Eastern European) and gender presentation across 2,880 interactions with three prominent SpeechLLMs. Adopting an intersectional lens, they introduce a multimodal evaluation framework combining pairwise LLM-based scoring, comparative ranking, best–worst scaling, and multidimensional human assessment. Their analysis reveals, for the first time, distinct bias patterns emerging at the intersection of accent and gender: voices with Eastern European accents—particularly those perceived as female—receive significantly lower helpfulness ratings. Notably, human evaluators demonstrate greater sensitivity than automated metrics, capturing nuanced intersectional disparities that algorithmic measures overlook.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Speech Large Language Models (SpeechLLMs) process spoken input directly, retaining cues such as accent and perceived gender that were previously removed in cascaded pipelines. This introduces speaker identity dependent variation in responses. We present a large-scale intersectional evaluation of accent and gender bias in three SpeechLLMs using 2,880 controlled interactions across six English accents and two gender presentations, keeping linguistic content constant through voice cloning. Using pointwise LLM-judge ratings, pairwise comparisons, and Best-Worst Scaling with human validation, we detect consistent disparities. Eastern European-accented speech receives lower helpfulness scores, particularly for female-presenting voices. The bias is implicit: responses remain polite but differ in helpfulness. While LLM judges capture the directional trend of these biases, human evaluators exhibit significantly higher sensitivity, uncovering sharper intersectional disparities.

Problem

• SpeechLLMs
• intersectional bias
• accent bias
• gender bias
• speaker identity

Innovation

• SpeechLLMs
• intersectional bias
• voice cloning
• accent bias
• gender presentation