Do Bias Benchmarks Generalise? Evidence from Voice-based Evaluation of Gender Bias in SpeechLLMs

2025-09-24

AI Summary

This study investigates whether gender bias evaluation benchmarks designed for multiple-choice question answering (MCQA) generalize to other tasks—such as long-text generation—and across diverse speech inputs. We fine-tune three SpeechLLMs using LoRA to construct biased models exhibiting stereotypical, counter-stereotypical, or neutral gender tendencies. These models are systematically evaluated for behavioral consistency across multi-format tasks: MCQA, open-ended text generation, and cross-voice-style generation. Results reveal a significant misalignment between bias manifestations in MCQA benchmarks and those in long-text generation, with poor cross-task predictive validity; thus, single-format evaluation fails to reflect real-world fairness. To address this, we propose a “cross-task behavioral transferability” evaluation paradigm, advocating for multi-format, task-coordinated bias assessment frameworks. This work establishes a novel standard and practical methodology for fairness evaluation of SpeechLLMs.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Recent work in benchmarking bias and fairness in speech large language models (SpeechLLMs) has relied heavily on multiple-choice question answering (MCQA) formats. The model is tasked to choose between stereotypical, anti-stereotypical, or neutral/irrelevant answers given an input speech prompt and an optional text prompt. Such MCQA benchmarks implicitly assume that model performance is consistent across other MCQA tasks, voices, and other task formats such as more realistic, long-form evaluations. In this paper, we probe that assumption. We fine-tune three SpeechLLMs using LoRA adapters to induce specific MCQA behaviours: preference for stereotypical, anti-stereotypical, or neutral/uncertain answers. We then evaluate whether these behaviours generalise to another, distinct MCQA benchmark, and more critically to long-form, creative generation tasks. Our results show that performance on MCQA bias benchmarks fails to reliably predict performances across other MCQA benchmarks, and more importantly across long-form tasks. We conclude that current MCQA bias benchmarks show limited evidence of cross-task generalisation in the speech domain, and also propose an evaluation suite for measuring behaviour transferability in future models and benchmarks.

Problem

• Evaluating cross-task generalization of bias benchmarks in SpeechLLMs
• Testing if MCQA bias assessments predict long-form task performance
• Assessing behavior transferability between different bias evaluation formats

Innovation

• Fine-tuned SpeechLLMs using LoRA adapters
• Evaluated bias generalization across MCQA benchmarks
• Assessed behavior transfer to long-form generation tasks