RedVox: Safety and Fairness Gaps in Speech Models Across Languages

2026-06-25

AI Summary

Current speech models suffer from a critical lack of evaluation regarding safety and fairness in non-English contexts and under natural speech conditions. This work proposes RedVox—the first multilingual benchmark for safety and fairness assessment that spans five languages and is grounded in real human voices. Integrating natural speech collection, safety testing, user studies, and model evaluation, the study systematically investigates how spoken inputs influence model behavior. Findings reveal that mainstream speech models are significantly more prone to generating unsafe or stereotypical outputs in non-English languages, and that speech inputs exacerbate these risks compared to text inputs, thereby exposing modality-specific vulnerabilities inherent to spoken language processing.

Abstract

Speech-capable models are increasingly deployed in real-world applications across languages. Yet their safety and fairness beyond English settings and under naturalistic conditions remain understudied. We survey safety reporting practices across state-of-the-art speech model releases, finding that only 8% document any multilingual analysis. To address this gap, we introduce RedVox, a multilingual safety and fairness benchmark for audio and speech built on real voices, covering unsafe and unfair stereotypical requests across five languages (English, French, Italian, Spanish, and German). Evaluating eight state-of-the-art models, we find that vulnerabilities persist even under non-adversarial conditions, worsen in non-English languages, and are amplified when the request comes from a spoken input. Finally, by surveying the participants who contributed to RedVox, we document the unique personal and privacy challenges of collecting speech data with human participants, pointing to broader sociotechnical challenges in naturalistic speech safety research.

Problem

• speech safety
• fairness
• multilingual
• voice input
• bias

Innovation

• multilingual speech safety
• fairness benchmark
• real-voice dataset
• non-adversarial evaluation
• sociotechnical challenges