SamaVaani: Auditing and Debiasing Multilingual Clinical ASR for Indian Languages

2026-06-25

AI Summary

This study addresses the performance disparities and fairness risks of mainstream automatic speech recognition (ASR) systems in multilingual clinical settings across India, where models exhibit suboptimal performance—particularly in psychiatric interviews conducted in Kannada, Hindi, and Indian English—and demonstrate systematic biases across language, gender, and speaker role. The authors present the first clinical audit of eight leading ASR models, including IndicWhisper, WhisperLargeV3, and Sarvam, and introduce SamaVaani, a unified debiasing framework that applies fairness-aware fine-tuning to top-performing open-source models such as Gemma3n and OmniLingual. Experimental results demonstrate that SamaVaani not only significantly improves overall ASR accuracy but also effectively mitigates performance gaps across demographic groups, achieving a synergistic optimization of both accuracy and fairness.

Abstract

Automatic Speech Recognition (ASR) is increasingly used to document clinical encounters, yet its reliability in multilingual and demographically diverse Indian healthcare context remains largely unknown. In this study, we first conduct the systematic audit of ASR performance on real-world psychiatric interview data spanning Kannada, Hindi and Indian English, comparing eight state-of-the-art models including IndicWhisper, WhisperLargeV3, Sarvam, GoogleS2T, Gemma3n, OmniLingual, Vaani, and Gemini. Our results reveal substantial variability across models and languages, with some systems performing competitively in Indian English but failing in regional speech. We further fine-tune two of the best performing opensource models, i.e., Gemma3n and OmniLingual, using various methods. With this, we uncover systematic performance gaps tied to speaker role and gender, raising concerns about equitable deployment in clinical settings, which are further mitigated by fairness-aware fine-tuning. To this end, we propose SamaVaani, a unified debiasing technique that simultaneously improves ASR performance and improves fairness across demographic groups.

Problem

• Automatic Speech Recognition
• clinical ASR
• multilingual
• bias
• fairness

Innovation

• multilingual clinical ASR
• bias auditing
• fairness-aware fine-tuning
• SamaVaani
• demographic equity