Improving the Inclusivity of Dutch Speech Recognition by Fine-tuning Whisper on the JASMIN-CGN Corpus

2025-02-24

AI Summary

This study systematically investigates performance disparities of the Whisper model in Dutch automatic speech recognition (ASR) across marginalized speaker groups—children, elderly speakers, and non-native speakers—and proposes targeted fine-tuning to enhance inclusivity. Using the multi-population annotated corpus JASMIN-CGN, we conduct subgroup-specific fine-tuning of Whisper-large and evaluate performance via a hierarchical word error rate (WER) framework. Our empirical analysis reveals, for the first time, that fine-tuning reduces WER by up to 81% (for native-speaking children) and 65% (for native-speaking elderly) relative to zero-shot inference, substantially improving recognition accuracy across all four underrepresented groups. The core contribution is the empirical validation that subgroup-specific fine-tuning is critical for building inclusive ASR systems. Moreover, we establish a reproducible methodology and empirical benchmark for fairness-aware ASR research in low-resource, high-diversity settings.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

We test and study the variation in speech recognition of fine-tuned versions of the Whisper model on child, elderly and non-native Dutch speech from the JASMIN-CGN corpus. Our primary goal is to evaluate how speakers' age and linguistic background influence Whisper's performance. Whisper achieves varying Word Error Rates (WER) when fine-tuned on subpopulations of specific ages and linguistic backgrounds. Fine-tuned performance is remarkably better than zero-shot performance, achieving a relative reduction in WER of 81% for native children, 72% for non-native children, 67% for non-native adults, and 65% for native elderly people. Our findings underscore the importance of training speech recognition models like Whisper on underrepresented subpopulations such as children, the elderly, and non-native speakers.

Problem

• Enhancing Dutch speech recognition inclusivity
• Evaluating Whisper model on diverse speakers
• Improving recognition for underrepresented subpopulations

Innovation

• Fine-tuning Whisper model
• JASMIN-CGN corpus utilization
• Inclusive speech recognition improvement