Improving low-resource ASR using bilingual fine-tuning with language identification: a cross-linguistic evaluation

2026-06-16

AI Summary

This work addresses the challenge of automatic speech recognition (ASR) for low-resource languages, where performance is hindered by scarce labeled data. The authors propose a unified bilingual fine-tuning framework that incorporates language identifier prefixes during training to jointly model language identification and speech transcription tasks. During inference, they further explore explicitly injecting language identifiers to enhance robustness. Evaluated across nine language pairs, the results demonstrate that ASR performance improves significantly when language identification accuracy is high; notably, even in cases of misidentification, providing the ground-truth language identifier at inference consistently yields better transcription outcomes. These findings underscore the critical role of accurate language identification in low-resource ASR and validate the efficacy of the proposed language identifier mechanism.

Abstract

This study explores how bilingual fine-tuning affects automatic speech recognition (ASR) in low-resource languages. We evaluate this method across nine linguistically and geographically diverse language pairs, covering a range of language families and writing systems. To distinguish the two languages, during training, we pre-pend each input text with a language identification token. At inference, the model jointly predicts both the language and transcription from the speech input alone. As texts for which the language is incorrectly determined show low ASR performance, we also conduct a follow-up experiment in which the language identification token is provided both during training and inference. Our results show that bilingual fine-tuning can be beneficial when language identification accuracy is high, and that in cases where language identification performance is low, including the language identification token at inference helps to improve ASR performance.

Problem

• low-resource ASR
• bilingual fine-tuning
• language identification
• cross-linguistic evaluation

Innovation

• bilingual fine-tuning
• language identification token
• low-resource ASR
• cross-linguistic evaluation
• joint language-transcription prediction