Improving Rare-Word Recognition in Zero-Shot Settings

2025-02-17

AI Summary

Whisper exhibits limited zero-shot recognition capability for rare words—such as domain-specific terminology—due to insufficient contextual grounding. To address this, we propose a supervised fine-tuning approach using only 670 hours of English speech data, explicitly optimizing the model’s responsiveness to contextual prompts. Our key contribution is the first demonstration of cross-dataset and cross-lingual contextual bias generalization: the fine-tuned model achieves significant performance gains across 11 heterogeneous English test sets and on unseen languages without any target-language annotations. Experiments show a 45.6% absolute improvement in rare-word recognition accuracy and a 60.8% gain in out-of-vocabulary word recognition. Crucially, the method enables instruction-guided zero-shot transfer while preserving Whisper’s original architecture. This substantially extends Whisper’s practical applicability to specialized domains and low-resource languages.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Whisper, despite being trained on 680K hours of web-scaled audio data, faces difficulty in recognising rare words like domain-specific terms, with a solution being contextual biasing through prompting. To improve upon this method, in this paper, we propose a supervised learning strategy to fine-tune Whisper for contextual biasing instruction. We demonstrate that by using only 670 hours of Common Voice English set for fine-tuning, our model generalises to 11 diverse open-source English datasets, achieving a 45.6% improvement in recognition of rare words and 60.8% improvement in recognition of words unseen during fine-tuning over the baseline method. Surprisingly, our model's contextual biasing ability generalises even to languages unseen during fine-tuning.

Problem

• Enhance rare-word recognition in zero-shot settings
• Fine-tune Whisper for contextual biasing instruction
• Improve recognition of domain-specific and unseen words

Innovation

• Supervised learning for fine-tuning
• Contextual biasing through prompting
• Generalization to unseen languages