Meta-Whisper: Speech-Based Meta-ICL for ASR on Low-Resource Languages

2024-09-16

arXiv.org

AI Summary

Low-resource automatic speech recognition (ASR) suffers from data scarcity and poor cross-lingual generalization; existing adaptation methods often rely on external resources, incur high computational costs, or lack support for test-time adaptation. To address these limitations, this work introduces Meta-In-Context Learning (Meta-ICL) to ASR for the first time, proposing a lightweight, fine-tuning-free adaptation framework. It leverages k-nearest neighbor (k-NN) retrieval to dynamically identify semantically similar speech samples and construct task-aware in-context examples, enabling zero-shot or few-shot cross-lingual transfer for Whisper. Evaluated on the ML-SUPERB multilingual benchmark, our approach substantially reduces character error rates (CER) for low-resource languages, achieving an average improvement of 32.7%. Results demonstrate the method’s effectiveness, efficiency, and scalability—without requiring parameter updates or external linguistic resources.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper presents Meta-Whisper, a novel approach to improve automatic speech recognition (ASR) for low-resource languages using the Whisper model. By leveraging Meta In-Context Learning (Meta-ICL) and a k-Nearest Neighbors (KNN) algorithm for sample selection, Meta-Whisper enhances Whisper's ability to recognize speech in unfamiliar languages without extensive fine-tuning. Experiments on the ML-SUPERB dataset show that Meta-Whisper significantly reduces the Character Error Rate (CER) for low-resource languages compared to the original Whisper model. This method offers a promising solution for developing more adaptable multilingual ASR systems, particularly for languages with limited resources.

Problem

• Improving ASR for low-resource languages with limited data
• Overcoming inefficiencies in existing adaptation strategies
• Enabling few-shot generalization without target domain fine-tuning

Innovation

• Meta-learning for low-resource ASR adaptation
• Speech in-context learning without fine-tuning
• Few-shot multilingual ASR performance improvement