Selective Attention Merging for low resource tasks: A case study of Child ASR

2025-01-14

AI Summary

To address the performance limitations of foundation models in low-resource speech tasks—such as child automatic speech recognition (Child ASR)—this paper proposes a knowledge transfer framework based on model merging. The core innovation is Selective Attention (SA)Merge, a fine-grained fusion method that selectively integrates task-relevant attention vectors from a high-resource pretrained model (e.g., Whisper) and a low-resource fine-tuned model at the attention mechanism level. Integrated with SpecAugment and fine-tuning of Whisper-small, our approach achieves a 14% relative reduction in word error rate (WER) on the MyST child speech corpus, yielding an absolute WER of 8.69—the current state-of-the-art. Notably, this work is the first to incorporate interpretability analysis of attention matrices into model merging, thereby significantly enhancing generalization and robustness in low-resource settings.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

While Speech Foundation Models (SFMs) excel in various speech tasks, their performance for low-resource tasks such as child Automatic Speech Recognition (ASR) is hampered by limited pretraining data. To address this, we explore different model merging techniques to leverage knowledge from models trained on larger, more diverse speech corpora. This paper also introduces Selective Attention (SA) Merge, a novel method that selectively merges task vectors from attention matrices to enhance SFM performance on low-resource tasks. Experiments on the MyST database show significant reductions in relative word error rate of up to 14%, outperforming existing model merging and data augmentation techniques. By combining data augmentation techniques with SA Merge, we achieve a new state-of-the-art WER of 8.69 on the MyST database for the Whisper-small model, highlighting the potential of SA Merge for improving low-resource ASR.

Problem

• Speech Recognition
• Children's Voices
• Limited Data

Innovation

• Selective Attention Fusion
• Speech Recognition
• Data Scarcity