Decoding while Adapting: Zero-Shot Online Speaker Adaptation via Audio-Textual Prompts for Elderly Speech Recognition

2026-06-15

AI Summary

This work addresses the performance degradation in elderly speech recognition caused by speaker variability, particularly the challenge of zero-shot real-time adaptation to unseen speakers. The authors propose an online speaker adaptation method based on cross-utterance audio-text prompts, which dynamically fuses acoustic and textual embeddings from both current and historical utterances during decoding to generate compact and consistent speaker representations. This approach introduces a cross-modal prompting mechanism into online adaptation for the first time, requiring no prior data from the target speaker. Experiments demonstrate absolute reductions of 0.61% and 1.22% (relative improvements of 2.99% and 4.48%) in WER and CER on the DementiaBank Pitt and JCCOCC MoCA datasets, respectively, with inference speeds up to 9.83× faster than offline batch processing, significantly outperforming conventional i/x-vector and ECAPA-TDNN features.

Abstract

This paper proposes a novel cross-utterance audio-textual prompts based speaker adaptation approach for elderly speech recognition. It enables zero-shot, real-time adaptation to unseen speakers. Speech and text embeddings are extracted from the current and a few preceding utterances, before being fused in a cross-modal manner to produce compact speaker prompts that are more consistent than i/x-vectors and ECAPA-TDNN features. Experiments on the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets suggest that the proposed online adaptation outperforms the speaker-independent (SI) model by statistically significant word error rate (WER) or character error rate (CER) reductions of 0.61% and 1.22% absolute (2.99% and 4.48% relative). Real-time factor (RTF) speed-up ratios of up to 9.83 times are obtained over offline batch-mode adaptation.

Problem

• zero-shot adaptation
• online speaker adaptation
• elderly speech recognition
• audio-textual prompts
• real-time adaptation

Innovation

• zero-shot adaptation
• audio-textual prompts
• online speaker adaptation
• elderly speech recognition
• cross-modal fusion