Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancement

2026-02-04

IEEE Transactions on Audio, Speech, and Language Processing

AI Summary

This work addresses the significant performance degradation of pre-trained speech recognition and enhancement models under unseen cross-domain compound distortions, such as unknown noise and channel effects. To tackle this challenge, the authors propose URSA-GAN, a unified domain-aware generative adversarial network framework that jointly models domain characteristics through dedicated noise and channel encoders. The extracted domain embeddings conditionally drive speech generation, enabling target-domain alignment while preserving linguistic content. Furthermore, dynamic stochastic perturbation regularization is introduced to enhance model generalization. Experimental results demonstrate that the proposed method reduces the character error rate for speech recognition by 16.16% and improves perceptual speech enhancement metrics by 15.58% in unseen cross-domain scenarios.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Generation
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Pre-trained models for automatic speech recognition (ASR) and speech enhancement (SE) have exhibited remarkable capabilities under matched noise and channel conditions. However, these models often suffer from severe performance degradation when confronted with domain shifts, particularly in the presence of unseen noise and channel distortions. In view of this, we in this paper present URSA-GAN, a unified and domain-aware generative framework specifically designed to mitigate mismatches in both noise and channel conditions. URSA-GAN leverages a dual-embedding architecture that consists of a noise encoder and a channel encoder, each pre-trained with limited in-domain data to capture domain-relevant representations. These embeddings condition a GAN-based speech generator, facilitating the synthesis of speech that is acoustically aligned with the target domain while preserving phonetic content. To enhance generalization further, we propose dynamic stochastic perturbation, a novel regularization technique that introduces controlled variability into the embeddings during generation, promoting robustness to unseen domains. Empirical results demonstrate that URSA-GAN effectively reduces character error rates in ASR and improves perceptual metrics in SE across diverse noisy and mismatched channel scenarios. Notably, evaluations on compound test conditions with both channel and noise degradations confirm the generalization ability of URSA-GAN, yielding relative improvements of 16.16% in ASR performance and 15.58% in SE metrics.

Problem

• domain shift
• noise distortion
• channel distortion
• speech recognition
• speech enhancement

Innovation

• domain adaptation
• GAN-based speech generation
• dual-embedding architecture
• dynamic stochastic perturbation
• cross-domain robustness