How to Leverage Synthetic Speech for LLM-Based ASR Systems?

2026-06-27

AI Summary

In privacy-sensitive domains, the scarcity of real speech data and the distributional gap between synthetic and real speech hinder the effective use of synthetic data in automatic speech recognition (ASR). This work addresses this challenge within the SLAM-ASR framework by revealing, for the first time, that discriminative signals distinguishing real from synthetic speech in large language model (LLM) backbones are predominantly localized in early-to-mid layers. Leveraging this insight, the authors propose a synergistic strategy combining a layer selection module with room impulse response (RIR) augmentation. This approach substantially narrows the distributional gap, achieving performance on par with a full real-data baseline using only 25% of real speech (13.6 hours) and even surpassing it at higher proportions, thereby significantly reducing reliance on real speech data.

Abstract

In regulated domains such as banking and healthcare, where privacy constraints make real speech costly to collect and retain, synthetic speech from modern text-to-speech (TTS) is an appealing alternative for training automatic speech recognition (ASR) without exposing sensitive customer recordings. Yet a persistent distributional gap between synthetic and real data limits how far it can replace genuine recordings. Prior work largely treats this gap as a black box to be engineered around, but in our work, we instead examine its origin directly by probing a SLAM-ASR architecture. Then, we localise where its LLM backbone separates real from synthetic speech and find the discriminative signal concentrated in the early-to-middle layers, where temporal and prosodic perturbations disrupt it most. We further show that representation-level separability, help, but does not directly predict downstream ASR gains. On the other hand, convolving synthetic audio with room impulse responses (RIRs) narrows the gap not by making synthetic speech sound cleaner or more natural, but by reproducing the acoustic irregularities of real recordings. Translating these findings into the training procedure, by adding a layer-selection module combined with RIR augmentation matches a fully real-data baseline using only 25% of the real speech (13.6h) and surpasses it at all higher proportions.

Problem

• synthetic speech
• automatic speech recognition
• distributional gap
• privacy constraints
• LLM-based ASR

Innovation

• synthetic speech
• distributional gap
• SLAM-ASR
• room impulse response (RIR)
• layer-wise analysis