Soft Posterior Speaker Injection for Multi-Talker Speech Recognition

2026-09-01

AI Summary

为解决多说话人语音识别中的重叠语音问题，提出软后验说话人注入（SPSI）方法，通过预测帧级说话人后验并进行特征调制和解码器提示，减少识别错误。

Abstract

Multi-talker automatic speech recognition (MT-ASR) remains challenging under overlapping speech. Hard diarization-based segmentation introduces irreversible errors, whereas serialized output training (SOT) avoids explicit segmentation but does not condition a pretrained encoder on speaker activity. We propose Soft Posterior Speaker Injection (SPSI): a lightweight head predicts frame-level speaker posteriors $\hat{\mathbf{P}}$ and injects them into Whisper through multi-layer feature-wise linear modulation (FiLM) and decoder speaker-memory prompts. On controlled two-speaker LibriSpeech overlap, SPSI reduces utterance-mean constrained permutation word error rate (cpWER) from 50.7\% (SOT) to 49.6\% (one-sided paired bootstrap $p{\approx}0.006$), with a larger reduction in the high-overlap bin (60.4\%$\to$58.8\%). Same-backbone speaker-auxiliary objectives and voice activity detection (VAD) pipelines do not outperform SOT; zero-shot (ZS) LibriCSS is comparable. Freeze-posterior adaptation with overlap-heavy (OV-heavy) continuation reduces held-out LibriCSS cpWER (sessions 8--9) to 32.4\% (versus 37.5\% for SOT). Ablations indicate complementary encoder FiLM and decoder prompts, and that the effective signal is a \emph{soft} simplex-valued speaker share.

Problem

• Multi-talker automatic speech recognition
• overlapping speech
• Hard diarization-based segmentation
• serialized output training (SOT)

Innovation

• Soft Posterior Speaker Injection
• frame-level speaker posteriors
• feature-wise linear modulation
• decoder speaker-memory prompts
• constrained permutation word error rate