Progressive Alignment Objectives for Aligner-Encoder based ASR

2026-06-23

AI Summary

This work addresses the instability and performance degradation in long-form speech recognition caused by the abrupt emergence of alignment information in deep layers of Aligner-Encoder models. To mitigate this issue, the authors propose InterAligner and InterCTC mechanisms that introduce progressive alignment objectives and CTC losses at intermediate encoder layers, enabling the first framework to learn alignment gradually along the depth dimension. Integrated into a 17-layer Conformer architecture, the joint optimization significantly enhances training stability and recognition accuracy, reducing word error rates on LibriSpeech from 5.0/7.8 to 3.1/5.6 on test-clean/test-other, with particularly pronounced improvements in long utterance scenarios.

Abstract

Aligner-Encoders are recently proposed seq2seq end-to-end ASR models that replace decoder attention by predicting the uth token directly from the u-th encoder position, so the encoder must learn the alignment internally without cross-attention or a transducer lattice. In practice, this alignment often forms abruptly in the upper layers, making training sensitive and brittle on long utterances. We propose InterAligner, which adds an intermediate Aligner objective so alignment can form progressively across depth, together with an intermediate CTC loss (InterCTC) to stabilize optimization. On LibriSpeech with a 17-layer Conformer, a final-only Aligner reaches 5.0/7.8 WER (test-clean/other). InterCTC improves to 3.4/6.0, and InterAligner further reduces WER to 3.1/5.6 with the largest gains on long utterances.

Problem

• Aligner-Encoder
• ASR
• alignment
• long utterances
• training stability

Innovation

• InterAligner
• progressive alignment
• Aligner-Encoder
• InterCTC
• end-to-end ASR