Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming

2026-06-24

AI Summary

This work proposes the first fully differentiable neural forced alignment architecture, addressing the long-standing limitation of traditional methods that rely on HMM-GMM frameworks and cannot be jointly optimized with modern end-to-end speech models. The approach employs a dual-branch encoder to jointly model phoneme recognition and boundary detection, integrates a differentiable soft dynamic programming decoder, and introduces a contrastive loss function that explicitly distinguishes between steady-state and transitional speech segments. Trained end-to-end, the system surpasses state-of-the-art performance on a manually annotated English phoneme alignment benchmark while demonstrating strong word-level generalization and cross-lingual transfer capabilities.

Abstract

Recent advances in sequence modeling have significantly improved ASR systems, bringing them close to human-level recognition accuracy and enhancing robustness across diverse acoustic conditions and languages. In contrast, Forced Alignment has not experienced comparable progress, and traditional HMM-GMM frameworks remain widely adopted and highly competitive. To address this gap, we propose an end-to-end, fully differentiable neural architecture specifically designed for phoneme alignment. The model consists of an encoder that processes the input signal and a decoder that produces alignment decisions. The encoder is structured into two complementary branches: one dedicated to phoneme identity verification and the other to phoneme boundary detection. The decoder is implemented as a trainable module based on differentiable soft dynamic programming. The entire system is optimized end-to-end using a novel contrastive loss that encourages clear separation between steady-state phoneme regions and transition boundaries. The proposed approach outperforms the current state of the art in phoneme alignment on hand-annotated English benchmarks, achieves strong word-level generalization results, and demonstrates generalization on unseen languages.

Problem

• Forced Alignment
• Phoneme Alignment
• End-to-End Learning
• Differentiable Dynamic Programming
• Sequence Modeling

Innovation

• differentiable alignment
• soft dynamic programming
• end-to-end phoneme alignment
• contrastive loss
• neural forced alignment