Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR

2026-07-10

AI Summary

This work addresses the challenges of word fragmentation and autoregressive collapse in lightweight autoregressive speech recognition models when applied to morphologically rich, non-Latin scripts such as Bengali, primarily caused by English-centric byte-level tokenizers. To overcome this without requiring full model re-pretraining, the authors propose a cross-script lexical transplantation method: replacing the decoder’s byte-level vocabulary with a Bengali-specific WordPiece vocabulary derived from BanglaBERT and rescaling the embedding matrix to better align with the target language’s linguistic structure. This approach enables the first efficient and reproducible tokenizer replacement for compact ASR architectures like Moonshine. Evaluated on the Lipi-Ghor dataset, the method achieves a word error rate of 21.54%, a real-time factor of 0.0053, an 85.8% reduction in autoregressive sequence length, and a substantial drop in word fertility from 9.16 to 1.30.

Abstract

Lightweight speech recognition models are critical for edge deployment, yet highly optimized architectures like Moonshine often fail on morphologically rich, non-Latin languages such as Bengali. This study identifies the root cause of this failure as the model's English-centric byte-level tokenizer, which fragments Bengali words into high-fertility byte chains and triggers catastrophic autoregressive collapse during inference. To resolve this, a novel vocabulary transplantation pipeline is proposed to replace the decoder vocabulary with the native-script BanglaBERT WordPiece vocabulary and resize the corresponding token embedding matrix. Experimental results demonstrate a reduction in token fertility from 9.16 to 1.30. By decreasing autoregressive sequence length by 85.8%, decoding instability is entirely mitigated. When evaluated on the 882-hour Lipi-Ghor dataset, the modified architecture achieves a competitive 21.54% Word Error Rate (WER) and a Real-Time Factor (RTF) of 0.0053. Ultimately, this research provides a scalable, reproducible blueprint for cross-script adaptation of compact ASR models without the need for resource-intensive pre-training.

Problem

• tokenizer
• autoregressive collapse
• Bengali ASR
• edge-efficient
• vocabulary fragmentation

Innovation

• Tokenizer Transplantation
• Autoregressive Collapse
• Edge-Efficient ASR
• Cross-Script Adaptation
• Word Fertility Reduction