Jointly Improving Dialect Identification and ASR in Indian Languages using Multimodal Feature Fusion

2026-07-02

AI Summary

This study addresses the challenge of degraded performance in automatic speech recognition (ASR) and dialect identification (DID) for Indian languages, which stems from severe data scarcity and pronounced dialectal variation. To tackle this issue, the authors propose the first end-to-end multimodal joint optimization framework. The approach employs a bottleneck encoder to extract dialect-specific features from a Conformer architecture and integrates RoBERTa to process CTC embeddings derived from ASR outputs. A gating mechanism fuses these two feature streams, followed by an attention-based encoder to enhance representation quality. Evaluated across 33 dialects spanning eight Indian languages, the method achieves a DID accuracy of 81.63% on average, while reducing character error rate (CER) and word error rate (WER) to 4.65% and 17.73%, respectively, substantially mitigating the adverse impact of dialectal diversity under low-resource conditions.

Abstract

Automatic Speech Recognition (ASR) and Dialect Identification (DID) are crucial for Indian languages, many of which are low-resource and exhibit significant dialectal differences. Existing methods often optimize ASR or DID individually, resulting in performance trade-offs. In this work, we propose a multimodal framework that jointly improves ASR and DID. Our method employs a Bottleneck Encoder to extract dialectal features from Conformer-based speech representations and a RoBERTa encoder to process ASR-generated CTC embeddings. A gating mechanism merges these features, followed by an attention encoder to refine the representations. The learned embeddings are concatenated with Conformer outputs to enhance ASR features. Evaluated on eight Indian languages with thirty-three dialects, our method achieves an average DID accuracy of 81.63% and average CER and WER of 4.65% and 17.73%, respectively. These results highlight the effectiveness of our method for joint ASR-DID modeling.

Problem

• Automatic Speech Recognition
• Dialect Identification
• Low-resource Languages
• Multimodal Feature Fusion
• Indian Languages

Innovation

• multimodal feature fusion
• joint ASR-DID modeling
• bottleneck encoder
• gating mechanism
• Conformer-RoBERTa integration