Cross-lingual Embedding Clustering for Hierarchical Softmax in Low-Resource Multilingual Speech Recognition

2025-01-29

AI Summary

To address weak decoding generalization in low-resource multilingual automatic speech recognition (ASR), this paper proposes a hierarchical Softmax (H-Softmax) decoder optimization method based on cross-lingual semantic embedding clustering. Unlike conventional H-Softmax approaches that construct Huffman trees using shallow statistical features, our method innovatively incorporates cross-lingual word embedding similarity into the tree topology design: K-means clustering groups semantically similar tokens—including those across languages—into shared internal nodes, thereby enabling low-resource languages to leverage semantic representations from high-resource languages. Evaluated on a downsampled multilingual ASR dataset spanning 15 languages, the proposed method yields significant improvements in word error rate for low-resource languages. Results demonstrate that semantic-driven, cross-lingual decoder representation sharing simultaneously enhances both decoding efficiency and acoustic modeling performance.

Technology Category

• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Semi-Supervised Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We present a novel approach centered on the decoding stage of Automatic Speech Recognition (ASR) that enhances multilingual performance, especially for low-resource languages. It utilizes a cross-lingual embedding clustering method to construct a hierarchical Softmax (H-Softmax) decoder, which enables similar tokens across different languages to share similar decoder representations. It addresses the limitations of the previous Huffman-based H-Softmax method, which relied on shallow features in token similarity assessments. Through experiments on a downsampled dataset of 15 languages, we demonstrate the effectiveness of our approach in improving low-resource multilingual ASR accuracy.

Problem

• Multilingual Speech Recognition
• Cross-lingual Feature Grouping
• Low-resource Languages

Innovation

• Cross-lingual Feature Grouping
• Resource-poor Language Recognition
• Improved Accuracy