TSPC: A Two-Stage Phoneme-Centric Architecture for code-switching Vietnamese-English Speech Recognition

2025-09-07

AI Summary

Addressing the challenges of phoneme-level language switching modeling, severe cross-lingual phonemic ambiguity, and heavy reliance on labeled data in Vietnamese–English code-switching (CS) automatic speech recognition (ASR), this paper proposes a phoneme-centric two-stage architecture, TSPC. Methodologically, TSPC introduces (1) an extended Vietnamese phoneme inventory as a cross-lingual intermediate representation to explicitly model phoneme-level language transitions, and (2) a decoupled design separating phoneme adaptation from language identification, thereby enhancing robustness to subtle acoustic variations. Evaluated on a Vietnamese–English CS speech dataset, TSPC achieves a word error rate (WER) of 20.8% using limited training resources—substantially outperforming end-to-end baselines such as PhoWhisper-base. These results empirically validate the efficacy of phoneme-level intermediate representations for low-resource CS-ASR.

Technology Category

• Natural Language Processing: Speech
• Computer Vision: Language and Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

Code-switching (CS) presents a significant challenge for general Auto-Speech Recognition (ASR) systems. Existing methods often fail to capture the subtle phonological shifts inherent in CS scenarios. The challenge is particularly difficult for language pairs like Vietnamese and English, where both distinct phonological features and the ambiguity arising from similar sound recognition are present. In this paper, we propose a novel architecture for Vietnamese-English CS ASR, a Two-Stage Phoneme-Centric model (TSPC). The TSPC employs a phoneme-centric approach, built upon an extended Vietnamese phoneme set as an intermediate representation to facilitate mixed-lingual modeling. Experimental results demonstrate that TSPC consistently outperforms existing baselines, including PhoWhisper-base, in Vietnamese-English CS ASR, achieving a significantly lower word error rate of 20.8% with reduced training resources. Furthermore, the phonetic-based two-stage architecture enables phoneme adaptation and language conversion to enhance ASR performance in complex CS Vietnamese-English ASR scenarios.

Problem

• Addresses code-switching Vietnamese-English speech recognition challenges
• Captures phonological shifts in mixed-language speech scenarios
• Resolves ambiguity from similar sounds between Vietnamese and English

Innovation

• Two-Stage Phoneme-Centric architecture for CS ASR
• Extended Vietnamese phoneme set as intermediate representation
• Phoneme adaptation and language conversion enhancement