speech recognition

To address the insufficient accuracy and robustness of the Kaldi ASR system across diverse scenarios, this work proposes three core optimizations: (1) a lightweight Conformer acoustic model integrating multi-stream TDNN-F architecture to enhance time-frequency modeling capability; (2) a dynamic hyperparameter tuning framework based on Bayesian optimization, jointly optimizing acoustic and language model hyperparameters; and (3) a dynamic n-gram language model pruning and caching strategy that balances recognition accuracy and inference efficiency. Experiments demonstrate that the proposed method reduces word error rate by 12.7% on Common Voice and 9.3% on AISHELL-2, significantly mitigating overfitting and improving cross-domain generalization. Moreover, the system supports low-latency, scalable industrial deployment.

Large language and speech models suffer from poor generalization to minority languages, dialects, and sociolinguistic variants due to skewed training data—exacerbating the digital divide and inequitable technology access. This project pioneers an integrated framework for linguistic justice, systematically bridging computational linguistics, sociolinguistics, and AI ethics. Methodologically, it combines corpus analysis, bias quantification, low-resource modeling, and interdisciplinary qualitative research. Key contributions include: (1) a reproducible, multi-dimensional evaluation metric suite for linguistic inclusivity; (2) a consensus-based white paper and 12 actionable, industry-deployable recommendations; and (3) an open-source dialect adaptation toolkit to advance standardized technical support for linguistic diversity. Collectively, these outcomes address the structural misalignment between model capabilities and real-world linguistic heterogeneity, offering both methodological rigor and practical pathways toward equitable AI.

This paper surveys the technological evolution of modern automatic speech recognition (ASR), addressing persistent challenges in real-world deployment, including streaming inference, edge-device compatibility, and fairness. Methodologically, it traces architectural advances from hybrid systems to end-to-end models—CTC, RNN-T, Transformer, and Conformer—and critically analyzes the impact of self-supervised pretraining (e.g., wav2vec 2.0, Whisper) and weakly supervised large-scale training in reducing annotation dependency while enhancing data diversity and robustness. Its contributions are threefold: (1) the first unified analysis of practical constraints—latency, computational efficiency, and demographic fairness; (2) a consolidated overview of performance trends across standard benchmarks (LibriSpeech, Switchboard) and standardized evaluation practices (e.g., WER); and (3) a proposed next-generation ASR paradigm centered on “foundation models + weak supervision + diverse data,” offering a theoretical framework and practical guidelines for jointly optimizing efficiency, generalization, and deployability.

Speech-language models struggle to jointly optimize speech understanding and textual capability preservation when labeled speech instruction data is scarce. Method: We propose an end-to-end paradigm requiring zero speech instruction data. It aligns a pre-trained speech model with a large language model (LLM) via cross-modal alignment to automatically synthesize high-quality speech-text pairs, thereby injecting paralinguistic understanding; concurrently, the LLM’s parameters are frozen, and only a lightweight speech adapter is introduced to prevent textual capability degradation. Contribution/Results: This work pioneers speech-language co-modeling without any speech instruction fine-tuning data, while supporting joint optimization of complex textual instructions (e.g., chain-of-thought reasoning, format control) and speech understanding. Our approach achieves state-of-the-art performance on Dynamic-SUPERB and AIR-Bench-Chat, significantly reducing reliance on manual speech annotation.

Nigerian Pidgin, a low-resource African language, suffers from a critical lack of publicly available speech data and pre-trained automatic speech recognition (ASR) models. Method: We introduce the first large-scale, open-source speech-text parallel corpus for Nigerian Pidgin and develop the first end-to-end ASR system for this language, trained on the corpus using QuartzNet and Jasper architectures with Connectionist Temporal Classification (CTC) loss and greedy decoding. Contribution/Results: Our model achieves a word error rate (WER) of 0.77% on a held-out test set—substantially outperforming existing baselines. All data, models, and training code are publicly released, establishing the first reproducible benchmark for Nigerian Pidgin ASR and providing a practical framework for low-resource language ASR research.

This work addresses the challenge of incomparable evaluations and irreproducible results in speech understanding models, which often arise from discrepancies in post-processing, data handling, and pipeline design during deployment-oriented model selection. To this end, the authors propose SURE, a unified experimental framework that enables fair evaluation across diverse paradigms—from conventional pipelines to speech large language models—under realistic acoustic and linguistic stressors. SURE achieves this through standardized prediction formats, consistent normalization strategies, and a unified scoring mechanism. Furthermore, it introduces an agent-assisted training conversion pipeline that automatically maps published code into versioned, executable training workflows. This study presents the first unified and reproducible approach for both evaluating and training speech understanding systems across modeling paradigms, substantially enhancing comparability and reproducibility in real-world deployment scenarios.

This work addresses the performance limitations of automatic speech recognition (ASR) for low-resource languages such as Swahili, which stem from severe scarcity of labeled training data. To mitigate this challenge, the authors propose a reproducible continual pre-training framework that leverages unlabeled audio through pseudo-labeling, followed by supervised fine-tuning, using only 20,000 labeled utterances. Built upon the wav2vec2-bert-2.0 architecture, the approach achieves a word error rate (WER) of 3.24% on the Common Voice Swahili test set—representing an 82% relative improvement over the baseline and a 61% reduction compared to the previous state-of-the-art academic system (8.3% WER). This substantial gain demonstrates a significant decrease in reliance on annotated data while maintaining high recognition accuracy.

This work addresses the limitations of pseudo-labeling in semi-supervised speech recognition, which is prone to confirmation bias and error accumulation that degrade model performance. To mitigate these issues, the authors propose ReHear, a novel framework that, for the first time, integrates an instruction-tuned audio-aware large language model into the self-training loop. By jointly modeling automatic speech recognition (ASR) hypotheses and raw audio inputs, ReHear enables iterative refinement and correction of phoneme-level pseudo-labels. This approach effectively suppresses error propagation and consistently outperforms both conventional pseudo-labeling baselines and fully supervised methods across multiple benchmark datasets, yielding significant improvements in end-to-end speech recognition accuracy.

This work investigates the effective utilization of unpaired data in unsupervised speech recognition, establishing for the first time the theoretical conditions under which such learning is feasible and deriving a provable upper bound on the classification error. Building upon this theoretical framework, the authors propose a single-stage sequence-level cross-entropy loss that directly optimizes the sequence-level objective. Through rigorous theoretical analysis, derivation of the error bound, and simulation experiments, they validate the correctness of the proposed bound and demonstrate that the new loss function significantly improves model performance in unsupervised settings. This study thus provides both a solid theoretical foundation and a practical training methodology for advancing unsupervised speech recognition.

Training end-to-end text-to-speech (TTS) models with purely synthetic data remains underexplored, particularly regarding feasibility, robustness, and controllability compared to real speech data. Method: This study systematically evaluates FastSpeech 2- and VITS-based TTS models trained exclusively on synthetic speech, conducting ablation experiments by controlling textual richness, speaker diversity, environmental noise level, and speaking style. Evaluation integrates MOS, WER, CMOS, and subjective listening tests. Contribution/Results: To our knowledge, this is the first empirical demonstration that synthetic-data-only training achieves a MOS of 4.12—significantly surpassing the real-data baseline (3.78) at equivalent scale. The synthetic-trained models exhibit 27% higher robustness to accent and noise, and 31% improved cross-speaker generalization similarity. Key findings identify high text/speaker diversity and low environmental noise as primary drivers of robustness, while standard speaking style accelerates convergence. These results establish a theoretically grounded, cost-effective paradigm for controllable, high-quality TTS data curation.