Automatic Speech Recognition in the Modern Era: Architectures, Training, and Evaluation

2025-10-11

AI Summary

This paper surveys the technological evolution of modern automatic speech recognition (ASR), addressing persistent challenges in real-world deployment, including streaming inference, edge-device compatibility, and fairness. Methodologically, it traces architectural advances from hybrid systems to end-to-end models—CTC, RNN-T, Transformer, and Conformer—and critically analyzes the impact of self-supervised pretraining (e.g., wav2vec 2.0, Whisper) and weakly supervised large-scale training in reducing annotation dependency while enhancing data diversity and robustness. Its contributions are threefold: (1) the first unified analysis of practical constraints—latency, computational efficiency, and demographic fairness; (2) a consolidated overview of performance trends across standard benchmarks (LibriSpeech, Switchboard) and standardized evaluation practices (e.g., WER); and (3) a proposed next-generation ASR paradigm centered on “foundation models + weak supervision + diverse data,” offering a theoretical framework and practical guidelines for jointly optimizing efficiency, generalization, and deployability.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Automatic Speech Recognition (ASR) has undergone a profound transformation over the past decade, driven by advances in deep learning. This survey provides a comprehensive overview of the modern era of ASR, charting its evolution from traditional hybrid systems, such as Gaussian Mixture Model-Hidden Markov Models (GMM-HMMs) and Deep Neural Network-HMMs (DNN-HMMs), to the now-dominant end-to-end neural architectures. We systematically review the foundational end-to-end paradigms: Connectionist Temporal Classification (CTC), attention-based encoder-decoder models, and the Recurrent Neural Network Transducer (RNN-T), which established the groundwork for fully integrated speech-to-text systems. We then detail the subsequent architectural shift towards Transformer and Conformer models, which leverage self-attention to capture long-range dependencies with high computational efficiency. A central theme of this survey is the parallel revolution in training paradigms. We examine the progression from fully supervised learning, augmented by techniques like SpecAugment, to the rise of self-supervised learning (SSL) with foundation models such as wav2vec 2.0, which drastically reduce the reliance on transcribed data. Furthermore, we analyze the impact of largescale, weakly supervised models like Whisper, which achieve unprecedented robustness through massive data diversity. The paper also covers essential ecosystem components, including key datasets and benchmarks (e.g., LibriSpeech, Switchboard, CHiME), standard evaluation metrics (e.g., Word Error Rate), and critical considerations for real-world deployment, such as streaming inference, on-device efficiency, and the ethical imperatives of fairness and robustness. We conclude by outlining open challenges and future research directions.

Problem

• Surveying evolution from hybrid to end-to-end neural ASR architectures
• Analyzing training paradigm shifts from supervised to self-supervised learning
• Evaluating deployment challenges including streaming and ethical considerations

Innovation

• End-to-end neural architectures replace hybrid systems
• Transformer models use self-attention for efficiency
• Self-supervised learning reduces transcribed data reliance