RAS: a Reliability Oriented Metric for Automatic Speech Recognition

2026-04-27

AI Summary

This work addresses the limitation of conventional automatic speech recognition (ASR) systems, which often produce highly confident yet erroneous transcriptions under noisy or ambiguous conditions—a failure mode not captured by standard word error rate metrics. To enhance reliability, the authors propose an ASR framework that supports active abstention, enabling the model to withhold output when uncertain. They introduce RAS, the first human-preference-based evaluation metric that jointly quantifies transcription informativeness and error-avoidance capability. The abstention-aware model is trained via a combination of supervised bootstrapping and reinforcement learning, with policy optimization guided by a calibrated RAS score. Experimental results demonstrate that the proposed approach significantly improves transcription reliability while maintaining high accuracy.

Technology Category

• Machine Learning: Active Learning
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Automatic speech recognition systems often produce confident yet incorrect transcriptions under noisy or ambiguous conditions, which can be misleading for both users and downstream applications. Standard evaluation based on Word Error Rate focuses solely on accuracy and fails to capture transcription reliability. We introduce an abstention-aware transcription framework that enables ASR models to explicitly abstain from uncertain segments. To evaluate reliability under abstention, we propose RAS, a reliability-oriented metric that balances transcription informativeness and error aversion, with its trade-off parameter calibrated by human preference. We then train an abstention-aware ASR model through supervised bootstrapping followed by reinforcement learning. Our experiments demonstrate substantial improvements in transcription reliability while maintaining competitive accuracy.

Problem

• Automatic Speech Recognition
• Reliability
• Abstention
• Word Error Rate
• Uncertainty

Innovation

• abstention-aware ASR
• reliability metric
• RAS
• reinforcement learning
• transcription uncertainty