Adopting Whisper for Confidence Estimation

2025-02-19

AI Summary

This work addresses the reliance of speech recognition confidence estimation on hand-crafted, lightweight Confidence Estimation Modules (CEMs). We propose an end-to-end unified modeling approach: directly fine-tuning the Whisper model to jointly output word-level confidence scores during decoding—eliminating the need for any external CEM. Our core innovation lies in the first use of the ASR backbone itself as the confidence estimator, achieved through joint audio–hypothesis encoding and confidence score regression. Experiments demonstrate that Whisper-tiny matches CEM performance on in-domain data and outperforms it across all eight out-of-domain datasets; Whisper-large surpasses the CEM baseline significantly on every test set. These results validate the proposed method’s dual advantages: superior generalization across domains and architectural simplicity—achieving confidence estimation without auxiliary modules.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent research on word-level confidence estimation for speech recognition systems has primarily focused on lightweight models known as Confidence Estimation Modules (CEMs), which rely on hand-engineered features derived from Automatic Speech Recognition (ASR) outputs. In contrast, we propose a novel end-to-end approach that leverages the ASR model itself (Whisper) to generate word-level confidence scores. Specifically, we introduce a method in which the Whisper model is fine-tuned to produce scalar confidence scores given an audio input and its corresponding hypothesis transcript. Our experiments demonstrate that the fine-tuned Whisper-tiny model, comparable in size to a strong CEM baseline, achieves similar performance on the in-domain dataset and surpasses the CEM baseline on eight out-of-domain datasets, whereas the fine-tuned Whisper-large model consistently outperforms the CEM baseline by a substantial margin across all datasets.

Problem

• Word-level confidence estimation
• End-to-end Whisper model
• Fine-tuning for confidence scores

Innovation

• Whisper for confidence estimation
• Fine-tuned end-to-end approach
• Outperforms CEM baselines