🤖 AI Summary
This work addresses the challenge of automatic speech recognition (ASR) errors caused by rare or missing domain-specific entities in specialized fields such as finance and healthcare, where post-processing correction is often difficult. The authors propose an agent-based entity correction framework that uniquely integrates ASR n-best hypotheses with tool-augmented large language models (LLMs), leveraging external entity retrieval and constraint-guided decoding for precise error correction. Several hypothesis fusion strategies—including 1-Best, Entity-Aware Select, ROVER Ensemble, and a novel LLM-Select—are introduced to substantially improve entity recall and accuracy. Evaluated across five datasets, the approach achieves relative reductions of 8%–46% in entity word error rate (E-WER) and up to a 22-percentage-point gain in entity recall, while maintaining overall word error rate (WER) stability.
📝 Abstract
Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. In domains such as finance, medicine, and air traffic control, these errors are costly. If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. To address this, we introduce RECOVER, an agentic correction framework that serves as a tool-using agent. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints. The hypotheses are used using different strategies, namely, 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, and LLM-Select. Evaluated across five diverse datasets, it achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points. The LLM-Select achieves the best overall performance in entity correction while maintaining overall WER.