HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics

2026-04-30

AI Summary

Current automatic speech recognition (ASR) evaluation metrics, such as word error rate (WER), often fail to accurately reflect human perception of transcription quality and lack user-centered assessment criteria. To address this gap, this study presents the first systematic construction of HATS, a human preference dataset for French ASR, comprising subjective pairwise preference annotations from 143 participants. The authors conduct a comprehensive correlation analysis between human judgments and both lexical-level and embedding-level metrics—including BERTScore and semantic distance—to evaluate their alignment with human perception. The findings reveal the extent to which prevailing ASR metrics correspond to human assessments, offering empirical insights and an open benchmark to support the development of more perceptually aligned evaluation methodologies.

Abstract

Conventionally, Automatic Speech Recognition (ASR) systems are evaluated on their ability to correctly recognize each word contained in a speech signal. In this context, the word error rate (WER) metric is the reference for evaluating speech transcripts. Several studies have shown that this measure is too limited to correctly evaluate an ASR system, which has led to the proposal of other variants of metrics (weighted WER, BERTscore, semantic distance, etc.). However, they remain system-oriented, even when transcripts are intended for humans. In this paper, we firstly present Human Assessed Transcription Side-by-side (HATS), an original French manually annotated data set in terms of human perception of transcription errors produced by various ASR systems. 143 humans were asked to choose the best automatic transcription out of two hypotheses. We investigated the relationship between human preferences and various ASR evaluation metrics, including lexical and embedding-based ones, the latter being those that correlate supposedly the most with human perception.

Problem

• Automatic Speech Recognition
• evaluation metrics
• human perception
• word error rate
• transcription quality

Innovation

• Human-centered evaluation
• Automatic Speech Recognition
• Word Error Rate
• Perceptual assessment
• HATS dataset