Problem
Research questions and friction points this paper is trying to address.
Caption Quality
Deaf and Hard-of-Hearing (DHH)
Latency
Word Error Rate (WER)
Automated Speech Recognition (ASR)
Are Caption Metrics Broken? Latency, Deaf and Hard of Hearing User Ratings, and Bias across Technologies
🤖 AI Summary
研究通过大规模在线调查评估电视字幕质量，对比不同技术下的字幕延迟对聋人和重听观众的影响，发现现有度量标准并不中立。
📝 Abstract
Live captions on TV often contain errors and timing issues, making it hard for deaf and hard-of-hearing (DHH) viewers to follow dialog. It is essential that caption quality metrics reflect the lived DHH TV viewing experience. To this end, we describe a U.S.-based large-scale online survey with 216 validated participants, who provided 302 responses containing a cumulative 4,832 data points. Participants viewed videos drawn from a pool of 70 clips recorded from live TV, and were asked to rate the caption quality and subjective understanding of the content across four conditions: TV captions as originally recorded with up to 7-12 seconds delay, TV captions synchronized with audio, Automatic Speech Recognition (ASR)-generated captions synchronized with audio, and ASR captions with an average two-second delay. All captions were evaluated against the Word Error Rate (WER), Automated Caption Evaluation (ACE2) and Number, Edition and Recognition (NER) metrics. Results show that TV and ASR captions were rated similarly. For TV captions, all three metrics were moderately-to-highly correlated with viewer ratings, but far less so for ASR captions, making them far from technology-neutral. Additionally, caption latencies significantly impact the viewer experience, especially typical 7-12-second TV delays. We discuss the implications for the adoption of caption quality metrics.
Innovation
Methods, ideas, or system contributions that make the work stand out.
caption latency
Automatic Speech Recognition (ASR)
Word Error Rate (WER)
🔎 Similar Papers
