Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs

2026-06-28

AI Summary

This work addresses a critical gap in current automatic speech recognition (ASR) evaluation: the lack of consideration for user style preferences, which renders conventional metrics unable to assess whether model outputs align with user instructions regarding numerals, named entities, disfluencies, and casing. To bridge this gap, we propose the first preference-aware ASR evaluation paradigm, introducing a multi-source test set augmented with natural language preference instructions and a selective skip-normalization scoring mechanism. Leveraging a two-stage large language model–assisted data construction pipeline followed by human validation, we release a high-quality dataset with fine-grained preference annotations alongside an open-source evaluation framework. Experiments across four state-of-the-art ASR systems reveal substantial performance variations under different stylistic preferences—variations entirely missed by standard evaluation protocols—thereby demonstrating the necessity and effectiveness of our approach.

Abstract

Popular ASR test sets adopt inconsistent conventions for numbers, disfluencies, entities, and casing, while standard normalizers erase the format distinctions users care about. Current benchmarks therefore cannot measure whether a model follows user preferences for output style. We introduce PreferenceASR, a test set evaluating ASR systems on their ability to follow natural-language preference instructions across four categories: normalization, entities, disfluencies, and case. Built from seven open-source corpora via a two-stage LLM-assisted pipeline with human verification, it is evaluated with a preference-aware normalizer that selectively skips steps matching the active instruction. Benchmarking four models shows rankings shift across preference types, exposing quality differences traditional evaluation obscures. We publicly release the dataset.

Problem

• ASR
• user preference
• output style
• normalization
• benchmarking

Innovation

• Preference-aware ASR
• Speech LLMs
• Output normalization
• User preferences
• Benchmarking