Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study

2026-08-26

AI Summary

本文通过比较生成式与编码器大型语言模型在自动语音识别评估中的表现，探索了不同配置下的语义相似度评价方法，以提高评估准确性。

Abstract

Automatic Speech Recognition (ASR) is typically evaluated using Word Error Rate (WER), which poorly reflects semantic similarity. While embedding-based metrics correlate better with human judgments, the respective roles of encoder and decoder-based Large Language Models (LLMs) remain underexplored. This paper presents a comparative study of both families for ASR evaluation. We analyze BERTScore and SemDist across different LLMs, layers, and pooling strategies, showing that both metrics can achieve strong correlation with human judgments when properly configured. For decoder models, we investigate generative LLMs in two settings: pairwise hypothesis selection via prompting and direct qualitative error classification. Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.

Problem

• Automatic Speech Recognition
• Word Error Rate
• Semantic Similarity
• Large Language Models
• Embedding-based Metrics

Innovation

• Generative LLMs
• Encoder-based Metrics
• ASR Evaluation
• Human Judgments Correlation
• Interpretability