Why We Need Speech to Evaluate Speech Translation

2026-05-27

AI Summary

Existing speech translation evaluation metrics struggle to effectively capture voice-specific attributes such as speaker gender, prosody, and emphasis. This work systematically exposes this limitation for the first time and constructs a targeted contrastive dataset to assess how both text- and speech-based quality estimation methods handle these phenomena. We propose SpeechCOMET, a model that integrates a speech encoder for quality estimation, and introduce SpeechLLM as an automatic evaluator. Experimental results show that both approaches match or surpass text-based baselines on standard translation quality benchmarks; however, they remain unstable on voice-specific dimensions, revealing three critical bottlenecks. These findings underscore the urgent need for speech-specialized training data and truly speech-conditioned evaluation models.

Abstract

Speech translation models are increasingly capable of preserving speech-specific information (e.g., speaker gender, prosody, and emphasis), yet evaluation metrics remain blind to such phenomena. We meta-evaluate both text- and speech-based quality estimation metrics on two contrastive datasets targeting gender agreement and prosody, and find that both fall short, even when given direct access to the speech signal. We then train SpeechCOMET, a family of quality estimation models with speech encoders, and evaluate a state-of-the-art SpeechLLM as a judge. Both match or exceed text-based COMET on standard quality estimation, but neither consistently assesses speech-specific phenomena. We identify three causes: (1) speech-specific features are not reliably preserved in current encoders, (2) models tend to ignore the speech source signal, and (3) quality estimation training data contains too few relevant examples. We release all models and code, and argue that progress requires dedicated speech-specific training data and models that genuinely condition on speech.

Problem

• speech translation
• evaluation metrics
• speech-specific information
• quality estimation
• prosody

Innovation

• Speech Translation Evaluation
• SpeechCOMET
• Speech-specific Features
• Quality Estimation
• SpeechLLM