STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity

2026-06-24

AI Summary

This study addresses the challenge that current speech-to-speech translation (S2ST) systems struggle to preserve expressive elements from the source speech—such as emotion, contextual style, and non-linguistic vocalizations—and lack scalable evaluation benchmarks. The authors formally define and quantify multidimensional expressiveness metrics, introduce STEB, a bilingual (Chinese–English) expressive S2ST evaluation benchmark, and propose an LLM-driven, reference-free assessment method using a caption-then-summarize framework for structured attribute comparison. Combining acoustic annotations, large language model judgments, and multidimensional human evaluations on a 32.6-hour dataset, they evaluate six representative S2ST systems, revealing significant deficiencies in preserving emotion (max 3.82/5) and non-linguistic vocalizations (max 2.31/5), thereby highlighting a gap between semantic and expressive transfer. Their approach demonstrates high correlation with human judgments, offering a new paradigm for expressive S2ST evaluation.

Abstract

Speech-to-speech translation (S2ST) should preserve not only lexical meaning, but also expressive attributes: emotion, scenario style (e.g., news reporting vs. dramatic dialogue), and nonverbal vocalizations (NVs). Moreover, collecting cross-lingual target speech that is both translation-faithful and expressively aligned with the source is difficult at scale, making reference-based evaluation impractical. We introduce STEB (Speech-to-Speech Translation Expressiveness Benchmark), a 32.6-hour Chinese--English benchmark that evaluates both standard dimensions (translation fidelity, speaker similarity, duration alignment) and expressiveness dimensions (emotion, scenario style, NV preservation). For expressiveness evaluation, STEB uses a caption-then-summarize framework that converts speech into structured expressive attributes and compares source and hypothesis attributes with an LLM judge. Human validation shows statistically significant correlations with listener judgments across all expressive dimensions. We evaluate six S2ST systems covering cascaded systems, end-to-end models, and speech large language models. Many systems, especially cascaded ones, achieve strong translation fidelity, but they still struggle with emotion preservation (best: 3.82/5) and NV preservation (best: 2.31/5). These results reveal a gap between semantic transfer and expressive transfer, identifying expressiveness preservation as an open challenge for S2ST. Audio samples are available at https://cmots.github.io/steb.github.io/.

Problem

• Speech-to-Speech Translation
• Expressiveness
• Emotion Preservation
• Nonverbal Vocalizations
• Scenario Style

Innovation

• expressiveness evaluation
• speech-to-speech translation
• reference-free benchmarking
• LLM-based assessment
• nonverbal vocalizations