A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation

2026-06-12

AI Summary

Existing simultaneous speech-to-speech translation (SimulS2ST) systems for long-form input lack practical and reproducible automatic evaluation methods, and the conventional end-to-end assumption often fails to hold. This work proposes the first fully automatic, human-annotation-free evaluation framework that recovers target speech timestamps using ASR and forced alignment, then leverages a sentence embedding aligner to match source utterances with their translations. This enables computation of sentence-level latency and quality metrics—such as YAAL and xCOMET—and their aggregation at the system level. Experiments on representative SimulS2ST systems validate the effectiveness of the proposed approach and reveal a significant issue of latency accumulation in current models when processing long audio sequences.

Abstract

Simultaneous speech-to-speech translation (SimulS2ST) enables real-time cross-lingual communication, but existing evaluation has focused largely on short or pre-segmented speech rather than long-form, continuous input. Prior approaches are difficult to reproduce and make assumptions that do not hold for end-to-end systems. We present a practical evaluation method for long-form SimulS2ST. Given source speech, pre-segmented source transcripts, and reference translations, we run automatic speech recognition (ASR) and forced alignment on the generated target speech to recover token-level timestamps, then apply a sentence-embedding-based aligner to match the target text to its corresponding source sentences. This enables sentence-level computation of latency and quality metrics, including YAAL and xCOMET, which are then aggregated into final system-level scores. Experiments on representative SimulS2ST systems show that the method is effective in practice and reveal that current systems suffer from substantial latency accumulation on long speech.

Problem

• Simultaneous speech-to-speech translation
• long-form evaluation
• latency accumulation
• real-time translation
• evaluation methodology

Innovation

• SimulS2ST
• long-form evaluation
• forced alignment
• sentence-level metrics
• latency accumulation