EviSI: An Evaluation Agent for Simultaneous Interpreting

2026-09-08

AI Summary

为解决同步口译中语义保真度评估问题，提出EviSI模型，采用MQM的质量评估原则，提高了英汉互译系统的评分一致性。

Abstract

Simultaneous speech-to-speech translation requires understanding, translation and spoken delivery while the source stream continues. To support timely delivery and limit accumulated delay, systems adopt reformulation and summarization, which can preserve meaning while departing from written references. BLEU and COMET may not reliably distinguish such variation from semantic loss. We introduce EviSI, a large language model evaluation agent adapting the error analysis and penalty principles of Multidimensional Quality Metrics (MQM). It constructs shared source evidence, assesses semantic fidelity and oral expression, reconciles overlapping errors and scores deterministically. EviSI recovers the aggregate human system ranking for English to Chinese. Mean Kendall agreement with human system rankings within corpora reaches 0.707 for English to Chinese and 0.467 for Chinese to English, exceeding evaluated baselines. An extension across five directions shows positive concordance with COMET without human ratings. Individual output agreement with humans remains mixed.

Problem

• Simultaneous Interpreting
• BLEU
• COMET
• Semantic Fidelity

Innovation

• EviSI
• Simultaneous Interpreting
• Multidimensional Quality Metrics (MQM)
• Semantic Fidelity
• Oral Expression