Do LLMs Need Architectural Changes for Simultaneous Speech Translation? A Prefix-to-Prefix Data Driven Approach

2026-07-14

AI Summary

This work addresses the challenges faced by large language models in simultaneous interpretation, where limited context length, cross-lingual word order mismatches, and ambiguous speech boundaries hinder the trade-off between latency and translation quality. The authors propose a data-driven approach that requires no architectural modifications: it employs cumulative streaming decoding over fixed-length speech chunks, incorporates a fallback mechanism to confirm translated prefixes, and leverages prefix-to-prefix (P2P) supervision signals for fine-tuning. By eschewing conventional explicit read/write policies, the method achieves substantially improved translation quality—yielding a 1.54-point gain in COMETKiwi score on an internal spoken dialogue benchmark—while maintaining low latency, with only a 0.15-second increase in average lag, thus demonstrating enhanced robustness and practicality.

Abstract

Simultaneous speech translation (SimulST) requires incremental translation under strict latency constraints, yet remains challenging for decoder-only LLM systems due to limited context and cross-lingual reordering. Recent approaches often introduce architectural changes or explicit read/write policies to control output timing, which can be brittle in conversational speech where segmentation boundaries are ambiguous. We present a simple data-driven alternative: fixed-length chunks for cumulative streaming decoding with a rewind-based committed prefix, and teacher-labeled prefix-to-prefix (P2P) targets with bounded waiting for fine-tuning, yielding CSSEL-P2P, where CSSEL is our proposed chunked streaming speech encoder LLM. In our in-house conversational speech evaluation, CSSEL-P2P improves streaming quality by +1.54 COMETKiwi over the CSSEL streaming baseline at comparable latency (+0.15s Average Lagging), suggesting effective SimulST without architectural changes via P2P supervision.

Problem

• Simultaneous Speech Translation
• Large Language Models
• Latency Constraints
• Cross-lingual Reordering
• Streaming Decoding

Innovation

• simultaneous speech translation
• prefix-to-prefix
• streaming decoding
• data-driven supervision
• large language models