SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translation

2026-03-11

AI Summary

This work proposes SimulU, a novel simultaneous speech-to-speech translation strategy that operates without any additional training and effectively handles long-form audio—a longstanding challenge for existing approaches, which typically rely on extensive task-specific training and are limited to short utterances. SimulU leverages cross-attention mechanisms within pretrained end-to-end models to dynamically regulate the use of input history and output generation through a history management and speech output selection scheme. For the first time, it enables high-quality simultaneous speech-to-speech translation on long sequences without fine-tuning. Evaluated on the MuST-C benchmark across eight languages, SimulU matches or surpasses strong cascaded baselines in the trade-off between translation quality and latency, thereby eliminating the dependency on specialized training and short-segment constraints.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Simultaneous speech-to-speech translation (SimulS2S) is essential for real-time multilingual communication, with increasing integration into meeting and streaming platforms. Despite this, SimulS2S remains underexplored in research, where current solutions often rely on resource-intensive training procedures and operate on short-form, pre-segmented utterances, failing to generalize to continuous speech. To bridge this gap, we propose SimulU, the first training-free policy for long-form SimulS2S. SimulU adopts history management and speech output selection strategies that exploit cross-attention in pre-trained end-to-end models to regulate both input history and output generation. Evaluations on MuST-C across 8 languages show that SimulU achieves a better or comparable quality-latency trade-off against strong cascaded models. By eliminating the need for ad-hoc training, SimulU offers a promising path to end-to-end SimulS2S in realistic, long-form scenarios.

Problem

• Simultaneous speech-to-speech translation
• long-form speech
• real-time multilingual communication
• continuous speech
• training-intensive methods

Innovation

• training-free
• simultaneous speech-to-speech translation
• long-form
• cross-attention
• history management