High-Fidelity Simultaneous Speech-To-Speech Translation

2025-02-05

AI Summary

This work addresses the dual challenges of synchronous source/target speech modeling and adaptive latency control in real-time speech-to-speech translation. We propose the first end-to-end speech-to-speech joint generation framework, built upon a decoder-only multi-stream language model that jointly encodes text and HuBERT+VQVAE-derived speech tokens to enable simultaneous listening and translating. We introduce a novel weakly supervised, word-level latency modeling approach: alignment-free synthetic data is constructed using translation perplexity, enabling dynamic latency optimization without explicit alignment or complex scheduling. Additionally, temperature-based sampling is incorporated to enhance speech naturalness and speaker fidelity. Evaluated on French–English real-time speech translation, our method achieves state-of-the-art performance—significantly improving BLEU scores, delivering superior speech quality, and supporting both batched inference and low-latency edge deployment.

Technology Category

• Natural Language Processing: Speech
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We introduce Hibiki, a decoder-only model for simultaneous speech translation. Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation. We furthermore address the fundamental challenge of simultaneous interpretation, which unlike its consecutive counterpart, where one waits for the end of the source utterance to start translating, adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk. To do so, we introduce a weakly-supervised method that leverages the perplexity of an off-the-shelf text translation system to identify optimal delays on a per-word basis and create aligned synthetic data. After supervised training, Hibiki performs adaptive, simultaneous speech translation with vanilla temperature sampling. On a French-English simultaneous speech translation task, Hibiki demonstrates state-of-the-art performance in translation quality, speaker fidelity and naturalness. Moreover, the simplicity of its inference process makes it compatible with batched translation and even real-time on-device deployment. We provide examples as well as models and inference code.

Problem

• Simultaneous speech-to-speech translation
• Real-time context accumulation
• Optimal delay identification

Innovation

• Decoder-only model for translation
• Multistream language model processing
• Weakly-supervised optimal delay identification