Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning

2026-09-11

AI Summary

为解决语音到语音翻译中高比特率语音令牌预测难题及对理想训练数据的依赖，提出使用低比特率令牌与Autowave-X解码器的方法，并基于此构建了名为Kraken的模型。

Abstract

Speech-to-speech translation (S2ST) has advanced significantly with speech LLMs, offering the potential for joint optimization and preserving non-linguistic information. However, these models struggle with predicting high-bitrate speech tokens in LLMs, and face the challenge of relying on S2ST training data with ideally aligned speaker identity and prosody. We propose using low-bitrate tokens based on single-layer vector quantization, trained to reconstruct self-supervised learning (SSL) features. We also employ a separate token-to-waveform decoder named Autowave-X, which is also conditioned on the source speech to improve non-linguistic transfer, thereby relaxing the training data constraints. With the integration of these techniques, we propose an S2ST model named Kraken, which augments a pre-trained LLM with speech feature inputs and the low-bitrate token outputs, followed by Autowave-X vocoder. We built the model upon Qwen3-8B and trained it using 150k hours of multilingual and multitask speech data. We demonstrated that our model exhibited better translation quality than SeamlessM4T-Large v2 and Qwen2.5-Omni, along with improved speaker and prosody transfer capabilities.

Problem

• Speech-to-speech translation
• high-bitrate speech tokens
• speaker identity
• prosody

Innovation

• low-bitrate tokens
• vector quantization
• Autowave-X
• non-linguistic transfer
• multilingual and multitask speech data