Preserving Speech-to-Text LLM Capabilities in Speech-to-Speech Generation

2026-06-29

AI Summary

This work proposes PRIME-Speech, a framework designed to overcome the degradation of speech-to-text (S2T) performance or the introduction of serial bottlenecks when endowing large speech models with speech-to-speech (S2S) capabilities. Operating under a frozen backbone, PRIME-Speech employs a causal audio post-decoder that synchronizes with intermediate hidden states of the backbone to directly generate speech codec tokens from the inference trajectory, bypassing the need for complete textual output. The approach innovatively integrates multi-source conditioning, adopts turn-level KV cache packing with position resetting, and supports parallel multi-token prediction, enabling efficient, low-latency spoken interaction across multiple turns. Experimental results demonstrate that PRIME-Speech preserves original S2T accuracy while significantly reducing first-audio latency and producing high-fidelity spoken responses with low word error rates (WER).

Abstract

Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can degrade the original S2T performance, while attaching a downstream talker reintroduces a serial text-to-speech bottleneck. We present PRIME-Speech, a frozen-backbone S2S conversion framework that trains only speech-generation modules. PRIME-Speech synchronizes a causal audio post-decoder with intermediate hidden states of the frozen backbone, so codec tokens are generated from the model's evolving reasoning trajectory rather than from completed text chunks. The post-decoder uses mixed hidden-state, text, and audio-history conditioning, and a training-time packing strategy with turn-level audio KV-cache and position reset stabilizes multi-turn spoken interaction without additional multi-turn S2S training data. Multi-token prediction further reduces the effective codec prediction rate and improves first-audio latency without modifying the reasoning path. Across speech translation, spoken QA, speech understanding, and multi-turn dialogue, PRIME-Speech preserves the S2T behavior of the frozen backbone while producing accurate, low-WER spoken responses.

Problem

• speech-to-speech generation
• speech-to-text LLM
• backbone degradation
• text-to-speech bottleneck
• spoken response generation

Innovation

• frozen-backbone
• speech-to-speech generation
• intermediate hidden-state synchronization
• multi-turn spoken interaction
• multi-token prediction