BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

2026-06-12

AI Summary

This work proposes a native full-duplex spoken dialogue approach that eliminates reliance on external voice activity detection modules for turn-taking, thereby enabling truly real-time interactive speech. By introducing a small set of special control tokens into the standard vocabulary, a single autoregressive large language model is empowered to autonomously decide when to listen, speak, or pause, without any auxiliary turn-control components. Built upon GLM-4-Voice and fine-tuned with lightweight direct preference optimization (DPO) using only 400,000 full-duplex training samples, the method achieves a 92% turn-switching success rate and 100% interruption success rate on InstructS2S-Eval, while improving the speech response rating from 2.17 to 3.39 and maintaining or surpassing baseline performance across multiple question-answering benchmarks.

Abstract

Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

Problem

• full-duplex speech
• speech language models
• turn-taking
• Voice Activity Detection
• real-time interaction

Innovation

• full-duplex speech
• autoregressive LLM
• native turn-taking
• speech language model
• barge-in handling