FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations

2025-09-08

AI Summary

Current full-duplex voice interaction systems face a fundamental trade-off: end-to-end approaches lack fine-grained controllability, while modular pipelines rely heavily on proprietary components, hindering holistic optimization. This paper proposes a modular, pluggable full-duplex framework supporting both conventional cascaded and novel semi-cascaded architectures—preserving paralinguistic and affective cues while reducing latency and error propagation. Key contributions include: (1) a streaming personalized Voice Activity Detection (pVAD) module synergized with semantic endpoint detection for precise, context-aware interruption control; and (2) a modular dialogue manager integrated with tool invocation and dynamic context management. Experiments demonstrate substantial reductions in false interruptions, improved endpoint detection accuracy, and end-to-end latency meeting industrial requirements. The system achieves a balanced optimization across naturalness, control precision, and real-time responsiveness.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Full-duplex voice interaction allows users and agents to speak simultaneously with controllable barge-in, enabling lifelike assistants and customer service. Existing solutions are either end-to-end, difficult to design and hard to control, or modular pipelines governed by turn-taking controllers that ease upgrades and per-module optimization; however, prior modular frameworks depend on non-open components and external providers, limiting holistic optimization. In this work, we present a complete, practical full-duplex voice interaction system comprising a turn-taking controller, an interaction module, and a dialogue manager. The controller integrates streaming personalized VAD (pVAD) to suppress false barge-ins from noise and non-primary speakers, precisely timestamp primary-speaker segments, and explicitly enable primary-speaker barge-ins; a semantic end-of-turn detector improves stop decisions. It upgrades heterogeneous half-duplex pipelines, cascaded, semi-cascaded, and speech-to-speech, to full duplex. Using internal models, we implement cascaded and semi-cascaded variants; the semi-cascaded one captures emotional and paralinguistic cues, yields more coherent responses, lowers latency and error propagation, and improves robustness. A dialogue manager extends capabilities via tool invocation and context management. We also propose three system-level metrics, barge-in, end-of-turn detection accuracy, and end-to-end latency, to assess naturalness, control accuracy, and efficiency. Experiments show fewer false interruptions, more accurate semantic ends, and lower latency approaching industrial systems, enabling robust, natural, real-time full-duplex interaction. Demos: https://fireredteam.github.io/demos/firered_chat.

Problem

• Enabling simultaneous speech with controllable barge-in for lifelike voice assistants
• Overcoming limitations of non-open modular components in full-duplex systems
• Reducing false interruptions and latency in real-time voice interactions

Innovation

• Full-duplex voice interaction with pluggable modular design
• Streaming personalized VAD for precise barge-in control
• Semi-cascaded architecture reducing latency and error propagation