MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models

2026-04-14

AI Summary

This work addresses the challenge of insufficient factual accuracy in full-duplex spoken language models while preserving real-time interactivity. To this end, the authors propose an asynchronous knowledge retrieval framework that dynamically injects external knowledge during natural pauses in user speech and the temporal gap between the model’s response onset and its core content generation. This plug-and-play mechanism enhances factual correctness without requiring model retraining. Through a compact full-duplex speech interface and a modular architecture, the system maintains a natural conversational experience while achieving factual accuracy on par with state-of-the-art non-full-duplex spoken language models and significantly outperforming baseline approaches on cross-domain mathematical reasoning tasks.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an open challenge. While scaling the model size could address this gap, it would make real-time inference prohibitively expensive. In this work, we propose MoshiRAG, a modular approach that combines a compact full-duplex interface with selective retrieval to access more powerful knowledge sources. Our asynchronous framework enables the model to identify knowledge-demanding queries and ground its responses in external information. By leveraging the natural temporal gap between response onset and the delivery of core information, the retrieval process can be completed while maintaining a natural conversation flow. With this approach, MoshiRAG achieves factuality comparable to the best publicly released non-duplex speech language models while preserving the interactivity inherent to full-duplex systems. Moreover, our flexible design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-domain mathematical reasoning tasks.

Problem

• full-duplex
• speech language models
• factuality
• knowledge retrieval
• real-time interaction

Innovation

• asynchronous retrieval
• full-duplex speech language model
• modular RAG
• real-time interactivity
• knowledge grounding