Large Speech Model Enabled Semantic Communication

2025-12-04

AI Summary

Existing semantic speech communication systems are constrained by task-specific model architectures, struggling to simultaneously achieve high compression efficiency, acceptable speech quality, and low latency under low-bandwidth, high-packet-loss channel conditions. This paper proposes a large speech model (Moshi)-based semantic communication framework. First, the Mimi codec is employed to generate discrete speech tokens. Second, a content- and channel-aware adaptive controller dynamically adjusts transmission bit rate and redundancy. Third, an in-band unequal error protection mechanism is introduced, coupled with LoRA fine-tuning to enable generative recovery of lost tokens. The system supports variable bit rates from 550 bps to 2.06 kbps, significantly outperforms conventional methods in speech quality under high packet loss, and achieves an end-to-end latency of approximately 460 ms—demonstrating feasibility for real-time deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent Communication
• Natural Language Processing: Speech

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Large language models for search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Existing speech semantic communication systems mainly based on Joint Source-Channel Coding (JSCC) architectures have demonstrated impressive performance, but their effectiveness remains limited by model structures specifically designed for particular tasks and datasets. Recent advances indicate that generative large models pre-trained on massive datasets, can achieve outstanding performance arexhibit exceptional performance across diverse downstream tasks with minimal fine-tuning. To exploit the rich semantic knowledge embedded in large models and enable adaptive transmission over lossy channels, we propose a Large Speech Model enabled Semantic Communication (LargeSC) system. Simultaneously achieving adaptive compression and robust transmission over lossy channels remains challenging, requiring trade-offs among compression efficiency, speech quality, and latency. In this work, we employ the Mimi as a speech codec, converting speech into discrete tokens compatible with existing network architectures. We propose an adaptive controller module that enables adaptive transmission and in-band Unequal Error Protection (UEP), dynamically adjusting to both speech content and packet loss probability under bandwidth constraints. Additionally, we employ Low-Rank Adaptation (LoRA) to finetune the Moshi foundation model for generative recovery of lost speech tokens. Simulation results show that the proposed system supports bandwidths ranging from 550 bps to 2.06 kbps, outperforms conventional baselines in speech quality under high packet loss rates and achieves an end-to-end latency of approximately 460 ms, thereby demonstrating its potential for real-time deployment.

Problem

• Enables adaptive semantic speech transmission over lossy channels
• Achieves robust compression and recovery under bandwidth and packet loss constraints
• Reduces latency for real-time deployment while maintaining speech quality

Innovation

• Uses Mimi speech codec for discrete token conversion
• Employs adaptive controller with Unequal Error Protection
• Applies Low-Rank Adaptation to fine-tune Moshi model