Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

2026-07-07

AI Summary

This work addresses a critical challenge in full-duplex spoken language modeling: the sharing of deep-layer parameters between acoustic and semantic modalities often induces gradient conflict, leading to knowledge degradation and compromised semantic integrity. The study is the first to uncover the underlying mechanism of this cross-modal interference and introduces Lychee-FD, a novel framework that decouples modalities through hierarchical parameter separation in deep layers while preserving cross-modal consistency via a dedicated semantic alignment channel. This enables native end-to-end full-duplex modeling without sacrificing coherence. Experimental results demonstrate that Lychee-FD achieves state-of-the-art performance across multiple benchmarks, yielding a 7.4% absolute improvement in spoken question-answering accuracy and a 28.5% gain in interaction fluency, all while maintaining efficient inference.

Abstract

Developing seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge and long-standing goal for the speech and NLP community. Despite notable progress, recent endeavors are fundamentally constrained by severe modality interference, which causes substantial knowledge degradation and compromises semantic integrity -- ultimately making full-duplex SLMs feel unnatural and unintelligent. In this paper, through an exhaustive fine-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises from inherent gradient conflicts between acoustic and semantic modeling when the two modalities are forced to share a deep parameter space. Guided by this key insight, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic alignment channel. Extensive experiments on multiple full-duplex benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both speech intelligence (+7.4% on Spoken QA) and full-duplex interaction fluidity (+28.5% on FullDuplexBench 1.5) without compromising inference efficiency. To the best of our knowledge, this work is the first to achieve two key advances: 1) uncovering and elucidating the root cause of modality interference in full-duplex SLMs, and 2) designing an elegant hierarchical model together with a practical solution for seamless, high-performance, native intelligent full-duplex SLMs.

Problem

• full-duplex Spoken Language Models
• modality interference
• acoustic-semantic modeling
• semantic coherence
• parameter sharing

Innovation

• modality interference
• hierarchical parameter separation
• semantic alignment
• full-duplex SLMs
• gradient conflict