DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects

2026-08-08

AI Summary

This work addresses the challenges of data scarcity and misalignment between semantic representations and fixed acoustic supervision in end-to-end spoken dialogue modeling for low-resource Chinese dialects. To overcome these issues, the authors construct a scalable data pipeline for dialectal spoken dialogue synthesis and propose a two-stage post-training strategy incorporating a self-aligned speech supervision mechanism that dynamically aligns acoustic targets with the model’s evolving semantic representations. This approach achieves the first successful end-to-end spoken dialogue modeling for low-resource Chinese dialects, significantly outperforming existing baselines across multiple dialects. Substantial improvements are observed in dialect consistency, response quality, and speech intelligibility. The complete framework is open-sourced to facilitate future research in this underexplored domain.

Abstract

Current end-to-end speech dialogue models are primarily optimized for mainstream languages and remain limited in low-resource dialect scenarios due to the scarcity of dialect speech data. Moreover, during dialect adaptation, the semantic representation space of speech dialogue models continuously evolves, while conventional speech supervision remains unchanged, leading to semantic inconsistency between hidden representations and speech targets and degrading speech stability and naturalness. To address these issues, we propose DialectS2S, an end-to-end speech dialogue model for Chinese dialects. We first develop a scalable dialect speech dialogue synthesis pipeline for efficient data construction. We further introduce a two-stage post-training strategy with self-aligned speech supervision, which aligns the semantic content of speech supervision with the evolved semantic representations of the model to improve dialect speech generation quality. Experimental results show that DialectS2S consistently outperforms existing baselines across multiple Chinese dialects in speech dialogue, achieving substantial improvements in dialect consistency, response quality, and speech intelligibility. Our work provides an efficient and scalable solution for end-to-end speech dialogue modeling in low-resource dialect scenarios. To facilitate future research and practical applications, we fully open-source the DialectS2S framework, including model checkpoints, training datasets, and fine-tuning code.

Problem

• low-resource dialects
• end-to-end speech dialogue
• semantic inconsistency
• speech stability
• Chinese dialects

Innovation

• end-to-end speech dialogue
• low-resource dialects
• self-aligned speech supervision
• semantic consistency
• scalable data pipeline