S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modeling

2026-08-14

AI Summary

This study addresses the challenge in existing multimodal dialogue retrieval of simultaneously maintaining global semantic coherence and stylistic consistency. To overcome this limitation, we propose s2dialog, a unified framework that incorporates a dialogue-level text-acoustic encoder. Through multimodal contrastive learning, this approach achieves joint alignment of semantics and style for retrieval, marking the first such integration in this domain. Experimental evaluations on the DailyTalk dataset demonstrate that s2dialog significantly outperforms current baselines, delivering superior multimodal dialogue retrieval performance. Consequently, this work effectively bridges a critical research gap in dialogue-level cross-modal joint modeling of semantics and style, offering a robust solution for holistic multimodal understanding in conversational systems.

Abstract

Multimodal dialogue retrieval aims to retrieve dialogues from multimodal dialogue banks that are similar to a target dialogue in terms of both textual semantics and acoustic conversational styles. Such dialogue-level retrieval is crucial for many dialogue-related tasks, including Emotion Recognition in Conversation, Spoken Dialogue Systems, and Conversational Speech Synthesis, where external dialogue examples can provide valuable semantic and stylistic references. However, existing retrieval methods are still largely limited to utterance-level or unimodal matching, and often fail to capture the global semantic coherence and stylistic consistency of an entire dialogue. To address this gap, we propose S2Dialog, a unified framework for dialogue-level semantic-style retrieval from multimodal dialogue banks. Specifically, S2Dialog consists of a Dialogue-level Textual Retriever and a Dialogue-level Acoustic Retriever, which encode the textual and acoustic modalities of a dialogue into dialogue-level representations, respectively. To further enhance multimodal retrieval, we introduce Dialogue-level Textual-Acoustic Contrastive Learning, which aligns semantically and stylistically similar dialogues while distinguishing unrelated ones. Extensive experiments on the multimodal dialogue dataset DailyTalk demonstrate that S2Dialog achieves outstanding retrieval performance.

Problem

• Multimodal Dialogue Retrieval
• Dialogue-level Matching
• Semantic Coherence
• Acoustic Style Consistency

Innovation

• Dialogue-level Retrieval
• Multimodal Contrastive Learning
• Acoustic-Style Modeling
• Semantic-Style Alignment