Multilingual Extraction and Recognition of Implicit Discourse Relations in Speech and Text

2026-02-04

AI Summary

This study addresses the challenge of implicit discourse relation classification, which requires inferring semantic connections from context—a task hindered by the limitations of text-only approaches in capturing cross-lingual and cross-modal cues. To this end, the authors present the first multilingual multimodal dataset for implicit discourse relations, covering English, French, and Spanish, and propose a multimodal method that integrates textual and acoustic features. Leveraging the Qwen2-Audio model for joint audio-text modeling, the approach enables effective cross-lingual transfer. Experimental results demonstrate that the proposed multimodal fusion significantly outperforms unimodal baselines relying solely on text or audio, with particularly pronounced gains observed for low-resource languages.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Implicit discourse relation classification is a challenging task, as it requires inferring meaning from context. While contextual cues can be distributed across modalities and vary across languages, they are not always captured by text alone. To address this, we introduce an automatic method for distantly related and unrelated language pairs to construct a multilingual and multimodal dataset for implicit discourse relations in English, French, and Spanish. For classification, we propose a multimodal approach that integrates textual and acoustic information through Qwen2-Audio, allowing joint modeling of text and audio for implicit discourse relation classification across languages. We find that while text-based models outperform audio-based models, integrating both modalities can enhance performance, and cross-lingual transfer can provide substantial improvements for low-resource languages.

Problem

• implicit discourse relations
• multilingual
• multimodal
• cross-lingual transfer
• low-resource languages

Innovation

• multimodal learning
• implicit discourse relations
• cross-lingual transfer
• Qwen2-Audio
• low-resource languages