MAD: A Benchmark for Multi-Turn Audio Dialogue Fact-Checking

2025-08-16

AI Summary

Existing fact-checking research primarily focuses on isolated textual or spoken utterances, failing to address the complexities of multi-turn spoken dialogues—such as speaker turn-taking, speech overlap, prosodic variation, and dynamic information propagation. To bridge this gap, we introduce MAD, the first benchmark dataset for fact-checking real-world multi-turn audio dialogues. MAD jointly models acoustic signals, dialogue structure, and information diffusion patterns, providing sentence-level and dialogue-level veracity labels, check-worthiness scores, and multimodal scenario annotations. It is the first effort to systematically formalize reasoning challenges inherent in spoken interaction, thereby filling a critical data void in speech-dialogue joint fact-checking. Extensive evaluation reveals that strong baseline models achieve only 72–74% accuracy on sentence-level verification and 71–72% on dialogue-level verification—substantially lower than corresponding text-based tasks—confirming the task’s heightened difficulty and exposing fundamental limitations of current multimodal understanding approaches.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Despite the growing popularity of audio platforms, fact-checking spoken content remains significantly underdeveloped. Misinformation in speech often unfolds across multi-turn dialogues, shaped by speaker interactions, disfluencies, overlapping speech, and emotional tone-factors that complicate both claim detection and verification. Existing datasets fall short by focusing on isolated sentences or text transcripts, without modeling the conversational and acoustic complexity of spoken misinformation. We introduce MAD (Multi-turn Audio Dialogues), the first fact-checking dataset aligned with multi-turn spoken dialogues and corresponding audio. MAD captures how misinformation is introduced, contested, and reinforced through natural conversation. Each dialogue includes annotations for speaker turns, dialogue scenarios, information spread styles, sentence-level check-worthiness, and both sentence- and dialogue-level veracity. The dataset supports two core tasks: check-worthy claim detection and claim verification. Benchmarking shows that even strong pretrained models reach only 72-74% accuracy at the sentence level and 71-72% at the dialogue level in claim verification, underscoring MAD's difficulty. MAD offers a high-quality benchmark for advancing multimodal and conversational fact-checking, while also surfacing open challenges related to reasoning over speech and dialogue dynamics.

Problem

• Detecting misinformation in multi-turn audio dialogues
• Verifying claims with conversational and acoustic complexity
• Addressing gaps in existing text-only fact-checking datasets

Innovation

• First fact-checking dataset for multi-turn audio dialogues
• Includes speaker turns, dialogue scenarios, and veracity annotations
• Supports check-worthy claim detection and claim verification