Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks

2025-11-08

AI Summary

Existing multimodal misinformation detection benchmarks suffer from pervasive modality bias, enabling models to achieve high accuracy using only a single modality—thereby undermining generalization. Method: This paper introduces the first automated method for identifying sample-level modality bias, establishing a three-tiered quantification framework: coarse-grained (whole-sample), medium-grained (modality combinations), and fine-grained (local regions/tokens). The framework integrates modality-wise gain estimation, information flow quantification, and causal analysis, enhanced by multi-perspective ensemble to improve robustness—validated through human evaluation. Contribution/Results: Experiments on two mainstream benchmarks demonstrate that our method precisely localizes biased samples, uncovers root causes of detector performance instability, and reveals systematic discrepancies across granularity levels: balanced samples exhibit consistent attributions, whereas biased ones show divergent modality importance patterns. This work establishes a novel paradigm for trustworthy multimodal misinformation detection.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Numerous multimodal misinformation benchmarks exhibit bias toward specific modalities, allowing detectors to make predictions based solely on one modality. While previous research has quantified bias at the dataset level or manually identified spurious correlations between modalities and labels, these approaches lack meaningful insights at the sample level and struggle to scale to the vast amount of online information. In this paper, we investigate the design for automated recognition of modality bias at the sample level. Specifically, we propose three bias quantification methods based on theories/views of different levels of granularity: 1) a coarse-grained evaluation of modality benefit; 2) a medium-grained quantification of information flow; and 3) a fine-grained causality analysis. To verify the effectiveness, we conduct a human evaluation on two popular benchmarks. Experimental results reveal three interesting findings that provide potential direction toward future research: 1)~Ensembling multiple views is crucial for reliable automated analysis; 2)~Automated analysis is prone to detector-induced fluctuations; and 3)~Different views produce a higher agreement on modality-balanced samples but diverge on biased ones.

Problem

• Automated detection of modality bias in misinformation benchmarks
• Sample-level analysis overcoming dataset-level limitations
• Quantifying bias through multi-granularity evaluation methods

Innovation

• Automated sample-level modality bias recognition
• Three granularity-based bias quantification methods
• Ensembling multiple views for reliable analysis