Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection

2025-01-24

AI Summary

Generative AI (GenAI) pollutes web-sourced multimodal evidence—e.g., mismatched image-text pairs—undermining the reliability of rumor detection systems that rely on such evidence. Existing approaches predominantly operate at the claim level and neglect evidence-level contamination. Method: This work pioneers evidence-layer contamination modeling for multimodal rumor detection. We systematically characterize and quantify GenAI-induced corruption in cross-modal evidence retrieval. To mitigate this, we propose a dual-strategy framework: (1) cross-modal evidence re-ranking to filter contaminated candidates, and (2) claim-evidence collaborative reasoning to jointly assess credibility. Our method integrates multimodal alignment, retrieval re-ranking, contrastive learning–driven evidence trustworthiness scoring, and a unified joint-reasoning architecture. Results: On two benchmark datasets, our approach improves out-of-context (OOC) evidence detection accuracy by over 9 percentage points, significantly enhancing model robustness and generalization against GenAI-forged evidence.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

While large generative artificial intelligence (GenAI) models have achieved significant success, they also raise growing concerns about online information security due to their potential misuse for generating deceptive content. Out-of-context (OOC) multimodal misinformation detection, which often retrieves Web evidence to identify the repurposing of images in false contexts, faces the issue of reasoning over GenAI-polluted evidence to derive accurate predictions. Existing works simulate GenAI-powered pollution at the claim level with stylistic rewriting to conceal linguistic cues, and ignore evidence-level pollution for such information-seeking applications. In this work, we investigate how polluted evidence affects the performance of existing OOC detectors, revealing a performance degradation of more than 9 percentage points. We propose two strategies, cross-modal evidence reranking and cross-modal claim-evidence reasoning, to address the challenges posed by polluted evidence. Extensive experiments on two benchmark datasets show that these strategies can effectively enhance the robustness of existing out-of-context detectors amidst polluted evidence.

Problem

• Generative AI
• Evidence Tampering
• Fake News Detection

Innovation

• Generative AI Forensics
• Multi-modal Analysis
• Evidence Rearrangement