Problem
Research questions and friction points this paper is trying to address.
Cross-domain Information
Fake News Detection
Multimodal Content
Can Out-of-Domain data help to Learn Domain-Specific Prompts for Multimodal Misinformation Detection?
🤖 AI Summary
To address the weak cross-domain generalization capability in multimodal fake news detection, this paper proposes DPOD—a novel framework that introduces out-of-domain data into domain-specific prompt learning for joint optimization across multiple target domains. It designs a label-aware CLIP feature alignment mechanism to enhance semantic consistency between images and text, and introduces a dynamic domain-relevance weighting strategy for prompt generation to adaptively model inter-domain discrepancies. The framework integrates an enhanced CLIP backbone, label-aware contrastive learning, and a multi-source domain joint training paradigm. Evaluated on NewsCLIPpings and VERITE benchmarks, DPOD achieves state-of-the-art performance, with significant improvements in accuracy for detecting fake image-text pairs across diverse domains—including politics, sports, and others.
📝 Abstract
Spread of fake news using out-of-context images and captions has become widespread in this era of information overload. Since fake news can belong to different domains like politics, sports, etc. with their unique characteristics, inference on a test image-caption pair is contingent on how well the model has been trained on similar data. Since training individual models for each domain is not practical, we propose a novel framework termed DPOD (Domain-specific Prompt tuning using Out-of-domain data), which can exploit out-of-domain data during training to improve fake news detection of all desired domains simultaneously. First, to compute generalizable features, we modify the Vision-Language Model, CLIP to extract features that helps to align the representations of the images and corresponding captions of both the in-domain and out-of-domain data in a label-aware manner. Further, we propose a domain-specific prompt learning technique which leverages training samples of all the available domains based on the extent they can be useful to the desired domain. Extensive experiments on the large-scale NewsCLIPpings and VERITE benchmarks demonstrate that DPOD achieves state of-the-art performance for this challenging task. Code: https://github.com/scviab/DPOD.
Innovation
Methods, ideas, or system contributions that make the work stand out.
DPOD
Multi-topic Information
Enhanced CLIP Model
🔎 Similar Papers
No similar papers found.
