Rethinking VLMs for Image Forgery Detection and Localization

2026-03-13

AI Summary

Existing vision-language models (VLMs) suffer from semantic bias in image forgery detection and localization (IFDL), leading to degraded performance. This work is the first to identify this issue and introduces a novel IFDL-VLM paradigm that incorporates forgery location masks as training priors to steer the VLM toward assessing image authenticity rather than semantic plausibility. By doing so, it establishes an end-to-end framework for joint detection and localization. The proposed approach substantially improves detection accuracy, localization precision, and result interpretability, achieving state-of-the-art performance across nine mainstream benchmarks. Moreover, it demonstrates remarkable robustness in both in-domain and cross-dataset generalization scenarios.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

With the rapid rise of Artificial Intelligence Generated Content (AIGC), image manipulation has become increasingly accessible, posing significant challenges for image forgery detection and localization (IFDL). In this paper, we study how to fully leverage vision-language models (VLMs) to assist the IFDL task. In particular, we observe that priors from VLMs hardly benefit the detection and localization performance and even have negative effects due to their inherent biases toward semantic plausibility rather than authenticity. Additionally, the location masks explicitly encode the forgery concepts, which can serve as extra priors for VLMs to ease their training optimization, thus enhancing the interpretability of detection and localization results. Building on these findings, we propose a new IFDL pipeline named IFDL-VLM. To demonstrate the effectiveness of our method, we conduct experiments on 9 popular benchmarks and assess the model performance under both in-domain and cross-dataset generalization settings. The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.Code is available at: https://github.com/sha0fengGuo/IFDL-VLM.

Problem

• Image Forgery Detection
• Vision-Language Models
• Forgery Localization
• AIGC
• Semantic Plausibility

Innovation

• Vision-Language Models
• Image Forgery Detection
• Localization
• Forgery Mask Prior
• Interpretability