Disharmony: Forensics using Reverse Lighting Harmonization

2025-01-17

AI Summary

Detecting illumination-inconsistent forgery regions—particularly harmonized objects—in AI-generated or edited images remains highly challenging due to seamless lighting integration. Method: This paper proposes a novel reverse illumination harmonization forensics paradigm. It is the first to embed reverse illumination harmonization signals into a semantic segmentation network, leveraging a synthetically constructed multi-strategy illumination-harmonized dataset and contrastive feature modeling for fine-grained localization of forged regions. Contribution/Results: The method overcomes the fundamental limitation of conventional forensic models in detecting illumination-fused forgeries and fills a critical technical gap in harmonized-object detection. Evaluated across diverse editing scenarios—including virtual try-on—it consistently outperforms state-of-the-art forensic models, achieving new SOTA performance in both localization accuracy and robustness.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Content generation and manipulation approaches based on deep learning methods have seen significant advancements, leading to an increased need for techniques to detect whether an image has been generated or edited. Another area of research focuses on the insertion and harmonization of objects within images. In this study, we explore the potential of using harmonization data in conjunction with a segmentation model to enhance the detection of edited image regions. These edits can be either manually crafted or generated using deep learning methods. Our findings demonstrate that this approach can effectively identify such edits. Existing forensic models often overlook the detection of harmonized objects in relation to the background, but our proposed Disharmony Network addresses this gap. By utilizing an aggregated dataset of harmonization techniques, our model outperforms existing forensic networks in identifying harmonized objects integrated into their backgrounds, and shows potential for detecting various forms of edits, including virtual try-on tasks.

Problem

• Image Manipulation Detection
• Object-Background Fusion
• Accuracy Improvement

Innovation

• Discordant Network
• Image Manipulation Detection
• Object-Background Harmony Analysis