Does the Manipulation Process Matter? RITA: Reasoning Composite Image Manipulations via Reversely-Ordered Incremental-Transition Autoregression

2025-09-24

AI Summary

Existing image manipulation localization (IML) methods employ single-step binary prediction, neglecting the inherent temporal and hierarchical nature of manipulation operations—leading to dimensional collapse and misalignment with the task’s structural semantics. To address this, we reformulate IML as a conditional sequence prediction problem and introduce, for the first time, explicit modeling of the manipulation process. We propose RITA, a novel framework built upon a hierarchical sequence prediction network, integrating backward incremental autoregressive decoding and differentiable mask generation. Furthermore, we construct HSIM—the first benchmark dataset supporting multi-step editing modeling—and introduce the HSS evaluation metric suite. Our approach achieves state-of-the-art performance on conventional benchmarks and significantly improves fine-grained localization accuracy in complex, multi-stage manipulation scenarios, thereby overcoming fundamental limitations of the prevailing single-step localization paradigm.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Motion & Tracking

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Image manipulations often entail a complex manipulation process, comprising a series of editing operations to create a deceptive image, exhibiting sequentiality and hierarchical characteristics. However, existing IML methods remain manipulation-process-agnostic, directly producing localization masks in a one-shot prediction paradigm without modeling the underlying editing steps. This one-shot paradigm compresses the high-dimensional compositional space into a single binary mask, inducing severe dimensional collapse, thereby creating a fundamental mismatch with the intrinsic nature of the IML task. To address this, we are the first to reformulate image manipulation localization as a conditional sequence prediction task, proposing the RITA framework. RITA predicts manipulated regions layer-by-layer in an ordered manner, using each step's prediction as the condition for the next, thereby explicitly modeling temporal dependencies and hierarchical structures among editing operations. To enable training and evaluation, we synthesize multi-step manipulation data and construct a new benchmark HSIM. We further propose the HSS metric to assess sequential order and hierarchical alignment. Extensive experiments show RITA achieves SOTA on traditional benchmarks and provides a solid foundation for the novel hierarchical localization task, validating its potential as a general and effective paradigm. The code and dataset will be publicly available.

Problem

• Existing image manipulation localization methods ignore the sequential editing process
• One-shot prediction paradigm causes dimensional collapse in manipulation detection
• Need to model temporal dependencies among hierarchical editing operations

Innovation

• Reformulates localization as sequential prediction task
• Uses reverse-ordered incremental autoregressive modeling approach
• Predicts manipulated regions layer-by-layer with conditional dependencies