Generative Location Modeling for Spatially Aware Object Insertion

2024-10-17

arXiv.org

AI Summary

This work addresses the lack of spatial awareness in image editing tasks involving object insertion. To achieve geometrically plausible and semantically aligned placement, we propose a novel position modeling framework. Our method introduces: (1) the first autoregressive bounding-box generation framework conditioned jointly on input images and category labels, directly predicting object coordinates; (2) direct preference optimization (DPO) to end-to-end learn spatial plausibility from sparse human annotations and counterfactual negative samples; and (3) a background-aware bounding-box generation mechanism tightly integrated with an inpainting module. Evaluated on object insertion, our approach significantly outperforms instruction-tuned models and state-of-the-art localization baselines. It achieves superior accuracy, high visual fidelity, and strong spatial consistency—ensuring inserted objects respect scene geometry, occlusion relationships, and contextual semantics.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Generative models have become a powerful tool for image editing tasks, including object insertion. However, these methods often lack spatial awareness, generating objects with unrealistic locations and scales, or unintentionally altering the scene background. A key challenge lies in maintaining visual coherence, which requires both a geometrically suitable object location and a high-quality image edit. In this paper, we focus on the former, creating a location model dedicated to identifying realistic object locations. Specifically, we train an autoregressive model that generates bounding box coordinates, conditioned on the background image and the desired object class. This formulation allows to effectively handle sparse placement annotations and to incorporate implausible locations into a preference dataset by performing direct preference optimization. Our extensive experiments demonstrate that our generative location model, when paired with an inpainting method, substantially outperforms state-of-the-art instruction-tuned models and location modeling baselines in object insertion tasks, delivering accurate and visually coherent results.

Problem

• Predicting plausible object placement locations in images
• Addressing one-to-many nature of feasible object placements
• Overcoming sparsity of labeled data in location modeling

Innovation

• Autoregressive transformer decodes bounding box coordinates
• Tokenizes image and object class for location prediction
• Uses Direct Preference Optimization to refine spatial predictions