BoxCtrl: 3D-Aware Visual Prompting for Geometric Image Editing

2026-06-22

AI Summary

Existing methods for 3D-aware image editing struggle to simultaneously achieve precise geometric control and high-fidelity appearance preservation. This work proposes BoxCtrl, a novel framework that introduces 3D bounding boxes defined by RGB-colored orthogonal planes as intuitive visual prompts to convey geometric intent directly on 2D images, effectively decoupling geometric transformation from appearance generation. To bridge the domain gap between synthetic and real data, the approach employs a two-stage training strategy comprising supervised fine-tuning (SFT) on synthetic data followed by reinforcement learning (RL) on real images. The method significantly improves geometric accuracy across translation, rotation, scaling, and composite editing tasks while maintaining high visual fidelity, establishing a new state of the art in 3D-guided image manipulation.

Abstract

As instruction-based editing models and multimodal large language models advance, diverse image editing tasks have become feasible. However, achieving precise and consistent geometric image editing, such as translating, scaling, and rotating in 3D space, remains a major challenge. In this work, we introduce BoxCtrl, a 3D-aware visual prompting framework. Unlike text-only or coarse 2D-guided approaches, our method introduces informative RGB 3D bounding boxes projected onto 2D images as visual prompts. The three orthogonal faces of each box are painted with distinct RGB colors, simultaneously encoding position, size, and orientation to provide a compact, intuitive in-context visual example. The key to BoxCtrl's success lies in these well-designed bounding boxes, which decouple geometric control from appearance control. This enables the model to learn consistent correspondences between faces of the same color in the latent space, leading to a precise understanding of geometric intentions and accurate editing results. We introduce a two-stage training paradigm: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL). To address paired data scarcity, we construct a large-scale synthetic dataset for SFT, equipping the model with fundamental editing capabilities. To bridge the synthetic-to-real domain gap, we incorporate an online RL stage leveraging unpaired real-world data. Guided by a reward function evaluating geometric accuracy and visual fidelity, our SFT-RL strategy significantly enhances geometric precision while maintaining photorealistic quality. Extensive experiments demonstrate that BoxCtrl achieves state-of-the-art performance across translation, rotation, scaling, and composite editing tasks.

Problem

• geometric image editing
• 3D-aware prompting
• visual prompting
• image manipulation
• 3D transformation

Innovation

• 3D-aware visual prompting
• geometric image editing
• RGB 3D bounding box
• decoupled geometric control
• SFT-RL training