Thinking in Boxes: 3D Editing in Real Images Made Easy

2026-06-18

AI Summary

Existing image editing methods struggle to precisely control large-scale object motion and viewpoint changes. This work proposes a structured editing framework based on 3D bounding boxes, where users define target transformations by specifying input and output 3D boxes, which the system then formulates as a geometric constraint problem. The approach introduces an innovative “box-based thinking” interface—using color-coded faces to explicitly represent orientation—and leverages a depth-aligned planar floor as a global reference frame. Combined with depth-aware rendering, a two-stage training strategy (utilizing synthetic multi-object scenes and real-world Objectron videos), and a structure-conditioned generative model, the method directly processes real images and significantly outperforms existing techniques in large-scale 3D editing tasks while preserving object identity and plausibly reconstructing occluded regions.

Abstract

Text and 2D-conditioning interfaces provide weak, ambiguous control over spatial transformations in image editing -- particularly under large object motions and camera changes. Prior work has used 3D primitives such as boxes, but only as loose conditioning signals indicating approximate object location rather than specifying the transformation. We instead use 3D boxes as structured specifications: the user provides the input and output boxes of the edit, casting editing as a well-posed geometry problem. This ``thinking in boxes'' interface, where each box face is color-coded to convey 3D orientation, gives precise control over translation, rotation, scaling, and viewpoint changes in real images while preserving scene and object identity, and recovering previously unseen object regions. To ground transformations in scene appearance, we introduce a depth-aligned planar floor as a global reference frame, shaded with depth-aware cues. Conditioned on this structure, an image generator produces consistent results under large transformations. Trained in two stages -- on synthetic multi-object scenes and a small set of real-world videos from Objectron -- the system generalizes to complex, in-the-wild real images. Our method operates directly on real photographs and substantially outperforms recent state-of-the-art methods on large 3D edits.

Problem

• 3D editing
• image manipulation
• spatial transformation
• 3D boxes
• real images

Innovation

• 3D editing
• structured 3D boxes
• depth-aligned reference frame
• geometry-aware image generation
• real-image manipulation