GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusion

2026-05-12

AI Summary

This work addresses the limitation of existing image-to-3D generation methods, which prioritize appearance over geometry, often resulting in structurally unreliable and cross-view inconsistent outputs. To overcome this, the authors propose GTA, a two-stage paradigm that first recovers geometry and then synthesizes appearance. GTA employs two specialized video diffusion models: the first generates multi-view coarse geometry from a single input image, and the second refines detailed appearance conditioned on this geometry. Inspired by human visual perception, the approach introduces a coarse-to-fine mechanism that effectively decouples geometry and appearance. Additionally, a random latent shuffling strategy during training and a scaling scheme at test time significantly enhance structural fidelity and view consistency. Experiments demonstrate that GTA outperforms current methods in both geometric accuracy and visual quality, and it can serve as a plug-and-play module to improve other 3D generation pipelines, offering strong data efficiency and downstream adaptability.

Abstract

Recent developments in generative models and large-scale datasets have substantially advanced 3D world generation, facilitating a broad range of domains including spatial intelligence, embodied intelligence, and autonomous driving. While achieving remarkable progress, existing approaches to 3D world generation typically prioritize appearance prediction with limited modeling of the underlying geometry, leading to issues such as unreliable scene structure estimation and degraded cross-view consistency. To address these limitations, motivated by the coarse-to-fine nature of human visual perception, we propose GTA, a novel image-to-3D world generation method following a Geometry-Then-Appearance paradigm. Specifically, given a single input image, to improve the structural fidelity of synthesized 3D scenes, GTA adopts a two-stage framework with two dedicated video diffusion models, which first generate coarse geometric structure from novel viewpoints and then synthesize fine-grained appearance conditioned on the predicted geometry. To further enhance cross-view appearance consistency, we introduce a random latent shuffle strategy during the training process, along with a test-time scaling scheme that improves perceptual quality without compromising quantitative performance. Extensive experiments have demonstrated that our proposed method consistently outperforms existing approaches in terms of fidelity, visual quality, and geometric accuracy. Moreover, GTA is shown to be effective as a general enhancement module that further improves the generation quality of existing image-to-3D world pipelines, as well as supporting multiple downstream applications and exhibiting favorable data efficiency during model training, highlighting its versatility and broad applicability. Project page: https://hanxinzhu-lab.github.io/GTA/.

Problem

• 3D world generation
• geometry modeling
• cross-view consistency
• appearance prediction
• scene structure

Innovation

• Geometry-Then-Appearance
• video diffusion models
• cross-view consistency
• 3D world generation
• two-stage framework