RGBX-Next: Towards Realistic Generative Rendering from G-Buffers

2026-08-13

AI Summary

This study addresses the limited controllability of diffusion models by proposing a unified forward-inverse rendering framework based on G-buffers. By fine-tuning Diffusion Transformers (DiT) into universal generative renderers and employing G-buffer conditional guidance alongside joint training strategies, the method achieves precise estimation and generation of photorealistic content from images and videos. This approach effectively bridges the gap between generative models and traditional rendering pipelines. It delivers high-quality performance in both photorealistic content synthesis and intrinsic attribute decomposition tasks, significantly enhancing physical consistency and controllability in generated outputs. Consequently, this work establishes a robust paradigm for integrating explicit 3D representations with generative AI to improve interpretability and editing capabilities in visual synthesis.

Abstract

Diffusion models have achieved impressive results in image, video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.

Problem

• Diffusion Models
• Controllable Generation
• G-Buffers
• Forward and Inverse Rendering

Innovation

• Diffusion Transformer
• G-Buffers
• Inverse Rendering
• Generative Rendering
• RGBX-Next