GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models

2026-03-24

AI Summary

This work addresses the limitations of existing methods in reconstructing renderable 3D models from images, which often struggle to accurately capture complex appearance and lack precise viewpoint control. The authors propose a novel approach that, for the first time, integrates a 3D reconstruction prior with a video diffusion model, leveraging 3D-aware conditioning to guide the generation process. Without explicitly modeling materials or lighting, the method enables high-fidelity rendering of objects under arbitrary viewpoints and illumination conditions. It significantly outperforms current state-of-the-art techniques in tasks such as novel view synthesis, relighting, and video insertion, achieving high-quality neural rendering with accurate viewpoint control.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reconstructing a renderable 3D model from images is a useful but challenging task. Recent feedforward 3D reconstruction methods have demonstrated remarkable success in efficiently recovering geometry, but still cannot accurately model the complex appearances of these 3D reconstructed models. Recent diffusion-based generative models can synthesize realistic images or videos of an object using reference images without explicitly modeling its appearance, which provides a promising direction for object rendering, but lacks accurate control over the viewpoints. In this paper, we propose GO-Renderer, a unified framework integrating the reconstructed 3D proxies to guide the video generative models to achieve high-quality object rendering on arbitrary viewpoints under arbitrary lighting conditions. Our method not only enjoys the accurate viewpoint control using the reconstructed 3D proxy but also enables high-quality rendering in different lighting environments using diffusion generative models without explicitly modeling complex materials and lighting. Extensive experiments demonstrate that GO-Renderer achieves state-of-the-art performance across the object rendering tasks, including synthesizing images on new viewpoints, rendering the objects in a novel lighting environment, and inserting an object into an existing video.

Problem

• 3D reconstruction
• object rendering
• viewpoint control
• lighting conditions
• appearance modeling

Innovation

• 3D-aware rendering
• controllable video diffusion
• generative object rendering
• viewpoint control
• lighting-condition synthesis