Relightable Holoported Characters: Capturing and Relighting Dynamic Human Performance from Sparse Views

2025-11-28

AI Summary

This work addresses free-viewpoint rendering and physically consistent relighting of dynamic full-body human subjects from sparse RGB video inputs, eliminating the need for conventional OLAT (Optical Light-Stage Acquisition Technique) capture. We propose RelightNet—the first Transformer architecture that explicitly incorporates the rendering equation into its feature design—jointly encoding a coarse mesh proxy and input views to estimate geometry, albedo, and shadowing end-to-end in a single forward pass. Our method leverages LightStage multi-view acquisition under randomized ambient illumination and uniform tracking light, models appearance using 3D Gaussian ellipsoid maps, and fuses novel lighting via cross-attention. On dynamic human relighting, RelightNet significantly outperforms state-of-the-art methods, achieving high-fidelity visual quality, accurate photometric response, free-viewpoint navigation, and promising real-time rendering capability.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We present Relightable Holoported Characters (RHC), a novel person-specific method for free-view rendering and relighting of full-body and highly dynamic humans solely observed from sparse-view RGB videos at inference. In contrast to classical one-light-at-a-time (OLAT)-based human relighting, our transformer-based RelightNet predicts relit appearance within a single network pass, avoiding costly OLAT-basis capture and generation. For training such a model, we introduce a new capture strategy and dataset recorded in a multi-view lightstage, where we alternate frames lit by random environment maps with uniformly lit tracking frames, simultaneously enabling accurate motion tracking and diverse illumination as well as dynamics coverage. Inspired by the rendering equation, we derive physics-informed features that encode geometry, albedo, shading, and the virtual camera view from a coarse human mesh proxy and the input views. Our RelightNet then takes these features as input and cross-attends them with a novel lighting condition, and regresses the relit appearance in the form of texel-aligned 3D Gaussian splats attached to the coarse mesh proxy. Consequently, our RelightNet implicitly learns to efficiently compute the rendering equation for novel lighting conditions within a single feed-forward pass. Experiments demonstrate our method's superior visual fidelity and lighting reproduction compared to state-of-the-art approaches. Project page: https://vcai.mpi-inf.mpg.de/projects/RHC/

Problem

• Develops a method for free-view rendering and relighting of dynamic humans from sparse videos
• Predicts relit appearance in a single network pass without costly OLAT capture
• Uses physics-informed features and a transformer to compute rendering equation efficiently

Innovation

• Transformer-based RelightNet predicts relit appearance in single network pass
• Physics-informed features encode geometry, albedo, shading from coarse mesh proxy
• Texel-aligned 3D Gaussian splats attached to mesh for relighting representation