LiTo: Surface Light Field Tokenization

2026-03-11

AI Summary

Existing methods struggle to jointly model 3D geometry and view-dependent appearance effects—such as specular highlights and Fresnel reflections—within a unified framework. This work proposes a novel implicit 3D representation that, for the first time, encodes stochastically subsampled surface light fields into compact latent vectors, enabling a unified latent space that co-represents both geometry and view-dependent appearance. Leveraging a single input image, the method employs an implicit flow-matching model to generate 3D objects with consistent illumination and material properties. The approach outperforms prior work in both visual realism and input fidelity, accurately reproducing complex view-dependent effects under challenging lighting conditions.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Representation Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We propose a 3D latent representation that jointly models object geometry and view-dependent appearance. Most prior works focus on either reconstructing 3D geometry or predicting view-independent diffuse appearance, and thus struggle to capture realistic view-dependent effects. Our approach leverages that RGB-depth images provide samples of a surface light field. By encoding random subsamples of this surface light field into a compact set of latent vectors, our model learns to represent both geometry and appearance within a unified 3D latent space. This representation reproduces view-dependent effects such as specular highlights and Fresnel reflections under complex lighting. We further train a latent flow matching model on this representation to learn its distribution conditioned on a single input image, enabling the generation of 3D objects with appearances consistent with the lighting and materials in the input. Experiments show that our approach achieves higher visual quality and better input fidelity than existing methods.

Problem

• view-dependent appearance
• surface light field
• 3D reconstruction
• specular highlights
• Fresnel reflections

Innovation

• surface light field
• 3D latent representation
• view-dependent appearance
• latent flow matching
• geometry-appearance joint modeling