Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance

2026-05-07

AI Summary

This work addresses the challenge of efficiently and accurately reconstructing both 3D geometry and reflectance properties of objects in a unified framework. It presents the first differentiable pipeline that jointly optimizes adaptive 4D structured illumination design and geometry-appearance reconstruction in an end-to-end manner. The method integrates a unified spatial-angular structured light pattern, a single-camera setup, a histogram-based per-pixel probabilistic model, and differentiable rendering, while optimizing illumination strategies through a joint consistency constraint between physical measurements and simulations. Experiments demonstrate that the approach achieves superior depth reconstruction accuracy compared to state-of-the-art methods across a variety of complex shapes and materials, and produces reflectance parameters that closely match ground-truth appearance in real captured images.

Abstract

We present a differentiable framework to adaptively compute 4D illumination conditions with respect to an object, for efficient, high-quality simultaneous acquisition of its shape and reflectance, with a unified spatial-angular structured light and a single camera. Using a simple histogram-based pixel-level probability model for depth and reflectance, we differentiably link the next illumination condition(s) with a loss that encourages the reduction in depth uncertainty. As new structured illumination is cast, corresponding image measurements are used to update the uncertainty at each pixel. Finally, a fine-tuning-based approach reconstructs the depth map and reflectance parameter maps, by minimizing the differences between all physical measurements and their simulated counterparts. The effectiveness of our framework is demonstrated on physical objects with wide variations in shape and appearance. Our depth results compare favorably with state-of-the-art techniques, while our reflectance results are comparable when validated against photographs.

Problem

• shape acquisition
• reflectance capture
• structured illumination
• 4D lighting
• joint reconstruction

Innovation

• differentiable rendering
• adaptive illumination
• structured light
• shape and reflectance capture
• uncertainty-aware optimization