DPCS: Path Tracing-Based Differentiable Projector-Camera Systems

2025-03-15

AI Summary

Existing neural network approaches struggle to faithfully simulate complex optical phenomena—such as soft shadows and interreflections—in projector-camera (ProCam) systems, and fail to explicitly disentangle scene-physical parameters (e.g., material properties, gamma, white balance), resulting in poor generalizability. This paper introduces the first differentiable ProCam simulation framework, integrating multi-bounce path tracing into a differentiable physically based rendering (PBR) pipeline to explicitly model the full physical imaging and projection chain. The method supports spatial AR tasks—including relighting and projector compensation—and enables efficient optimization of scene parameters from only a few observations. Experiments demonstrate that our approach significantly outperforms purely learning-based methods in indirect illumination fidelity, parameter interpretability, and generalization to unseen scenes, while reducing training sample requirements and improving downstream task performance.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Security and Privacy: Data transparency and provenance

Abstract

Projector-camera systems (ProCams) simulation aims to model the physical project-and-capture process and associated scene parameters of a ProCams, and is crucial for spatial augmented reality (SAR) applications such as ProCams relighting and projector compensation. Recent advances use an end-to-end neural network to learn the project-and-capture process. However, these neural network-based methods often implicitly encapsulate scene parameters, such as surface material, gamma, and white balance in the network parameters, and are less interpretable and hard for novel scene simulation. Moreover, neural networks usually learn the indirect illumination implicitly in an image-to-image translation way which leads to poor performance in simulating complex projection effects such as soft-shadow and interreflection. In this paper, we introduce a novel path tracing-based differentiable projector-camera systems (DPCS), offering a differentiable ProCams simulation method that explicitly integrates multi-bounce path tracing. Our DPCS models the physical project-and-capture process using differentiable physically-based rendering (PBR), enabling the scene parameters to be explicitly decoupled and learned using much fewer samples. Moreover, our physically-based method not only enables high-quality downstream ProCams tasks, such as ProCams relighting and projector compensation, but also allows novel scene simulation using the learned scene parameters. In experiments, DPCS demonstrates clear advantages over previous approaches in ProCams simulation, offering better interpretability, more efficient handling of complex interreflection and shadow, and requiring fewer training samples.

Problem

• Simulate projector-camera systems with explicit scene parameters.
• Improve handling of complex projection effects like soft-shadow.
• Enable high-quality ProCams tasks with fewer training samples.

Innovation

• Differentiable physically-based rendering for ProCams simulation
• Explicit integration of multi-bounce path tracing
• Efficient scene parameter learning with fewer samples