DiffPhysCam: Differentiable Physics-Based Camera Simulation for Inverse Rendering and Embodied AI

2025-08-12

AI Summary

Existing virtual cameras lack differentiable modeling of optical properties and intrinsic parameters, leading to distorted optical artifact simulation and limited sim-to-real transfer performance. This paper introduces DiffPhysCam—the first differentiable physical camera simulator enabling gradient-based optimization—unifying forward and inverse rendering via multi-stage optical modeling, including defocus blur. Its key innovation lies in embedding tunable calibration parameters (e.g., focal length, aperture, image distance) directly into a physics-based imaging pipeline, while tightly coupling multi-physics simulation with inverse optimization algorithms. This enables digital twin scene reconstruction from real images and end-to-end optimization of 3D geometry and material properties. Evaluated on synthetic datasets and autonomous driving navigation tasks, DiffPhysCam significantly improves robotic visual perception robustness and sim-to-real consistency.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Data transparency and provenance

Abstract

We introduce DiffPhysCam, a differentiable camera simulator designed to support robotics and embodied AI applications by enabling gradient-based optimization in visual perception pipelines. Generating synthetic images that closely mimic those from real cameras is essential for training visual models and enabling end-to-end visuomotor learning. Moreover, differentiable rendering allows inverse reconstruction of real-world scenes as digital twins, facilitating simulation-based robotics training. However, existing virtual cameras offer limited control over intrinsic settings, poorly capture optical artifacts, and lack tunable calibration parameters -- hindering sim-to-real transfer. DiffPhysCam addresses these limitations through a multi-stage pipeline that provides fine-grained control over camera settings, models key optical effects such as defocus blur, and supports calibration with real-world data. It enables both forward rendering for image synthesis and inverse rendering for 3D scene reconstruction, including mesh and material texture optimization. We show that DiffPhysCam enhances robotic perception performance in synthetic image tasks. As an illustrative example, we create a digital twin of a real-world scene using inverse rendering, simulate it in a multi-physics environment, and demonstrate navigation of an autonomous ground vehicle using images generated by DiffPhysCam.

Problem

• Enables gradient-based optimization in visual perception pipelines
• Improves sim-to-real transfer with fine-grained camera control
• Supports inverse rendering for 3D scene reconstruction

Innovation

• Differentiable camera simulator for gradient-based optimization
• Models optical effects like defocus blur accurately
• Supports both forward and inverse rendering capabilities