Differentiable Mobile Display Photometric Stereo

2025-02-07

AI Summary

Existing differentiable photometric stereo (DDPS) methods rely on fixed desktop setups and specialized polarization hardware, limiting practical deployment. This paper proposes Differentiable Mobile Display Photometric Stereo (DMDPS), the first framework to jointly integrate physics-based modeling and end-to-end learning on mobile platforms. DMDPS leverages a smartphone display as a programmable light source and synchronizes HDR image capture via the built-in camera for portable surface normal and albedo reconstruction. The method comprises mobile-native differentiable rendering, HDR image alignment, physically grounded illumination modeling, and neural optimization of display patterns. We implement native iOS and Android applications. Evaluations on 3D-printed objects and real-world fallen leaves demonstrate significantly improved normal estimation accuracy. DMDPS enables real-time, closed-loop acquisition–reconstruction directly on smartphones. We further release the first public dataset of surface normals and albedos for natural fallen leaves.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Learning with Manifolds

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Display photometric stereo uses a display as a programmable light source to illuminate a scene with diverse illumination conditions. Recently, differentiable display photometric stereo (DDPS) demonstrated improved normal reconstruction accuracy by using learned display patterns. However, DDPS faced limitations in practicality, requiring a fixed desktop imaging setup using a polarization camera and a desktop-scale monitor. In this paper, we propose a more practical physics-based photometric stereo, differentiable mobile display photometric stereo (DMDPS), that leverages a mobile phone consisting of a display and a camera. We overcome the limitations of using a mobile device by developing a mobile app and method that simultaneously displays patterns and captures high-quality HDR images. Using this technique, we capture real-world 3D-printed objects and learn display patterns via a differentiable learning process. We demonstrate the effectiveness of DMDPS on both a 3D printed dataset and a first dataset of fallen leaves. The leaf dataset contains reconstructed surface normals and albedos of fallen leaves that may enable future research beyond computer graphics and vision. We believe that DMDPS takes a step forward for practical physics-based photometric stereo.

Problem

• Enhances photometric stereo using mobile devices
• Overcomes fixed setup with mobile app
• Improves 3D reconstruction with learned display patterns

Innovation

• Mobile app for HDR imaging
• Differentiable learning display patterns
• Physics-based photometric stereo