Polarimetric BSSRDF Acquisition of Dynamic Faces

2024-11-19

ACM Transactions on Graphics

AI Summary

Existing methods struggle to acquire dynamic facial scattering properties with high fidelity—particularly under complex geometry, multi-layer skin subsurface scattering, and expressive deformations—thereby limiting physically plausible rendering. This paper introduces the first integrated polarization + multispectral + BSSRDF acquisition framework specifically designed for dynamic faces, enabling simultaneous reconstruction of spatially varying appearance, high-fidelity dynamic geometry, and biophysically meaningful skin parameters (hemoglobin and melanin concentrations). We propose a novel polarization-aware skin reflectance model that is directly embeddable into rendering pipelines, unifying polarization imaging, multispectral inversion, BSSRDF modeling, and non-rigid geometric registration. Evaluated on real human face data, our method achieves estimation errors of <8% for constituent layer concentrations. It significantly enhances realism and generalizability in dynamic facial rendering while maintaining compatibility with mainstream production renderers.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Acquisition and modeling of polarized light reflection and scattering help reveal the shape, structure, and physical characteristics of an object, which is increasingly important in computer graphics. However, current polarimetric acquisition systems are limited to static and opaque objects. Human faces, on the other hand, present a particularly difficult challenge, given their complex structure and reflectance properties, the strong presence of spatially-varying subsurface scattering, and their dynamic nature. We present a new polarimetric acquisition method for dynamic human faces, which focuses on capturing spatially varying appearance and precise geometry, across a wide spectrum of skin tones and facial expressions. It includes both single and heterogeneous subsurface scattering, index of refraction, and specular roughness and intensity, among other parameters, while revealing biophysically-based components such as inner- and outer-layer hemoglobin, eumelanin and pheomelanin. Our method leverages such components' unique multispectral absorption profiles to quantify their concentrations, which in turn inform our model about the complex interactions occurring within the skin layers. To our knowledge, our work is the first to simultaneously acquire polarimetric and spectral reflectance information alongside biophysically-based skin parameters and geometry of dynamic human faces. Moreover, our polarimetric skin model integrates seamlessly into various rendering pipelines.

Problem

• Dynamic Facial Scattering
• Complex Subdermal Scattering
• Dynamic Expressions

Innovation

• Dynamic Facial Scattering Analysis
• Spectral Polarimetry
• Skin Model Integration