Pygmalion Effect in Vision: Image-to-Clay Translation for Reflective Geometry Reconstruction

2025-11-26

AI Summary

Viewpoint-dependent reflectance strongly couples appearance with geometry, severely degrading 3D reconstruction accuracy. To address this, we introduce the “Pygmalion Effect in Vision”: a dual-branch network comprising (i) a BRDF modeling branch that explicitly encodes reflectance properties, and (ii) a clay-guided branch leveraging synthetically rendered, reflection-free, neutral-appearance clay images as weak supervision—introducing a novel geometric inductive bias. Both branches jointly optimize surface normals and mesh geometry for consistency. Evaluated on synthetic and real-world datasets, our method achieves significant improvements in normal estimation accuracy (+12.7%) and mesh completeness. Crucially, it demonstrates superior robustness in complex multi-reflection scenarios, outperforming state-of-the-art approaches. To our knowledge, this is the first work to enable *co-modeling* of reflection suppression and geometrically stable surface recovery.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Learning with Manifolds
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Understanding reflection remains a long-standing challenge in 3D reconstruction due to the entanglement of appearance and geometry under view-dependent reflections. In this work, we present the Pygmalion Effect in Vision, a novel framework that metaphorically "sculpts" reflective objects into clay-like forms through image-to-clay translation. Inspired by the myth of Pygmalion, our method learns to suppress specular cues while preserving intrinsic geometric consistency, enabling robust reconstruction from multi-view images containing complex reflections. Specifically, we introduce a dual-branch network in which a BRDF-based reflective branch is complemented by a clay-guided branch that stabilizes geometry and refines surface normals. The two branches are trained jointly using the synthesized clay-like images, which provide a neutral, reflection-free supervision signal that complements the reflective views. Experiments on both synthetic and real datasets demonstrate substantial improvement in normal accuracy and mesh completeness over existing reflection-handling methods. Beyond technical gains, our framework reveals that seeing by unshining, translating radiance into neutrality, can serve as a powerful inductive bias for reflective object geometry learning.

Problem

• Reconstructing 3D geometry from images with complex view-dependent reflections
• Disentangling appearance and geometry for reflective object reconstruction
• Suppressing specular cues while preserving intrinsic geometric consistency

Innovation

• Dual-branch network suppresses reflections while preserving geometry
• Clay-guided branch stabilizes geometry and refines surface normals
• Synthesized clay-like images provide reflection-free supervision signal