RealSkin: Spatio-Spectral Partial Neural Adjoint Maps for Image-to-3D Attribute Transfer

2026-07-14

AI Summary

This work addresses the challenge of visual attribute transfer between real-world images and synthetic 3D models, which is hindered by mismatches in resolution, topological discrepancies, and non-isometric deformations. To tackle this, the authors propose RealSkin, a framework that integrates spatially guided coarse alignment with a spectral-aware neural correspondence network, optimizing partial correspondences in a learned spectral domain. By incorporating spectral-domain neural function space modeling, self-supervised learning, and explicit modeling of non-isometric residuals, RealSkin overcomes the conventional reliance on near-isometry and topological consistency. This enables, for the first time, robust handling of scenarios involving non-isometric, partial, and topologically inconsistent data. Experiments demonstrate that RealSkin achieves state-of-the-art performance in real-to-synthetic 3D attribute transfer, significantly improving fidelity and robustness in complex settings.

Abstract

Creating photorealistic 3D assets requires bridging the appearance gap between real-world observations and synthetic models. A promising approach is to transfer visual attributes from real images onto synthetic 3D surfaces. Traditional methods struggle with resolution mismatch and the inherent discreteness of point correspondences. In contrast, resolution-robust functional maps enable smooth attribute propagation but rely on near-isometry assumptions and topological consistency. To address these limitations, we propose RealSkin, a self-supervised framework that performs correspondence optimization in a learned spectral domain, guided by spatial correspondences. We first introduce a spatial-guided registration algorithm to establish coarse correspondences under severe topological discrepancies. To relax strict isometric assumptions and handle partial correspondences, we further design a spectral-aware neural adjoint network that incorporates partial correspondences into a neural function space and models non-isometric residuals for correspondence refinement. Experimental results demonstrate that our method achieves state-of-the-art performance on challenging real-to-synthetic scenarios. The code will be publicly released.

Problem

• attribute transfer
• 3D reconstruction
• correspondence
• non-isometric deformation
• topological discrepancy

Innovation

• neural adjoint maps
• spectral correspondence
• self-supervised 3D transfer
• non-isometric alignment
• partial functional maps