OpticFusion: Multi-Modal Neural Implicit 3D Reconstruction of Microstructures by Fusing White Light Interferometry and Optical Microscopy

2025-01-16

AI Summary

Traditional white-light interferometry (WLI) enables sub-micrometer-accurate 3D topography reconstruction but lacks authentic surface color information. This paper introduces the first cross-modal neural implicit fusion framework integrating WLI and optical microscopy (OM) to jointly reconstruct microstructural 3D geometry and natural color. Methodologically, we propose a two-stage geometry-appearance co-registration strategy; incorporate chromatic decoupling to separate and recover sample-intrinsic reflectance; and unify NeRF-style implicit representation, cross-modal feature alignment, and differentiable rendering. Evaluated on a newly constructed multimodal microscale dataset, our approach achieves sub-micrometer geometric accuracy and high-fidelity photorealistic texture reconstruction. Both source code and the benchmark dataset are publicly released.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

White Light Interferometry (WLI) is a precise optical tool for measuring the 3D topography of microstructures. However, conventional WLI cannot capture the natural color of a sample's surface, which is essential for many microscale research applications that require both 3D geometry and color information. Previous methods have attempted to overcome this limitation by modifying WLI hardware and analysis software, but these solutions are often costly. In this work, we address this challenge from a computer vision multi-modal reconstruction perspective for the first time. We introduce OpticFusion, a novel approach that uses an additional digital optical microscope (OM) to achieve 3D reconstruction with natural color textures using multi-view WLI and OM images. Our method employs a two-step data association process to obtain the poses of WLI and OM data. By leveraging the neural implicit representation, we fuse multi-modal data and apply color decomposition technology to extract the sample's natural color. Tested on our multi-modal dataset of various microscale samples, OpticFusion achieves detailed 3D reconstructions with color textures. Our method provides an effective tool for practical applications across numerous microscale research fields. The source code and our real-world dataset are available at https://github.com/zju3dv/OpticFusion.

Problem

• White Light Interferometry
• Color Information
• 3D Shape Measurement

Innovation

• OpticFusion
• Computer Vision
• True Color 3D Reconstruction