OpenMaterial: A Comprehensive Dataset of Complex Materials for 3D Reconstruction

2024-06-13

arXiv.org

AI Summary

Reconstructing 3D geometry of objects with complex optical materials—such as metals and glass—is challenging due to specular reflection, refraction, and transparency, which cause severe multi-view color inconsistency. To address this, we introduce the first large-scale, high-fidelity complex-material reconstruction dataset: comprising 1,001 real-world objects, 295 physically measured material types, and multi-view images rendered under 723 distinct lighting conditions using PBRT-based physically based rendering. The dataset provides comprehensive annotations, including ground-truth 3D shapes, material categories, camera poses, depth maps, and segmentation masks. Key contributions include: (i) the first systematic benchmark supporting quantitative evaluation of conductors and rough dielectrics; (ii) a novel rendering pipeline driven by laboratory-measured indices of refraction (IOR); and (iii) unprecedented diversity in both material properties and illumination. This dataset has become a de facto evaluation standard for state-of-the-art methods—including NeRF and IDR—yielding significant improvements in reconstruction accuracy on challenging materials. Code and data are publicly available and widely adopted.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Recent advances in deep learning such as neural radiance fields and implicit neural representations have significantly propelled the field of 3D reconstruction. However, accurately reconstructing objects with complex optical properties, such as metals and glass, remains a formidable challenge due to their unique specular and light-transmission characteristics. To facilitate the development of solutions to these challenges, we introduce the OpenMaterial dataset, comprising 1001 objects made of 295 distinct materials-including conductors, dielectrics, plastics, and their roughened variants- and captured under 723 diverse lighting conditions. To this end, we utilized physics-based rendering with laboratory-measured Indices of Refraction (IOR) and generated high-fidelity multiview images that closely replicate real-world objects. OpenMaterial provides comprehensive annotations, including 3D shape, material type, camera pose, depth, and object mask. It stands as the first large-scale dataset enabling quantitative evaluations of existing algorithms on objects with diverse and challenging materials, thereby paving the way for the development of 3D reconstruction algorithms capable of handling complex material properties.

Problem

• Reconstructing objects with complex optical properties remains challenging
• Lack of benchmark datasets modeling material-dependent light transport
• Need robust techniques for handling real-world optical complexities

Innovation

• Created large-scale semi-synthetic dataset with 1001 objects
• Integrated lab-measured refractive index spectra for simulation
• Provided first extensive benchmark for material-aware reconstruction