Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity

2025-08-07

AI Summary

Existing 3D generation evaluation methods rely predominantly on image-level metrics, failing to adequately assess spatial consistency, material realism, and fine-grained geometric details. Method: We propose Hi3DEval, the first hierarchical evaluation framework jointly quantifying object-level and part-level fidelity. It introduces fine-grained material modeling—including albedo and metallicness—and integrates spatiotemporal consistency via video-sequence modeling alongside part-aware analysis using pre-trained 3D features. Contribution/Results: To support Hi3DEval, we construct Hi3DBench—the first high-quality, dual-level benchmark annotated via multi-agent collaboration. Our end-to-end 3D-perceptual automatic scoring system achieves state-of-the-art performance, significantly outperforming conventional image metrics in spatial plausibility, material fidelity, and alignment with human preferences—establishing a scalable, automated paradigm for 3D generation evaluation.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Despite rapid advances in 3D content generation, quality assessment for the generated 3D assets remains challenging. Existing methods mainly rely on image-based metrics and operate solely at the object level, limiting their ability to capture spatial coherence, material authenticity, and high-fidelity local details. 1) To address these challenges, we introduce Hi3DEval, a hierarchical evaluation framework tailored for 3D generative content. It combines both object-level and part-level evaluation, enabling holistic assessments across multiple dimensions as well as fine-grained quality analysis. Additionally, we extend texture evaluation beyond aesthetic appearance by explicitly assessing material realism, focusing on attributes such as albedo, saturation, and metallicness. 2) To support this framework, we construct Hi3DBench, a large-scale dataset comprising diverse 3D assets and high-quality annotations, accompanied by a reliable multi-agent annotation pipeline. We further propose a 3D-aware automated scoring system based on hybrid 3D representations. Specifically, we leverage video-based representations for object-level and material-subject evaluations to enhance modeling of spatio-temporal consistency and employ pretrained 3D features for part-level perception. Extensive experiments demonstrate that our approach outperforms existing image-based metrics in modeling 3D characteristics and achieves superior alignment with human preference, providing a scalable alternative to manual evaluations. The project page is available at https://zyh482.github.io/Hi3DEval/.

Problem

• Evaluating 3D asset quality beyond image-based metrics
• Assessing spatial coherence and material authenticity in 3D generation
• Developing automated scoring for 3D-aware hierarchical evaluation

Innovation

• Hierarchical evaluation framework for 3D content
• Large-scale dataset with multi-agent annotations
• 3D-aware automated scoring system