MMP-2K: A Benchmark Multi-Labeled Macro Photography Image Quality Assessment Database

2025-05-25

AI Summary

Macro photography image quality assessment (MPIQA) lacks dedicated benchmark datasets, hindering progress in this specialized domain. Method: We introduce MMP-2K, the first large-scale, multi-annotated macro image quality assessment benchmark, comprising 2,000 diverse macro photographs. Each image is rated by 17 human subjects, and—uniquely—we establish a fine-grained, multi-dimensional distortion annotation scheme covering distortion type, spatial location, and severity. Rigorous laboratory-based subjective experiments incorporate outlier rejection and multi-source sampling to ensure annotation reliability. Contribution/Results: Our analysis systematically demonstrates the substantial failure of general-purpose full-reference and no-reference IQA metrics on macro imagery, confirming the necessity of domain-specific evaluation. Experimental results show that state-of-the-art IQA metrics suffer average performance degradation exceeding 40% on MMP-2K. The complete dataset—including all images, subjective scores, and fine-grained distortion annotations—is publicly released to support standardized algorithm development and evaluation in MPIQA.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Image and Video Retrieval
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Macro photography (MP) is a specialized field of photography that captures objects at an extremely close range, revealing tiny details. Although an accurate macro photography image quality assessment (MPIQA) metric can benefit macro photograph capturing, which is vital in some domains such as scientific research and medical applications, the lack of MPIQA data limits the development of MPIQA metrics. To address this limitation, we conducted a large-scale MPIQA study. Specifically, to ensure diversity both in content and quality, we sampled 2,000 MP images from 15,700 MP images, collected from three public image websites. For each MP image, 17 (out of 21 after outlier removal) quality ratings and a detailed quality report of distortion magnitudes, types, and positions are gathered by a lab study. The images, quality ratings, and quality reports form our novel multi-labeled MPIQA database, MMP-2k. Experimental results showed that the state-of-the-art generic IQA metrics underperform on MP images. The database and supplementary materials are available at https://github.com/Future-IQA/MMP-2k.

Problem

• Lack of macro photography image quality assessment (MPIQA) data.
• Need for diverse MPIQA metrics for scientific and medical applications.
• Existing generic IQA metrics perform poorly on macro photography images.

Innovation

• Created multi-labeled MPIQA database MMP-2k
• Collected 2000 diverse macro photography images
• Gathered detailed quality ratings and reports