Rethinking Image Evaluation in Super-Resolution

2025-03-17

AI Summary

Existing super-resolution (SR) evaluation heavily relies on reference ground-truth (GT) images, yet real-world GTs often exhibit inconsistent quality, leading to distorted and perceptually misaligned metrics such as PSNR and LPIPS. Method: This work is the first to systematically characterize the detrimental impact of GT quality bias on SR evaluation and proposes RQI—the first relative quality-aware metric designed for scenarios where GTs are unreliable. RQI models pairwise relative quality relationships by jointly leveraging multi-scale structural and semantic feature discrepancies, without assuming access to a perfect GT. Contribution/Results: Extensive validation across multiple real-world SR benchmarks demonstrates that RQI achieves significantly higher correlation with human subjective scores than state-of-the-art metrics. It exhibits superior robustness, stability, and reproducibility, and effectively discriminates models’ true perceptual performance—enabling more reliable and human-aligned SR assessment.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements

Abstract

While recent advancing image super-resolution (SR) techniques are continually improving the perceptual quality of their outputs, they can usually fail in quantitative evaluations. This inconsistency leads to a growing distrust in existing image metrics for SR evaluations. Though image evaluation depends on both the metric and the reference ground truth (GT), researchers typically do not inspect the role of GTs, as they are generally accepted as `perfect' references. However, due to the data being collected in the early years and the ignorance of controlling other types of distortions, we point out that GTs in existing SR datasets can exhibit relatively poor quality, which leads to biased evaluations. Following this observation, in this paper, we are interested in the following questions: Are GT images in existing SR datasets 100% trustworthy for model evaluations? How does GT quality affect this evaluation? And how to make fair evaluations if there exist imperfect GTs? To answer these questions, this paper presents two main contributions. First, by systematically analyzing seven state-of-the-art SR models across three real-world SR datasets, we show that SR performances can be consistently affected across models by low-quality GTs, and models can perform quite differently when GT quality is controlled. Second, we propose a novel perceptual quality metric, Relative Quality Index (RQI), that measures the relative quality discrepancy of image pairs, thus issuing the biased evaluations caused by unreliable GTs. Our proposed model achieves significantly better consistency with human opinions. We expect our work to provide insights for the SR community on how future datasets, models, and metrics should be developed.

Problem

• Examines trustworthiness of ground truth images in SR datasets
• Investigates impact of GT quality on super-resolution model evaluations
• Proposes new metric to address bias from unreliable ground truths

Innovation

• Analyzes impact of low-quality ground truth on SR models
• Introduces Relative Quality Index for fair image evaluation
• Enhances consistency with human perceptual quality assessments