Position: Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered

2026-02-28

AI Summary

This work proposes a human-centered evaluation paradigm for visual processing systems that moves beyond reliance on single image quality metrics, which often fail to capture human perception and user preferences. By integrating objective image quality assessment (IQA), human perceptual experiments, and fine-grained modeling of user preferences, the study establishes a context-aware, comprehensive evaluation framework. The research uncovers significant discrepancies between widely used image quality metrics and actual human judgments, thereby offering both theoretical insights and methodological support for developing more application-aligned evaluation protocols for visual models.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Humans and AI: Learning Human Values and Preferences
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

This position paper argues that the evaluation of modern visual processing systems should no longer be driven primarily by single-metric image quality assessment benchmarks, particularly in the era of generative and perception-oriented methods. Image restoration exemplifies this divergence: while objective IQA metrics enable reproducible, scalable evaluation, they have increasingly drifted apart from human perception and user preferences. We contend that this mismatch risks constraining innovation and misguiding research progress across visual processing tasks. Rather than rejecting metrics altogether, this paper calls for a rebalancing of evaluation paradigms, advocating a more human-centered, context-aware, and fine-grained approach to assessing the visual models' outcomes.

Problem

• image quality assessment
• human perception
• visual processing
• evaluation metrics
• user preference

Innovation

• human-centered evaluation
• image quality assessment
• perception-oriented methods
• visual processing
• context-aware evaluation