Uncertainty-driven Sampling for Efficient Pairwise Comparison Subjective Assessment

2024-11-27

arXiv.org

AI Summary

Pairwise comparison in subjective image quality assessment is costly and inefficient. To address this, we propose an uncertainty-aware active sampling method that jointly models human preferences and prediction uncertainty to intelligently select the most informative image pairs for human annotation. Our approach integrates deep learning–based preference modeling, Bayesian uncertainty estimation, and pairwise comparison graph optimization, enabling efficient ranking learning under constrained annotation budgets. The key innovation lies in the unified use of model prediction uncertainty—both for preference modeling and for active sample selection—thereby achieving synergistic optimization of ranking accuracy and annotation efficiency. Experiments demonstrate that, under identical annotation budgets, our method improves ranking accuracy by over 12% compared to conventional active learning baselines, significantly reducing human labeling effort.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Assessing image quality is crucial in image processing tasks such as compression, super-resolution, and denoising. While subjective assessments involving human evaluators provide the most accurate quality scores, they are impractical for large-scale or continuous evaluations due to their high cost and time requirements. Pairwise comparison subjective assessment tests, which rank image pairs instead of assigning scores, offer more reliability and accuracy but require numerous comparisons, leading to high costs. Although objective quality metrics are more efficient, they lack the precision of subjective tests, which are essential for benchmarking and training learning-based quality metrics. This paper proposes an uncertainty-based sampling method to optimize the pairwise comparison subjective assessment process. By utilizing deep learning models to estimate human preferences and identify pairs that need human labeling, the approach reduces the number of required comparisons while maintaining high accuracy. The key contributions include modeling uncertainty for accurate preference predictions and for pairwise sampling. The experimental results demonstrate superior performance of the proposed approach compared to traditional active sampling methods. Software is publicly available at: shimamohammadi/LBPS-EIC

Problem

• Optimize pairwise comparison for image quality assessment
• Reduce human comparisons using uncertainty-driven sampling
• Maintain accuracy while lowering cost and time

Innovation

• Uncertainty-driven sampling for pairwise comparisons
• Deep learning predicts human preference uncertainty
• Reduces human comparisons while maintaining accuracy