ICME 2025 Generalizable HDR and SDR Video Quality Measurement Grand Challenge

2025-06-28

AI Summary

Current video quality assessment (VQA) models exhibit limited generalization across dynamic ranges (HDR/SDR), distortion types, and diverse content. To address this, we introduce the first standardized benchmark framework for joint HDR and SDR video evaluation, supporting both full-reference and no-reference paradigms, and incorporating multi-distortion and multi-content test cases to advance unified VQA modeling. Leveraging this benchmark, five teams submitted seven models; four significantly outperformed the VMAF baseline, with the top-performing model achieving state-of-the-art performance in both cross-domain consistency and absolute accuracy. This work constitutes the first systematic effort to overcome the generalization bottleneck in multi-dynamic-range VQA, establishing a new, broadly applicable standard for video quality assessment.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This paper reports IEEE International Conference on Multimedia & Expo (ICME) 2025 Grand Challenge on Generalizable HDR and SDR Video Quality Measurement. With the rapid development of video technology, especially High Dynamic Range (HDR) and Standard Dynamic Range (SDR) contents, the need for robust and generalizable Video Quality Assessment (VQA) methods has become increasingly demanded. Existing VQA models often struggle to deliver consistent performance across varying dynamic ranges, distortion types, and diverse content. This challenge was established to benchmark and promote VQA approaches capable of jointly handling HDR and SDR content. In the final evaluation phase, five teams submitted seven models along with technical reports to the Full Reference (FR) and No Reference (NR) tracks. Among them, four methods outperformed VMAF baseline, while the top-performing model achieved state-of-the-art performance, setting a new benchmark for generalizable video quality assessment.

Problem

• Develop generalizable HDR and SDR video quality assessment methods
• Address inconsistent VQA performance across dynamic ranges and distortions
• Benchmark models for joint HDR and SDR content evaluation

Innovation

• Generalizable HDR and SDR video assessment
• Benchmarking diverse VQA models
• State-of-the-art performance achieved