ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Images

2024-07-31

arXiv.org

AI Summary

This work addresses the challenge of perceptual quality assessment for egocentric spatial images (ESIs) displayed on head-mounted displays (HMDs). To this end, we introduce ESIQAD—the first dedicated benchmark database comprising 500 ESIs, each annotated with subjective quality scores across three display modalities: 2D windowed, 3D windowed, and 3D immersive. We propose ESIQAnet, a multi-stage quality prediction model built upon Mamba2, which innovatively integrates visual state-space dual modeling (VSSD), cross-view cross-attention, and transposed attention to achieve unified and accurate quality modeling across heterogeneous display paradigms. Extensive experiments on ESIQAD demonstrate that ESIQAnet consistently outperforms 22 state-of-the-art image quality assessment (IQA) methods, achieving new SOTA performance across all three display modes. Both the ESIQAD database and the implementation code are publicly released to foster further research.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

With the development of eXtended Reality (XR), photo capturing and display technology based on head-mounted displays (HMDs) have experienced significant advancements and gained considerable attention. Egocentric spatial images and videos are emerging as a compelling form of stereoscopic XR content. The assessment for the Quality of Experience (QoE) of XR content is important to ensure a high-quality viewing experience. Different from traditional 2D images, egocentric spatial images present challenges for perceptual quality assessment due to their special shooting, processing methods, and stereoscopic characteristics. However, the corresponding image quality assessment (IQA) research for egocentric spatial images is still lacking. In this paper, we establish the Egocentric Spatial Images Quality Assessment Database (ESIQAD), the first IQA database dedicated for egocentric spatial images as far as we know. Our ESIQAD includes 500 egocentric spatial images and the corresponding mean opinion scores (MOSs) under three display modes, including 2D display, 3D-window display, and 3D-immersive display. Based on our ESIQAD, we propose a novel mamba2-based multi-stage feature fusion model, termed ESIQAnet, which predicts the perceptual quality of egocentric spatial images under the three display modes. Specifically, we first extract features from multiple visual state space duality (VSSD) blocks, then apply cross attention to fuse binocular view information and use transposed attention to further refine the features. The multi-stage features are finally concatenated and fed into a quality regression network to predict the quality score. Extensive experimental results demonstrate that the ESIQAnet outperforms 22 state-of-the-art IQA models on the ESIQAD under all three display modes. The database and code are available at https://github.com/IntMeGroup/ESIQA.

Problem

• Assessing quality of egocentric spatial images.
• Developing a database for quality assessment.
• Proposing a model for perceptual quality prediction.

Innovation

• Mamba2-based multi-stage fusion
• Cross attention for binocular views
• Transposed attention feature refinement