RAISE: Realness Assessment for Image Synthesis and Evaluation

2025-05-25

AI Summary

Reliable assessment of perceived realism in AI-generated images—critical for determining their viability as substitutes for scarce or costly real-world visual content—remains an open challenge. Method: We introduce RAISE, the first subjective benchmark dedicated to perceptual realism evaluation: (i) conducting large-scale human studies to collect fine-grained realism scores; (ii) systematically decomposing the multidimensional nature of subjective realism perception; (iii) releasing the first large-scale, cross-model generated image dataset with continuous realism annotations; and (iv) proposing realism prediction as a novel task paradigm. Leveraging feature transfer from foundational vision models (e.g., DINOv2, CLIP) coupled with regression modeling, we demonstrate efficient prediction of subjective realism. Contribution/Results: RAISE achieves strong correlation with human judgments (Spearman ρ > 0.78). We provide an open evaluation platform and robust baselines, advancing image quality assessment from discriminative classification toward perceptual alignment.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The rapid advancement of generative AI has enabled the creation of highly photorealistic visual content, offering practical substitutes for real images and videos in scenarios where acquiring real data is difficult or expensive. However, reliably substituting real visual content with AI-generated counterparts requires robust assessment of the perceived realness of AI-generated visual content, a challenging task due to its inherent subjective nature. To address this, we conducted a comprehensive human study evaluating the perceptual realness of both real and AI-generated images, resulting in a new dataset, containing images paired with subjective realness scores, introduced as RAISE in this paper. Further, we develop and train multiple models on RAISE to establish baselines for realness prediction. Our experimental results demonstrate that features derived from deep foundation vision models can effectively capture the subjective realness. RAISE thus provides a valuable resource for developing robust, objective models of perceptual realness assessment.

Problem

• Assessing perceived realness of AI-generated images
• Developing models for objective realness evaluation
• Creating dataset with realness scores for AI visuals

Innovation

• Human study evaluates perceptual realness of images
• Dataset RAISE pairs images with realness scores
• Deep vision models predict subjective realness effectively