IE-Bench: Advancing the Measurement of Text-Driven Image Editing for Human Perception Alignment

2025-01-17

AI Summary

Text-guided image editing lacks human-perception-aligned evaluation methods. Method: This paper proposes a source-image-aware sensitivity paradigm for editing quality assessment, introduces IE-Bench—the first human-perception-aligned benchmark for this task—comprising 3,010 subjective ratings and diverse text-image editing samples, and releases IE-QA, the first dedicated image quality assessment (IQA) dataset and model. IE-QA integrates multimodal features, incorporates a source-image-aware attention mechanism, and is trained with Mean Opinion Score (MOS) as the supervisory signal. Contribution/Results: Experiments demonstrate that IE-QA significantly outperforms mainstream metrics (e.g., CLIPScore, DISTS) in subjective consistency and exhibits strong robustness and generalization across diverse text-to-image editing models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent advances in text-driven image editing have been significant, yet the task of accurately evaluating these edited images continues to pose a considerable challenge. Different from the assessment of text-driven image generation, text-driven image editing is characterized by simultaneously conditioning on both text and a source image. The edited images often retain an intrinsic connection to the original image, which dynamically change with the semantics of the text. However, previous methods tend to solely focus on text-image alignment or have not aligned with human perception. In this work, we introduce the Text-driven Image Editing Benchmark suite (IE-Bench) to enhance the assessment of text-driven edited images. IE-Bench includes a database contains diverse source images, various editing prompts and the corresponding results different editing methods, and total 3,010 Mean Opinion Scores (MOS) provided by 25 human subjects. Furthermore, we introduce IE-QA, a multi-modality source-aware quality assessment method for text-driven image editing. To the best of our knowledge, IE-Bench offers the first IQA dataset and model tailored for text-driven image editing. Extensive experiments demonstrate IE-QA's superior subjective-alignments on the text-driven image editing task compared with previous metrics. We will make all related data and code available to the public.

Problem

• Image Quality Assessment
• Text-guided Image Editing
• Human Perception

Innovation

• IE-Bench
• IE-QA
• Human Visual Perception