An Extensive Benchmark for Single-round and Multi-round Instruction-based Image Editing

2026-06-13

AI Summary

This work addresses the lack of a unified evaluation framework for instruction-based image editing models, which struggle to handle both single-step and multi-turn editing tasks effectively. To bridge this gap, we introduce I2EBench2.0, the first systematic benchmark supporting 16 single-step and 7 multi-turn editing scenarios, comprehensively assessing editing accuracy, consistency, and multi-level quality dimensions. Through large-scale user studies aligned with human judgments, complemented by multi-dimensional manual evaluations and comparative model analyses, we conduct a thorough assessment of eight state-of-the-art models. The project releases code, datasets, and generated results to establish a standardized foundation for future research in this rapidly evolving field.

Abstract

In recent years, there have been notable advancements in the area of instruction-based image editing (IIE), which focuses on the automatic alteration of input images using a model. Nevertheless, assessing the effectiveness of these editing models poses a considerable challenge due to the intricate nature of instructions and the wide variety of edits. To tackle this problem, one urgent task in this domain is the development of a robust evaluation framework that can precisely gauge the quality of editing outcomes and offer valuable benchmarks to guide future improvements. To address this challenge, we present a comprehensive evaluation benchmark named I2EBench2.0, designed for single-round and multi-round assessment of IIE models. I2EBench2.0 has four key features: 1) Evaluation Across Single and Multi-rounds: I2EBench2.0 simultaneously evaluates both single-round and multi-round instruction-based edits, assessing the precision and consistency of the edits. 2) Extensive Evaluation Criteria: I2EBench2.0 encompasses a broad range of criteria, evaluating both high-level and low-level aspects of each IIE model. Specifically, it incorporates 16 dimensions for single-round evaluations and 7 for multi-round evaluations. 3) Alignment with Human Judgment: To ensure our benchmark aligns with human evaluation, we conducted a comprehensive user study for each criterion. 4) Research-driven Insights: By analyzing the strengths and weaknesses of current IIE models across all 16 single-round and 7 multi-round dimensions, we provide critical insights aimed at directing future research in this area. We tested eight recently developed IIE models using I2EBench2.0 and derived academic insights through meticulous comparison and analysis. The related code, dataset, and images generated by all IIE models are available on GitHub: https://github.com/cocoshe/I2EBench.

Problem

• instruction-based image editing
• evaluation benchmark
• single-round editing
• multi-round editing
• model assessment

Innovation

• instruction-based image editing
• evaluation benchmark
• multi-round editing
• human-aligned assessment
• I2EBench2.0