Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing

2025-04-03

AI Summary

Current large multimodal models struggle with understanding complex instructions, preserving appearance consistency, and adapting to diverse input formats in general-purpose visual editing. To address this, we introduce RISEBench—the first benchmark explicitly designed for reasoning-enhanced visual editing—systematically covering temporal, causal, spatial, and logical reasoning capabilities. It evaluates models across three dimensions: instruction understanding, appearance consistency, and visual plausibility. We propose a novel human + LMM dual-track evaluation framework, integrating structured test cases and multidimensional metrics. Experiments reveal that even GPT-4o-Native—the top-performing model overall—achieves less than 40% accuracy on logical reasoning–driven editing tasks. RISEBench is publicly released to foster research on interpretable, reasoning-capable visual editing, establishing a new methodological paradigm for the field.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Multi-modality Models (LMMs) have made significant progress in visual understanding and generation, but they still face challenges in General Visual Editing, particularly in following complex instructions, preserving appearance consistency, and supporting flexible input formats. To address this gap, we introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE). RISEBench focuses on four key reasoning types: Temporal, Causal, Spatial, and Logical Reasoning. We curate high-quality test cases for each category and propose an evaluation framework that assesses Instruction Reasoning, Appearance Consistency, and Visual Plausibility with both human judges and an LMM-as-a-judge approach. Our experiments reveal that while GPT-4o-Native significantly outperforms other open-source and proprietary models, even this state-of-the-art system struggles with logical reasoning tasks, highlighting an area that remains underexplored. As an initial effort, RISEBench aims to provide foundational insights into reasoning-aware visual editing and to catalyze future research. Though still in its early stages, we are committed to continuously expanding and refining the benchmark to support more comprehensive, reliable, and scalable evaluations of next-generation multimodal systems. Our code and data will be released at https://github.com/PhoenixZ810/RISEBench.

Problem

• Evaluating Reasoning-Informed Visual Editing (RISE) in Large Multi-modality Models (LMMs)
• Assessing four key reasoning types: Temporal, Causal, Spatial, and Logical Reasoning
• Addressing challenges in complex instruction following, appearance consistency, and flexible input formats

Innovation

• Introduces RISEBench benchmark for visual editing
• Evaluates four key reasoning types in editing
• Uses human and LMM-as-a-judge evaluation