Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning

2025-10-14

AI Summary

Existing multimodal large language models (MLLMs) are predominantly evaluated under a “thinking about images” paradigm, treating images as static inputs—thus failing to support dynamic visual understanding and tool-augmented reasoning required in real-world scenarios involving active image manipulation (e.g., cropping, editing, enhancement). Method: We introduce IRIS, the first systematic benchmark for the “thinking with images” paradigm, featuring 1,204 multi-turn, cross-domain, open-ended tasks. IRIS formalizes the image as an interactive cognitive workspace and jointly evaluates MLLMs alongside general-purpose visual tools across perception, transformation, and reasoning stages. Contribution/Results: Experiments reveal that even the state-of-the-art model (GPT-5-think) achieves only 18.68% task success, exposing fundamental limitations in tool-call consistency and multi-step visual planning. IRIS establishes a critical evaluation foundation and actionable direction for next-generation embodied visual intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Multimodal Large Language Models (MLLMs) are increasingly applied in real-world scenarios where user-provided images are often imperfect, requiring active image manipulations such as cropping, editing, or enhancement to uncover salient visual cues. Beyond static visual perception, MLLMs must also think with images: dynamically transforming visual content and integrating it with other tools to solve complex tasks. However, this shift from treating vision as passive context to a manipulable cognitive workspace remains underexplored. Most existing benchmarks still follow a think about images paradigm, where images are regarded as static inputs. To address this gap, we introduce IRIS, an Interactive Reasoning with Images and Systems that evaluates MLLMs'ability to perceive, transform, and reason across complex visual-textual tasks under the think with images paradigm. IRIS comprises 1,204 challenging, open-ended vision tasks (603 single-turn, 601 multi-turn) spanning across five diverse domains, each paired with detailed rubrics to enable systematic evaluation. Our evaluation shows that current MLLMs struggle with tasks requiring effective integration of vision and general-purpose tools. Even the strongest model (GPT-5-think) reaches only 18.68% pass rate. We further observe divergent tool-use behaviors, with OpenAI models benefiting from diverse image manipulations while Gemini-2.5-pro shows no improvement. By introducing the first benchmark centered on think with images, IRIS offers critical insights for advancing visual intelligence in MLLMs.

Problem

• Evaluating MLLMs' ability to perceive, transform, and reason with images
• Assessing integration of vision and tools for complex visual-textual tasks
• Addressing limitations in dynamic image manipulation and cognitive reasoning

Innovation

• Introduces IRIS benchmark for tool-enabled image reasoning
• Evaluates MLLMs on dynamic image transformation and manipulation
• Tests integration of vision with general-purpose tools