HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?

2025-04-25

AI Summary

Current vision-language models (VLMs) lack a systematic, standardized benchmark for evaluating high-resolution image (HRI) understanding—critical for domains such as digital pathology and aerial imaging. Method: We introduce HRScene, the first unified benchmark for HRI understanding, encompassing 25 real-world scene categories and 2 synthetic diagnostic datasets, with images up to 35,503×26,627 pixels. We propose a novel multi-granularity evaluation framework and, for the first time, formalize the “region utilization effectiveness” metric to diagnose spatial reasoning deficits in VLMs. Contribution/Results: Large-scale evaluation across 28 state-of-the-art VLMs—including Gemini 2.0 Flash and GPT-4o—reveals pervasive “region divergence” and “mid-level feature loss” phenomena, with average accuracy dropping to ~50% on real-world tasks and pronounced region utilization failures on synthetic diagnostic tasks. HRScene provides a reproducible benchmark, diagnostic toolkit, and concrete pathways for advancing HRI-aware VLM development.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

High-resolution image (HRI) understanding aims to process images with a large number of pixels, such as pathological images and agricultural aerial images, both of which can exceed 1 million pixels. Vision Large Language Models (VLMs) can allegedly handle HRIs, however, there is a lack of a comprehensive benchmark for VLMs to evaluate HRI understanding. To address this gap, we introduce HRScene, a novel unified benchmark for HRI understanding with rich scenes. HRScene incorporates 25 real-world datasets and 2 synthetic diagnostic datasets with resolutions ranging from 1,024 $ imes$ 1,024 to 35,503 $ imes$ 26,627. HRScene is collected and re-annotated by 10 graduate-level annotators, covering 25 scenarios, ranging from microscopic to radiology images, street views, long-range pictures, and telescope images. It includes HRIs of real-world objects, scanned documents, and composite multi-image. The two diagnostic evaluation datasets are synthesized by combining the target image with the gold answer and distracting images in different orders, assessing how well models utilize regions in HRI. We conduct extensive experiments involving 28 VLMs, including Gemini 2.0 Flash and GPT-4o. Experiments on HRScene show that current VLMs achieve an average accuracy of around 50% on real-world tasks, revealing significant gaps in HRI understanding. Results on synthetic datasets reveal that VLMs struggle to effectively utilize HRI regions, showing significant Regional Divergence and lost-in-middle, shedding light on future research.

Problem

• Lack of comprehensive benchmark for VLM high-resolution image evaluation
• Current VLMs show limited accuracy (50%) in real-world HRI tasks
• VLMs struggle with effective region utilization in high-resolution images

Innovation

• Introduces HRScene benchmark for HRI understanding
• Combines 25 real-world and 2 synthetic datasets
• Evaluates 28 VLMs including Gemini 2.0 and GPT-4o