V$^2$R-Bench: Holistically Evaluating LVLM Robustness to Fundamental Visual Variations

2025-04-23

AI Summary

This work addresses the insufficient robustness of Large Vision-Language Models (LVLMs) to fundamental visual variations—such as position, scale, orientation, and contextual shifts—in natural scenes, by introducing the first systematic benchmark for evaluating such robustness. Methodologically, it proposes automated synthetic data generation, multi-granularity robustness metrics, component-level error attribution analysis, and cross-modal feature alignment visualization. Key contributions include: (i) the first empirical revelation of pronounced positional bias in LVLMs and identification of human-like visual acuity thresholds; and (ii) an interpretable error propagation diagnostic framework. Empirical evaluation across 21 state-of-the-art LVLMs demonstrates severely limited robustness even on basic recognition tasks. The primary bottlenecks are architectural-level error accumulation and deficiencies in cross-modal alignment—not data scarcity. These findings highlight critical limitations in current LVLM design and provide actionable insights for improving visual grounding and multimodal integration.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Large Vision Language Models (LVLMs) excel in various vision-language tasks. Yet, their robustness to visual variations in position, scale, orientation, and context that objects in natural scenes inevitably exhibit due to changes in viewpoint and environment remains largely underexplored. To bridge this gap, we introduce V$^2$R-Bench, a comprehensive benchmark framework for evaluating Visual Variation Robustness of LVLMs, which encompasses automated evaluation dataset generation and principled metrics for thorough robustness assessment. Through extensive evaluation on 21 LVLMs, we reveal a surprising vulnerability to visual variations, in which even advanced models that excel at complex vision-language tasks significantly underperform on simple tasks such as object recognition. Interestingly, these models exhibit a distinct visual position bias that contradicts theories of effective receptive fields, and demonstrate a human-like visual acuity threshold. To identify the source of these vulnerabilities, we present a systematic framework for component-level analysis, featuring a novel visualization approach for aligned visual features. Results show that these vulnerabilities stem from error accumulation in the pipeline architecture and inadequate multimodal alignment. Complementary experiments with synthetic data further demonstrate that these limitations are fundamentally architectural deficiencies, scoring the need for architectural innovations in future LVLM designs.

Problem

• Evaluating LVLM robustness to visual variations like position, scale, orientation.
• Identifying vulnerabilities in LVLMs despite their complex task performance.
• Analyzing error sources in LVLM pipeline architecture and alignment.

Innovation

• Automated evaluation dataset generation for LVLMs
• Component-level analysis with novel visualization approach
• Synthetic data experiments revealing architectural deficiencies