How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures

2026-08-13

AI Summary

This work addresses a critical gap in the evaluation of vision-language models (VLMs), which typically emphasize perceptual and reasoning accuracy while neglecting behavioral reliability under missing or misleading visual evidence. To this end, we introduce SciFigBench—a challenging diagnostic benchmark for scientific figure understanding comprising over 34,000 test samples—designed with image perturbations, adversarial probes, and selective blurring, complemented by human-annotated labels and multidimensional metrics including MQM scores and reasoning accuracy. We further propose the A-R-I framework to systematically assess whether models Acknowledge insufficient evidence, Resist misleading cues, and Infer cautiously under uncertainty. Empirical results reveal that while GPT-5.2 achieves high accuracy, it frequently hallucinates; in contrast, Gemini 3.1 Pro demonstrates comparable performance with markedly higher reliability, explicitly acknowledging uncertainty in 71% of cases and attaining a resistance-to-misleading score of 0.91.

Abstract

Existing vision-language model (VLM) benchmarks emphasize perception and reasoning accuracy (how well VLMs describe and reason about what they see in an image), with limited attention to behavioral reliability under uncertainty (how they behave when visual evidence is missing or misleading). We introduce SciFigBench, a diagnostic VLM benchmark for scientific figure understanding that jointly evaluates perception, reasoning, and behavioral reliability under uncertainty. It contains 250 figures with high-quality human annotations across three evaluation aspects, totaling 600+ hours of annotation effort. We further extend these figures via image transformations, reasoning questions, resistance probes, caption-bias probes, and confirmed selective-blur targets, producing over 34,000 evaluation setups for stress testing. We further propose the Admittance-Resistance-Inductance (A-R-I) framework to evaluate whether models acknowledge insufficient evidence, resist misleading context, and infer cautiously from partial information. Our results reveal substantial behavioral differences among models. GPT-5.2 achieves the highest description quality (MQM 91.6) with strong reasoning accuracy (78.4%), yet hallucinates unreadable content in 96% of cases, whereas Gemini 3.1 Pro, a comparably capable model (MQM 90.2, reasoning 81.0%), admits uncertainty in 71% of such cases and achieves the strongest resistance score (0.91). These findings show that high perception and reasoning accuracy alone do not guarantee behavioral reliability, a dimension critical for deployment in scientific workflows.

Problem

• vision-language models
• behavioral reliability
• scientific figures
• uncertainty
• hallucination

Innovation

• behavioral reliability
• vision-language models
• scientific figure understanding
• uncertainty evaluation
• A-R-I framework