LADBench: A Benchmark for Logical Fault Detection in Images

2026-06-15

AI Summary

This study addresses the limited capability of current vision-language models in physical and social commonsense logical reasoning, as well as the absence of dedicated evaluation benchmarks. To this end, we introduce LADBench, the first systematically constructed multi-domain synthetic image benchmark comprising over 1,000 samples across four real-world scenarios: residential, urban, collaborative, and natural environments. We further propose a hierarchical prompting protocol based on progressive disclosure to evaluate models’ ability to detect logical anomalies under varying levels of prompt guidance. Experimental results reveal that even state-of-the-art models achieve a peak accuracy of only 70.11% and frequently fail to identify anomalies under deeper prompting, highlighting significant deficiencies in sequential multimodal logical reasoning. This work establishes a new benchmark and evaluation paradigm for logical anomaly detection.

Abstract

Large Vision Language Models (VLMs) excel at visual question answering and semantic grounding, but their capacity for autonomous logical reasoning remains underexplored. Existing anomaly benchmarks emphasize visual errors or direct prompting rather than the physical and social common sense needed for open-world deployment. To address this, we introduce LAD-bench, a benchmark of more than 1,000 curated synthetic images with logical anomalies across four domains: Residential, Urban, Collaborative, and Nature. We further propose a Tiered Prompting Protocol based on progressive disclosure, which measures how much explicit assistance a model needs to localize and reason about a logical fault. Evaluating leading foundation models reveals substantial weaknesses: even the best achieves only 70.11% overall accuracy, showing that implicit logical fault detection remains unsolved. Crucially, models often fail to identify anomalies even after receiving explicit hints in deeper tiers. By surfacing these limitations in sequential multimodal reasoning, LAD-Bench offers a rigorous framework for advancing the safety, reliability, and cognitive alignment of autonomous visual systems. Dataset and Code: https://huggingface.co/datasets/SahasraK/LADBench

Problem

• logical fault detection
• vision language models
• anomaly benchmark
• common sense reasoning
• multimodal reasoning

Innovation

• Logical Fault Detection
• Vision-Language Models
• Benchmark
• Tiered Prompting Protocol
• Multimodal Reasoning