BareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs

2026-04-12

AI Summary

This work addresses the open question of whether current vision-language models (VLMs) genuinely comprehend geometric structure or merely rely on RGB textures and semantic priors, for which effective evaluation methods have been lacking. To this end, we introduce BareBones, the first zero-shot benchmark dedicated exclusively to pure geometric understanding. Built upon six datasets—including a newly curated fine-grained WTP-Bench—it generates pixel-level contour images that systematically strip away texture and semantic cues. We employ this benchmark to rigorously evaluate the geometric perception capabilities of 26 prominent VLMs. Our experiments reveal a dramatic performance drop across all models under texture-free conditions, exposing a pervasive “texture bias cliff.” BareBones thus establishes a stringent, quantifiable standard for assessing geometric understanding in vision-language systems.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

While Vision-Language Models (VLMs) demonstrate remarkable zero-shot recognition capabilities across a diverse spectrum of multimodal tasks, it yet remains an open question whether these architectures genuinely comprehend geometric structure or merely exploit RGB textures and contextual priors as statistical shortcuts. Existing evaluations fail to isolate this mechanism, conflating semantic reasoning with texture mapping and relying on imprecise annotations that inadvertently leak environmental cues. To address this gap, we introduce \textbf{BareBones}, a zero-shot benchmark designed to stress-test pure geometric shape comprehension. We curate pixel-level silhouettes of geometrically distinct classes across six datasets: five established segmentation sources (ImageNet-S, DIS5K, ThinObject5K, PASCAL VOC, CUB-200) and our novel flagship collection, WTP-Bench, establishing a noise-free geometric taxonomy. WTP-Bench is an extreme, fine-grained visual puzzle that forces models to identify inter-class geometric concepts from boundary contours alone. Our evaluation of 26 state-of-the-art proprietary and open-weight VLMs (\eg, GPT-4.1, Gemini, Claude Sonnet 4.5, LLaVA) reveals a consistent, severe performance collapse under RGB deprivation, a phenomenon we term the \textit{Texture Bias Cliff}. By documenting universal structural blindspots, BareBones establishes a rigorous yardstick for genuine geometric grounding.

Problem

• geometric comprehension
• vision-language models
• zero-shot learning
• texture bias
• shape understanding

Innovation

• geometric comprehension
• zero-shot benchmark
• texture bias
• silhouette-based evaluation
• vision-language models