Decoupling the components of geometric understanding in Vision Language Models

2025-03-05

AI Summary

Visual language models (VLMs) exhibit ambiguous geometric reasoning capabilities, confounded by entanglements of low-level vision, high-order reasoning, and world knowledge. Method: We systematically evaluate VLMs’ understanding of foundational geometric concepts—parallelism, symmetry, and mental rotation—using a cognitively grounded, stimulus-controlled benchmark that isolates geometric cognition from visual recognition and linguistic priors. Our novel geometric test set compares performance across U.S. adults and Amazonian Indigenous participants with no formal schooling, and evaluates leading VLMs—including LLaVA and Qwen-VL—in zero-shot and few-shot settings. Contribution/Results: This work presents the first fine-grained, component-wise decomposition of VLM geometric competence. Results reveal severe reliance on textual training data, lacking the embodied geometric representations humans acquire through physical interaction or education. VLMs underperform both human groups across all tasks—e.g., mental rotation accuracy is >40% lower—and demonstrate only limited, brittle recognition of static properties like parallelism, with poor generalization and robustness.

Technology Category

• Computer Vision: Large Vision Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Understanding geometry relies heavily on vision. In this work, we evaluate whether state-of-the-art vision language models (VLMs) can understand simple geometric concepts. We use a paradigm from cognitive science that isolates visual understanding of simple geometry from the many other capabilities it is often conflated with such as reasoning and world knowledge. We compare model performance with human adults from the USA, as well as with prior research on human adults without formal education from an Amazonian indigenous group. We find that VLMs consistently underperform both groups of human adults, although they succeed with some concepts more than others. We also find that VLM geometric understanding is more brittle than human understanding, and is not robust when tasks require mental rotation. This work highlights interesting differences in the origin of geometric understanding in humans and machines -- e.g. from printed materials used in formal education vs. interactions with the physical world or a combination of the two -- and a small step toward understanding these differences.

Problem

• Evaluate VLMs' ability to understand simple geometric concepts.
• Compare VLM performance with human adults in geometric tasks.
• Highlight differences in geometric understanding between humans and VLMs.

Innovation

• Cognitive science paradigm isolates visual understanding
• Compares VLM performance with human adults
• Highlights differences in geometric understanding origins