GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models

2024-12-30

AI Summary

Existing multimodal large language models (MLLMs) lack rigorous evaluation of fundamental geometric perception capabilities—such as shape understanding, spatial relation reasoning, and abstraction of visual patterns. Method: We introduce GePBench, the first benchmark dedicated to low-level geometric perception. It formally defines and quantifies geometric perception in MLLMs and proposes a structured, generalizable evaluation framework. Leveraging synthetically generated geometric figures and logical constraints, GePBench produces diverse test samples covering multi-granularity tasks—including collinearity judgment, symmetry detection, and topological reasoning. Results: Extensive experiments reveal significant deficiencies in state-of-the-art MLLMs on these geometric tasks. Fine-tuning models on GePBench yields an average accuracy improvement of 4.2% on downstream visual reasoning and chart comprehension tasks, demonstrating the benchmark’s validity and cross-task generalization value.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Multimodal large language models (MLLMs) have achieved significant advancements in integrating visual and linguistic understanding. While existing benchmarks evaluate these models in context-rich, real-life scenarios, they often overlook fundamental perceptual skills essential for environments deviating from everyday realism. In particular, geometric perception, the ability to interpret spatial relationships and abstract visual patterns, remains underexplored. To address this limitation, we introduce GePBench, a novel benchmark designed to assess the geometric perception capabilities of MLLMs. Results from extensive evaluations reveal that current state-of-the-art MLLMs exhibit significant deficiencies in such tasks. Additionally, we demonstrate that models trained with data sourced from GePBench show notable improvements on a wide range of downstream tasks, underscoring the importance of geometric perception as a foundation for advanced multimodal applications. Our code and datasets will be publicly available.

Problem

• Multimodal Large Language Models
• Geometric Understanding
• Shape and Spatial Perception

Innovation

• GePBench
• Multi-modal Language Models
• Geometric Understanding