GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training

2024-12-16

AI Summary

To address key challenges in geometric problem solving (GPS) with multimodal large models—namely, poor comprehension of diagrams and symbolic representations, lack of self-verification capability, and limited generalization of existing geometry-specialized models—this paper proposes GS-Former, an end-to-end verifiable vision-language unified modeling framework. GS-Former introduces geometric diagram-only pretraining, a symbol–language alignment paradigm, and a generation-sampling Transformer architecture, enabling visual instruction tuning and automatic answer verification. Evaluated on four major benchmarks—GeoQA, UniGeo, Geometry3K, and PGPS9k—the method consistently outperforms both general-purpose multimodal models and geometry-specific baselines. It achieves significant improvements in cross-problem-type generalization and reasoning reliability, demonstrating robustness across diverse geometric reasoning tasks.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Despite their proficiency in general tasks, Multi-modal Large Language Models (MLLMs) struggle with automatic Geometry Problem Solving (GPS), which demands understanding diagrams, interpreting symbols, and performing complex reasoning. This limitation arises from their pre-training on natural images and texts, along with the lack of automated verification in the problem-solving process. Besides, current geometric specialists are limited by their task-specific designs, making them less effective for broader geometric problems. To this end, we present GeoX, a multi-modal large model focusing on geometric understanding and reasoning tasks. Given the significant differences between geometric diagram-symbol and natural image-text, we introduce unimodal pre-training to develop a diagram encoder and symbol decoder, enhancing the understanding of geometric images and corpora. Furthermore, we introduce geometry-language alignment, an effective pre-training paradigm that bridges the modality gap between unimodal geometric experts. We propose a Generator-And-Sampler Transformer (GS-Former) to generate discriminative queries and eliminate uninformative representations from unevenly distributed geometric signals. Finally, GeoX benefits from visual instruction tuning, empowering it to take geometric images and questions as input and generate verifiable solutions. Experiments show that GeoX outperforms both generalists and geometric specialists on publicly recognized benchmarks, such as GeoQA, UniGeo, Geometry3K, and PGPS9k.

Problem

• Multimodal Large Language Models
• Geometric Problems Solving
• Expert Model Limitations

Innovation

• GeoX
• Multi-modal Learning
• Geometric Problem Solving