MATP-BENCH: Can MLLM Be a Good Automated Theorem Prover for Multimodal Problems?

2025-06-06

AI Summary

This work investigates the feasibility of multimodal large language models (MLLMs) as multimodal automated theorem provers (ATPs), requiring joint reasoning over visual inputs (e.g., geometric diagrams), mathematical knowledge, and symbolic deduction to generate formal proofs. Method: We introduce MATP-BENCH, the first benchmark for multimodal theorem proving, comprising 1,056 visually grounded mathematical statements—each annotated with formal proofs in Lean 4, Coq, and Isabelle. We formally define the multimodal ATP task and propose a unified evaluation framework spanning multiple abstraction levels, formal languages, and modalities, ensuring cross-assistant coverage. Contribution/Results: Experiments reveal that state-of-the-art MLLMs achieve negligible performance on MATP-BENCH, confirming its substantial difficulty. The benchmark establishes a rigorous, reproducible foundation for advancing interdisciplinary research at the intersection of multimodal AI and formal reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Numerous theorems, such as those in geometry, are often presented in multimodal forms (e.g., diagrams). Humans benefit from visual reasoning in such settings, using diagrams to gain intuition and guide the proof process. Modern Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in solving a wide range of mathematical problems. However, the potential of MLLMs as Automated Theorem Provers (ATPs), specifically in the multimodal domain, remains underexplored. In this paper, we introduce the Multimodal Automated Theorem Proving benchmark (MATP-BENCH), a new Multimodal, Multi-level, and Multi-language benchmark designed to evaluate MLLMs in this role as multimodal automated theorem provers. MATP-BENCH consists of 1056 multimodal theorems drawn from high school, university, and competition-level mathematics. All these multimodal problems are accompanied by formalizations in Lean 4, Coq and Isabelle, thus making the benchmark compatible with a wide range of theorem-proving frameworks. MATP-BENCH requires models to integrate sophisticated visual understanding with mastery of a broad spectrum of mathematical knowledge and rigorous symbolic reasoning to generate formal proofs. We use MATP-BENCH to evaluate a variety of advanced multimodal language models. Existing methods can only solve a limited number of the MATP-BENCH problems, indicating that this benchmark poses an open challenge for research on automated theorem proving.

Problem

• Evaluating MLLMs as multimodal automated theorem provers
• Assessing visual and symbolic reasoning in theorem proving
• Benchmarking MLLMs on high-level multimodal math problems

Innovation

• MLLMs as multimodal automated theorem provers
• MATP-BENCH benchmark for multimodal theorem evaluation
• Integration of visual understanding with symbolic reasoning