Pipeline for Verifying LLM-Generated Mathematical Solutions

2026-02-24

AI Summary

Current benchmarks for mathematical reasoning predominantly rely on answer matching, which fails to assess the logical correctness of solution processes. This work proposes a hybrid verification pipeline that integrates automated and interactive validation by leveraging structured prompting to guide large language models in generating verifiable solutions. The framework supports both formal and informal reasoning and interfaces with proof assistants such as Lean 4, enabling even small-scale models (≤8B parameters) to participate effectively in collaborative verification. Through a multi-agent architecture and advanced prompt engineering, the approach substantially reduces false positive rates. Experimental results demonstrate high verification accuracy across multiple datasets, and the codebase along with deployment guidelines has been publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

With the growing popularity of Large Reasoning Models and their results in solving mathematical problems, it becomes crucial to measure their capabilities. We introduce a pipeline for both automatic and interactive verification as a more accurate alternative to only checking the answer which is currently the most popular approach for benchmarks. The pipeline can also be used as a generator of correct solutions both in formal and informal languages. 3 AI agents, which can be chosen for the benchmark accordingly, are included in the structure. The key idea is the use of prompts to obtain the solution in the specific form which allows for easier verification using proof assistants and possible use of small models ($\le 8B$). Experiments on several datasets suggest low probability of False Positives. The open-source implementation with instructions on setting up a server is available at https://github.com/LogicEnj/lean4_verification_pipeline.

Problem

• Large Language Models
• Mathematical Reasoning
• Solution Verification
• Benchmarking
• False Positives

Innovation

• verification pipeline
• large reasoning models
• proof assistants
• prompt engineering
• formal mathematics