Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics

2026-07-07

AI Summary

This study addresses the widespread neglect of computer algebra systems (CAS) integration in current AI-for-math research. The authors propose a ReAct agent framework that synergistically combines large language models with SageMath, enhanced by Context7-powered real-time documentation retrieval and a multi-stage verification mechanism, to emulate authentic mathematical research workflows on a newly introduced RealMath benchmark. This work presents the first systematic evaluation of performance gains when mainstream models are augmented with CAS access: all models exhibit an average accuracy improvement of 9.7 percentage points (up to 27.8 pp), substantially narrowing the gap between open- and closed-source models. Notably, Qwen3.7-Max shows the most significant gain, while GPT-5.5 achieves the highest problem-solving rate at 75.2% with minimal token consumption, advancing automated mathematical conjecture discovery.

Abstract

Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS) in agentic LLM workflows underexplored. We propose a ReAct-style agentic setup that combines LLM reasoning with verifiable feedback from SageMath, together with Context7 for the up-to-date documentation. We evaluate this agentic setup across frontier models for solving research-level mathematical problems from the RealMath benchmark in a setting that emulates a computational-mathematics research loop. We also propose a refinement to the RealMath benchmark by introducing a multi-step post-processing procedure and a multi-stage validation pipeline, both of which improve the quality and reliability of the extracted problem set. Our experiments reveal substantial performance gains from SageMath access across all evaluated models on +9.7~pp on average, the gains range from 1.5~pp to 27.8~pp and narrow the gap between open-weight and closed models. Qwen~3.7-Max benefits from SageMath the most, while GPT-5.5 achieves the highest solve rate of $75.2\%$ and the lowest token usage among tool-enabled configurations. Our findings suggest that CAS-augmented agents represent a promising direction for assisting mathematicians in computational exploration, and we believe that this work is a step towards automated conjecture discovery. The project repository is available online.

Problem

• Computer Algebra Systems
• LLM Agents
• Computational Mathematics
• Experimental Mathematics
• SageMath

Innovation

• SageMath
• LLM agents
• Computer Algebra Systems
• RealMath benchmark
• ReAct framework