MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs

2026-05-08

AI Summary

Existing reasoning benchmarks for large language models often suffer from saturation due to fixed datasets or reliance on unreliable self-evaluation. This work proposes an adaptive evaluation framework grounded in constraint satisfaction problems, which dynamically generates reasoning instances through parameterized templates and a difficulty-control mechanism. These instances are automatically verifiable by SAT/SMT solvers and embedded within a sandboxed Python tool-calling environment, yielding a scalable benchmark resilient to model advancements. The released MathConstraint (329 instances) and MathConstraint-Easy (266 instances) datasets reveal a significant performance drop among state-of-the-art models on harder tasks, while appropriate tool usage improves average accuracy by 28 percentage points, thereby demonstrating the benchmark’s challenge and evaluative utility.

Abstract

We introduce MathConstraint, a hard, adaptive benchmark for evaluating the combinatorial reasoning capabilities of LLMs. We combine constraint satisfaction problems with rigorous solver-based verification and design an adaptive generator to create instances that remain challenging as the LLMs improve in their reasoning capabilities. Unlike existing benchmarks that quickly saturate on fixed datasets or use LLM-as-a-judge for checking solutions,MathConstraint uses parameterized problem types that enable scalable generation of arbitrarily difficult and automatically verifiable instances. We release MathConstraint-Easy ($266$ instances), on which frontier models achieve between $72.6\%$ (gemini-3.1-flash-lite) and $87.6\%$ (gpt-5.5) accuracy, and MathConstraint ($329$ instances) on which the same models drop to between $18.5\%$ (claude-4.6-sonnet) and $66.9\%$ (gpt-5.5) accuracy, demonstrating the resilience of our benchmark generator against rapid progress in LLM reasoning capabilities. We evaluate 12 frontier and open-weight models with and without access to a sandboxed Python environment that includes generic SAT/SMT solvers. Tool access roughly doubles frontier accuracy on MathConstraint (mean $+28$pp; up to $+52$pp for claude-4.6-sonnet). Further, halving the tool-call budget from $8$ to $4$ rounds erases up to $37$ points -- a sensitivity that most single-budget benchmarks miss. We release the generator, dataset, and evaluation harness as a robust environment for studying combinatorial reasoning and tool-use behavior under adversarially-tunable difficulty.

Problem

• combinatorial reasoning
• benchmark
• constraint satisfaction
• LLM evaluation
• scalable generation

Innovation

• constraint satisfaction problems
• adaptive benchmark generation
• solver-based verification
• combinatorial reasoning
• tool-augmented LLMs