A$^{2}$utoLPBench: An Auto-Generated, Agent-Friendly LP Benchmark via Inverse-KKT Construction

2026-07-02

AI Summary

Existing linear programming (LP) text benchmarks are predominantly static, handcrafted datasets that suffer from fixed scale, non-adjustable difficulty, and the risk of training data contamination. This work proposes a novel method for automatically generating LP instances based on inverse KKT conditions: by pre-specifying a feasible primal solution and dual variables, it constructs LP problems for which the given solution is provably optimal. This approach guarantees correctness by design, eliminating the need for manual annotation or solver-based verification. For the first time, it enables the creation of dynamic benchmarks that are infinitely scalable, difficulty-controllable, resistant to data leakage, and low-cost to produce. The authors also provide a Dockerized evaluation environment and baseline systems, allowing large language model (LLM) agents to be deployed and evaluated reproducibly with minimal setup.

Abstract

Most LP-from-text benchmarks are static datasets of word problems written and labeled by hand. Once such a dataset is released, its size is fixed, its difficulty is fixed, and every problem can leak into the training data of future LLMs. We present \textbf{A$^{2}$utoLPBench}, a benchmark for testing LLM-driven agents on linear programming problems written in plain text. We first pick a feasible point and dual, then write down a problem for which that point is optimal and the objective value is known. The answer is known by construction, with no solver call and no human annotator. The evaluation environment bundles a reference solver-critic baseline and a Docker image whose usage instructions are written for an LLM-driven agent to read. With these in place, any agent can run the benchmark and get a calibrated score with one command. Because the benchmark is a generator rather than a fixed dataset, it has properties no fixed dataset can match: an unlimited supply of fresh problems, a difficulty knob set by $(n,m)$, ground-truth answers correct by construction, low LLM-side cost per problem relative to human authoring, repeatable scores across independent batches, and resistance to training-data leakage when fresh post-cutoff seed ranges are used.

Problem

• linear programming
• benchmark
• LLM
• training-data leakage
• static dataset

Innovation

• Inverse-KKT
• Auto-generated benchmark
• LLM agent evaluation
• Linear programming
• Training-data leakage resistance