ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?

2026-06-18

AI Summary

Current large language model agents exhibit significant limitations in end-to-end execution of real-world operations research (OR) tasks, primarily due to the absence of comprehensive benchmarks evaluating the entire pipeline from raw input to verifiable decisions. This work proposes ORAgentBench, the first execution-driven benchmark for OR agents, comprising 107 diverse, human-curated tasks that require agents to interpret natural language instructions, process multi-file data inputs, and submit formalized solutions within an isolated environment. ORAgentBench enables, for the first time, end-to-end evaluation of the full workflow—including modeling, solving, and verification—and incorporates hidden validators to assess both solution feasibility and objective quality. Experimental results reveal that even state-of-the-art agents solve only 35.51% of tasks overall (20.59% on hard instances), with primary failure modes including constraint neglect, fragile modeling, and suboptimal solution quality.

Abstract

Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear. Existing OR evaluations often decouple modeling from solving, rely on pre-formalized or text-only instances, and rarely test the full workflow from operational artifacts to validated decisions. In this work, we introduce ORAgentBench, an execution-grounded benchmark for evaluating autonomous agents on challenging end-to-end operations research tasks. It contains 107 human-reviewed tasks across diverse operational scenarios, each packaged in an isolated environment with a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents must write and run solution code, and their submissions are evaluated by hidden validators for schema validity, hard-constraint feasibility, and normalized objective quality. Experiments with fourteen frontier agent-model configurations show that current agents remain far from reliable OR practice. The best agent passes only 35.51% of all tasks and 20.59% of hard tasks, and many feasible submissions still fall below the required quality threshold. Failure analysis further shows that errors are dominated by strategic weaknesses, including missed operational rules, brittle formulations, weak feasible-solution construction, and insufficient solution improvement. OR-specific procedural skills increase hard-task feasibility, but do not reliably improve solution quality or pass rate. These results suggest that progress in OR agents requires moving beyond plausible optimization code toward dependable, high-quality operational decision-making.

Problem

• Operations Research
• LLM Agents
• End-to-End Evaluation
• Autonomous Decision-Making
• Benchmarking

Innovation

• ORAgentBench
• end-to-end operations research
• execution-grounded benchmark
• autonomous LLM agents
• feasibility and objective quality evaluation