Toward a Trustworthy Optimization Modeling Agent via Verifiable Synthetic Data Generation

2025-08-05

AI Summary

Bridging natural language to optimization modeling—specifically linear programming (LP) and mixed-integer linear programming (MILP)—remains challenging due to accuracy, verifiability, and trustworthiness limitations in LLM-based approaches. Method: We propose a fully verifiable end-to-end data generation framework: (1) programmatically constructing problem instances with known optimal solutions; (2) leveraging teacher models for candidate generation coupled with automated filtering to ensure high-quality, traceable training data; and (3) adopting symbolic problem representation to jointly generate natural language descriptions, mathematical formulations, and multilingual solver code, enhanced by supervised fine-tuning, multi-stage translation, multilingual reasoning, and majority-voting cross-validation. Contribution/Results: Evaluated on seven standard benchmarks, our method achieves state-of-the-art (SOTA) accuracy on six datasets—including three where it surpasses the second-best approach by ≥8 percentage points—demonstrating substantial improvements in modeling fidelity and reliability.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present a framework for training trustworthy large language model (LLM) agents for optimization modeling via a verifiable synthetic data generation pipeline. Focusing on linear and mixed-integer linear programming, our approach begins with structured symbolic representations and systematically produces natural language descriptions, mathematical formulations, and solver-executable code. By programmatically constructing each instance with known optimal solutions, the pipeline ensures full verifiability and enables automatic filtering of low-quality demonstrations generated by teacher models. Each dataset instance includes a structured representation of the optimization problem, a corresponding natural language description, the verified optimal solution, and step-by-step demonstrations - generated by a teacher model - that show how to model and solve the problem across multiple optimization modeling languages. This enables supervised fine-tuning of open-source LLMs specifically tailored to optimization tasks. To operationalize this pipeline, we introduce OptiTrust, a modular LLM agent that performs multi-stage translation from natural language to solver-ready code, leveraging stepwise demonstrations, multi-language inference, and majority-vote cross-validation. Our agent achieves state-of-the-art performance on standard benchmarks. Out of 7 datasets, it achieves the highest accuracy on six and outperforms the next-best algorithm by at least 8 percentage on three of them. Our approach provides a scalable, verifiable, and principled path toward building reliable LLM agents for real-world optimization applications.

Problem

• Training trustworthy LLM agents for optimization modeling
• Generating verifiable synthetic data for optimization tasks
• Achieving state-of-the-art performance in optimization benchmarks

Innovation

• Verifiable synthetic data generation pipeline
• Modular LLM agent with multi-stage translation
• Supervised fine-tuning with structured representations