OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling

2024-07-13

International Conference on Learning Representations

AI Summary

Existing OPT benchmarks exclusively cover linear programming, failing to assess large language models’ (LLMs) capability in end-to-end optimization modeling and solving for real-world tasks. Method: We introduce OptiBench—the first comprehensive, end-to-end optimization modeling benchmark—spanning linear programming, nonlinear programming, and tabular-data-driven optimization scenarios, requiring models to invoke solvers and output precise numerical solutions. To generate high-quality training data, we propose ReSocratic, a novel data synthesis paradigm comprising mathematical derivation, structured demonstration formatting, and problem back-translation. Our training framework integrates code-augmented reasoning, supervised fine-tuning (SFT), and multi-model adaptation (e.g., Llama-3-8B). Contribution/Results: We publicly release OptiBench and the ReSocratic-29K dataset. After fine-tuning, open-source models achieve performance on par with GPT-4, closing the capability gap between open- and closed-source LLMs on optimization modeling for the first time.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Optimization
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning. Solving realistic optimization (OPT) problems in application scenarios requires advanced and applied mathematics ability. However, current OPT benchmarks that merely solve linear programming are far from complex realistic situations. In this work, we propose OptiBench, a benchmark for End-to-end optimization problem-solving with human-readable inputs and outputs. OptiBench contains rich optimization problems, including linear and nonlinear programming with or without tabular data, which can comprehensively evaluate LLMs' solving ability. In our benchmark, LLMs are required to call a code solver to provide precise numerical answers. Furthermore, to alleviate the data scarcity for optimization problems, and to bridge the gap between open-source LLMs on a small scale (e.g., Llama-3-8b) and closed-source LLMs (e.g., GPT-4), we further propose a data synthesis method namely ReSocratic. Unlike general data synthesis methods that proceed from questions to answers, ReSocratic first incrementally synthesizes formatted optimization demonstration with mathematical formulations step by step and then back-translates the generated demonstrations into questions. Based on this, we synthesize the ReSocratic-29k dataset. We further conduct supervised fine-tuning with ReSocratic-29k on multiple open-source models. Experimental results show that ReSocratic-29k significantly improves the performance of open-source models.

Problem

• Measure LLMs' ability to solve complex optimization problems
• Address scarcity of diverse optimization benchmark datasets
• Bridge performance gap between open and closed-source LLMs

Innovation

• OptiBench benchmark for end-to-end optimization problem-solving
• ReSocratic method synthesizes optimization demonstrations step-by-step
• ReSocratic-29k dataset enhances open-source LLMs performance