ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling

2024-05-28

AI Summary

To address high costs, weak data privacy, and poor interpretability arising from reliance on proprietary large language models (LLMs) in automated operations research (OR) optimization modeling, this paper introduces ORLM—the first high-quality open-source LLM specifically designed for optimization modeling and solver code generation. Methodologically, we develop OR-Instruct, a semi-automatic data synthesis framework integrating domain-specific knowledge injection, instruction tuning, synthetic data augmentation, reinforcement learning, and scaling law analysis, enabling end-to-end training atop the Llama-7B architecture. We further introduce two novel contributions: IndustryOR, a realistic industrial benchmark, and NL4OPT/MAMO, a unified evaluation framework for natural-language-to-optimization tasks. Experiments demonstrate that ORLM achieves state-of-the-art performance across all three benchmarks, supports customizable modeling across diverse application scenarios, and significantly enhances modeling efficiency, data privacy preservation, and industrial deployability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Optimization modeling plays a critical role in the application of Operations Research (OR) tools to address real-world problems, yet they pose challenges and require extensive expertise from OR experts. With the advent of large language models (LLMs), new opportunities have emerged to streamline and automate such task. However, current research predominantly relies on closed-source LLMs such as GPT-4, along with extensive prompt engineering techniques. This reliance stems from the scarcity of high-quality training datasets for optimization modeling, resulting in elevated costs, prolonged processing times, and privacy concerns. To address these challenges, our work is the first to propose a viable path for training open-source LLMs that are capable of optimization modeling and developing solver codes, eventually leading to a superior ability for automating optimization modeling and solving. Particularly, we introduce OR-Instruct, a semi-automated data synthesis framework for optimization modeling that enables customizable enhancements for specific scenarios or model types. We also introduce IndustryOR, the first industrial benchmark for evaluating LLMs in solving practical OR problems. We train several 7B-scale open-source LLMs using synthesized data (dubbed ORLMs{https://github.com/Cardinal-Operations/ORLM}), which exhibit significantly enhanced optimization modeling capabilities, achieving state-of-the-art performance across the NL4OPT, MAMO, and IndustryOR benchmarks. Additionally, our experiments highlight the potential of scaling law and reinforcement learning to further enhance the performance of ORLMs. The workflows and human-machine interaction paradigms of ORLMs in practical industrial applications are also discussed in the paper.

Problem

• Open-Source Large Language Models
• Cost-Efficient Modeling
• Privacy Concerns

Innovation

• ORLM
• Synthetic Data
• IndustryOR Testing