A Comprehensive Evaluation of Contemporary ML-Based Solvers for Combinatorial Optimization

2025-05-22

AI Summary

Existing combinatorial optimization (CO) benchmarks predominantly rely on small-scale synthetic data, lacking systematic evaluation of machine learning (ML) solvers’ effectiveness and scalability on real-world, large-scale industrial instances, and suffer from severe training data scarcity. To address this, we introduce FrontierCO—the first unified benchmark covering eight classical CO problem classes, featuring industrially relevant instance scales and abundant high-quality labeled training data. We conduct the first comprehensive empirical evaluation of 16 state-of-the-art ML solvers—including graph neural networks (GNNs) and large language model (LLM)-based agents—within a unified end-to-end modeling, training, and evaluation framework. Our experiments uncover critical bottlenecks in cross-scale generalization, computational scalability, and practical deployability of current approaches. All benchmark data, source code, and trained models are publicly released on Hugging Face to ensure full reproducibility and community accessibility.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Machine Learning: Graph-based Machine Learning
• Search and Optimization: Combinatorial Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Machine learning (ML) has demonstrated considerable potential in supporting model design and optimization for combinatorial optimization (CO) problems. However, much of the progress to date has been evaluated on small-scale, synthetic datasets, raising concerns about the practical effectiveness of ML-based solvers in real-world, large-scale CO scenarios. Additionally, many existing CO benchmarks lack sufficient training data, limiting their utility for evaluating data-driven approaches. To address these limitations, we introduce FrontierCO, a comprehensive benchmark that covers eight canonical CO problem types and evaluates 16 representative ML-based solvers--including graph neural networks and large language model (LLM) agents. FrontierCO features challenging instances drawn from industrial applications and frontier CO research, offering both realistic problem difficulty and abundant training data. Our empirical results provide critical insights into the strengths and limitations of current ML methods, helping to guide more robust and practically relevant advances at the intersection of machine learning and combinatorial optimization. Our data is available at https://huggingface.co/datasets/CO-Bench/FrontierCO.

Problem

• Evaluating ML-based solvers for real-world combinatorial optimization problems
• Addressing lack of training data in existing CO benchmarks
• Assessing practical effectiveness of ML methods in large-scale CO scenarios

Innovation

• Introduces FrontierCO benchmark for CO problems
• Evaluates 16 ML-based solvers including GNNs and LLMs
• Provides industrial-scale data for realistic evaluation