FORGE: Foundational Optimization Representations from Graph Embeddings

2025-08-27

AI Summary

Existing learning-based combinatorial optimization approaches require separate model training for each problem distribution, suffering from poor generalization and high computational overhead. This paper proposes the first unsupervised pre-training framework for mixed-integer programming (MIP), built upon a vector-quantized graph autoencoder (VQ-GAE) to learn transferable instance representations: MIP structures are encoded as discrete, reusable symbols in a shared codebook, enabling unified modeling across diverse problem types. The framework operates without reliance on solver trajectories or labeled data, and supports multiple downstream tasks—including variable warm-starting and cutting-plane generation. After fine-tuning on several MIP distributions, a single pre-trained Forge model significantly enhances the performance of commercial MIP solvers and demonstrates strong clustering capability on unseen instances.

Technology Category

• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Combinatorial optimization problems are ubiquitous in science and engineering, yet learning-based approaches to accelerate their solution often require solving a large number of hard-to-solve optimization instances to collect training data, incurring significant computational overhead. Existing methods require training dedicated models for each problem distribution for each downstream task, severely limiting their scalability and generalization. In this work, we introduce Forge, a method of pre-training a vector-quantized graph autoencoder on a large and diverse collection of mixed-integer programming (MIP) instances in an unsupervised fashion without dependency on their solution. The vector quantization process creates discrete code assignments that act as a vocabulary to represent optimization instances. We evaluate our approach under both supervised and unsupervised settings. For the unsupervised setting, we demonstrate that Forge embeddings effectively differentiate and cluster unseen instances. For the supervised setting, we fine-tune Forge embeddings and show that a single model predicts both the variables for warm-starts and integrality gaps for cut-generation across multiple problem type distributions. Both predictions help improve performance of a state-of-the-art, commercial optimization solver. Finally, we release our code and pre-trained Forge weights to encourage further research and practical use of instance-level MIP embeddings at https://github.com/skadio/forge/

Problem

• Pre-training graph embeddings for combinatorial optimization without solution dependency
• Enhancing solver performance via warm-start variables and cut-generation predictions
• Enabling generalization across diverse mixed-integer programming problem distributions

Innovation

• Unsupervised pre-training on diverse MIP instances
• Vector-quantized graph autoencoder for discrete representations
• Single model predicts warm-starts and integrality gaps