Behavior and Representation in Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection

2025-12-15

AI Summary

This work investigates the implicit structural representation capability of large language models (LLMs) for combinatorial optimization problems and its utility for downstream decision-making tasks—specifically solver selection. Methodologically, we systematically analyze how hidden layers of LLMs encode problem structure across four benchmark optimization problems and three instance encodings, employing both direct prompting and neuron probing techniques. Our key contributions are threefold: First, we provide the first empirical evidence that intermediate LLM layers capture optimization problem structures highly aligned with classical hand-crafted features. Second, these implicit representations achieve solver recommendation accuracy on par with traditional feature-engineering approaches at the instance level. Third, the representations exhibit strong cross-problem generalization robustness. Collectively, these findings offer new insights into the internal mechanisms of LLMs in symbolic reasoning tasks and advance their trustworthy deployment in operations research and optimization.

Technology Category

• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Solvers and Tools
• Machine Learning: Optimization

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advances in Large Language Models (LLMs) have opened new perspectives for automation in optimization. While several studies have explored how LLMs can generate or solve optimization models, far less is understood about what these models actually learn regarding problem structure or algorithmic behavior. This study investigates how LLMs internally represent combinatorial optimization problems and whether such representations can support downstream decision tasks. We adopt a twofold methodology combining direct querying, which assesses LLM capacity to explicitly extract instance features, with probing analyses that examine whether such information is implicitly encoded within their hidden layers. The probing framework is further extended to a per-instance algorithm selection task, evaluating whether LLM-derived representations can predict the best-performing solver. Experiments span four benchmark problems and three instance representations. Results show that LLMs exhibit moderate ability to recover feature information from problem instances, either through direct querying or probing. Notably, the predictive power of LLM hidden-layer representations proves comparable to that achieved through traditional feature extraction, suggesting that LLMs capture meaningful structural information relevant to optimization performance.

Problem

• Investigates LLM internal representation of combinatorial optimization problems
• Assesses LLM ability to extract features from problem instances
• Evaluates if LLM representations can predict best-performing solver

Innovation

• Probing hidden layers for implicit problem representations
• Extending probing to per-instance algorithm selection
• Comparing LLM-derived representations with traditional feature extraction