Benchmarking Graph Neural Networks in Solving Hard Constraint Satisfaction Problems

2026-02-20

AI Summary

This work addresses the lack of standardized benchmarks for evaluating graph neural networks (GNNs) on hard constraint satisfaction problems (CSPs), which has led to unreliable performance assessments. Drawing from statistical physics, the authors introduce a new class of random hard instances that serve as a principled benchmark, enabling fair and rigorous comparisons between neural solvers and classical heuristic algorithms. Systematic evaluations on this benchmark reveal that state-of-the-art GNNs are still significantly outperformed by classical methods, highlighting fundamental limitations of current neural approaches in tackling complex CSPs. By establishing a reliable evaluation framework, this study not only fills a critical gap in standardized CSP benchmarking but also delineates clear challenges and directions for the future development of neural combinatorial solvers.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability
• Search and Optimization: Combinatorial Optimization
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Graph neural networks (GNNs) are increasingly applied to hard optimization problems, often claiming superiority over classical heuristics. However, such claims risk being unsolid due to a lack of standard benchmarks on truly hard instances. From a statistical physics perspective, we propose new hard benchmarks based on random problems. We provide these benchmarks, along with performance results from both classical heuristics and GNNs. Our fair comparison shows that classical algorithms still outperform GNNs. We discuss the challenges for neural networks in this domain. Future claims of superiority can be made more robust using our benchmarks, available at https://github.com/ArtLabBocconi/RandCSPBench.

Problem

• Graph Neural Networks
• Constraint Satisfaction Problems
• Benchmarking
• Hard Instances
• Statistical Physics

Innovation

• Graph Neural Networks
• Constraint Satisfaction Problems
• Hard Benchmarks
• Statistical Physics
• Random CSP