R-ConstraintBench: Evaluating LLMs on NP-Complete Scheduling

2025-08-20

AI Summary

This work investigates the reasoning reliability of large language models (LLMs) on the highly constrained NP-complete Resource-Constrained Project Scheduling Problem (RCPSP). We propose R-ConstraintBench, a novel evaluation framework that generates synthetic RCPSP instances of controllable difficulty using directed acyclic graphs, systematically incorporating non-redundant precedence, downtime, time-window, and mutual-exclusion constraints. Our feasibility analysis and error-mode diagnosis reveal that constraint interaction—not individual constraint complexity—is the primary bottleneck causing LLM failure. Experiments across diverse LLMs show near-optimal performance under single precedence constraints but sharp feasibility degradation under composite constraints; moreover, strong synthetic-data performance fails to generalize to real-world domain instances. This is the first study to quantitatively characterize the detrimental impact of constraint interaction on LLM reasoning for scheduling. The work establishes a scalable, diagnostic benchmark and methodology for developing trustworthy AI-driven scheduling systems.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Constraint Satisfaction and Optimization: Constraint Programming

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Effective scheduling under tight resource, timing, and operational constraints underpins large-scale planning across sectors such as capital projects, manufacturing, logistics, and IT fleet transitions. However, the reliability of large language models (LLMs) when reasoning under high-constraint regimes is insufficiently characterized. To address this gap, we present R-ConstraintBench, a scalable framework that evaluates models on Resource-Constrained Project Scheduling Problems (RCPSP), an NP-Complete feasibility class, while difficulty increases via linear growth in constraints. R-ConstraintBench incrementally increases non-redundant precedence constraints in Directed Acyclic Graphs (DAGs) and then introduces downtime, temporal windows, and disjunctive constraints. As an illustrative example, we instantiate the benchmark in a data center migration setting and evaluate multiple LLMs using feasibility and error analysis, identifying degradation thresholds and constraint types most associated with failure. Empirically, strong models are near-ceiling on precedence-only DAGs, but feasibility performance collapses when downtime, temporal windows, and disjunctive constraints interact, implicating constraint interaction, not graph depth, as the principal bottleneck. Performance on clean synthetic ramps also does not guarantee transfer to domain-grounded scenarios, underscoring limited generalization.

Problem

• Evaluating LLM reliability in high-constraint NP-Complete scheduling problems
• Assessing performance degradation under increasing operational constraints
• Testing generalization from synthetic to real-world scheduling scenarios

Innovation

• Scalable framework for NP-Complete scheduling evaluation
• Incremental constraint introduction with DAG precedence constraints
• Feasibility analysis with downtime and disjunctive constraints