DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents

2026-05-26

AI Summary

This study addresses the challenge of objectively evaluating algorithm performance in the Dynamic Flexible Job Shop Scheduling Problem (DFJSP), which is hindered by reliance on static benchmarks and uncalibrated instance generators. To overcome this, the authors propose DynaSchedBench, a diagnostic framework featuring a Sequential Event Space Calibrator (SESC) that computes a Scheduling Stress Index (SSI) to enable controllable generation of problem instances with tunable difficulty. The framework supports snapshot-based simulation, agent testing, and visualization. It achieves, for the first time, precise and efficient control over DFJSP instance hardness. Empirical analysis reveals an “observability paradox” in large language model (LLM)-based scheduling agents—access to complete information unexpectedly degrades performance. Furthermore, most LLM agents exhibit only heuristic-level approximation capabilities, failing to surpass strong handcrafted heuristics, with limited gains from tool augmentation.

Abstract

Progress in neural combinatorial optimization for Dynamic Flexible Job Shop Scheduling Problem (DFJSP) is currently hindered by a methodological tension: static benchmarks encourage benchmark overfitting, while uncalibrated generators obscure algorithmic capability with stochastic noise. To resolve this, we introduce \textbf{DynaSchedBench}, a diagnostic framework for DFJSP that rigorously controls the instance-generation process. Instead of relying on parameter sampling, our approach utilizes Sequential Event-Space Calibrator (SESC) that computes a novel Schedule Stress Index (SSI) to stratify instances by difficulty. We demonstrate that SESC is substantially more computationally efficient than evolutionary baselines while converging reliably to the target metrics. The framework integrates modular components for instance generation, snapshot-based simulation, agents, evaluation, and visualization, thereby enabling rigorous testing of reactive and lookahead-based policies. Leveraging this calibrated environment, we identify key limitations of LLM-based scheduling agents. Specifically, in step-wise online decision-making for dynamic scheduling, we identify an ``Observability Paradox'': providing agents with oracle access to full structural information can degrade policy performance, underperforming concise information. Furthermore, despite substantial token overhead, tool-augmented and refinement strategies fail to reliably improve performance, and most LLM agents fail to consistently surpass strong dispatching baselines-behaving more like robust heuristic approximators than superior optimizers.

Problem

• Dynamic Flexible Job Shop Scheduling
• Benchmark Overfitting
• Observability Paradox
• LLM-based Scheduling Agents
• Schedule Stress Index

Innovation

• DynaSchedBench
• Sequential Event-Space Calibrator
• Schedule Stress Index
• Observability Paradox
• LLM-based scheduling agents