A Benchmark Study of Deep Reinforcement Learning Algorithms for the Container Stowage Planning Problem

2025-10-02

AI Summary

Container Stowage Planning Problem (CSPP) is a critical challenge for enhancing port operational efficiency; traditionally addressed via heuristic rules, while existing reinforcement learning (RL) studies lack systematic algorithmic benchmarks. This work introduces the first standardized Gym environment supporting quay crane coordination, and proposes both single-agent and multi-agent modeling frameworks. We conduct the first unified evaluation of five deep RL algorithms—DQN, QR-DQN, A2C, PPO, and TRPO—across scenarios of varying complexity. Experimental results demonstrate significant performance divergence as problem scale increases, empirically validating the decisive impact of modeling paradigm and algorithm selection on real-world maritime decision-making. Our contribution includes a reproducible, extensible benchmark platform for intelligent stowage planning, and provides empirical guidance for RL algorithm selection and optimization in port automation systems.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Container stowage planning (CSPP) is a critical component of maritime transportation and terminal operations, directly affecting supply chain efficiency. Owing to its complexity, CSPP has traditionally relied on human expertise. While reinforcement learning (RL) has recently been applied to CSPP, systematic benchmark comparisons across different algorithms remain limited. To address this gap, we develop a Gym environment that captures the fundamental features of CSPP and extend it to include crane scheduling in both multi-agent and single-agent formulations. Within this framework, we evaluate five RL algorithms: DQN, QR-DQN, A2C, PPO, and TRPO under multiple scenarios of varying complexity. The results reveal distinct performance gaps with increasing complexity, underscoring the importance of algorithm choice and problem formulation for CSPP. Overall, this paper benchmarks multiple RL methods for CSPP while providing a reusable Gym environment with crane scheduling, thus offering a foundation for future research and practical deployment in maritime logistics.

Problem

• Benchmarking deep reinforcement learning for container stowage planning
• Evaluating algorithm performance gaps under varying complexity scenarios
• Providing reusable environment with crane scheduling for maritime logistics

Innovation

• Developed Gym environment for container stowage planning
• Extended framework with crane scheduling capabilities
• Benchmarked five RL algorithms across complexity scenarios