TLDR: This research benchmarks five deep reinforcement learning algorithms (DQN, QR-DQN, A2C, PPO, TRPO) for the Container Stowage Planning Problem (CSPP), including crane scheduling. It introduces a new Gym environment (SPGE) and evaluates algorithms across various complexities and problem formulations (single-agent vs. multi-agent). Findings show that PPO and TRPO outperform others, especially in complex scenarios, with TRPO achieving the best results. The study highlights the importance of algorithm choice and problem formulation for efficient maritime logistics.
The global supply chain relies heavily on efficient maritime transportation, with container ports serving as critical hubs. At the heart of these operations lies the Container Stowage Planning Problem (CSPP), a complex challenge involving the optimal loading sequence of containers onto vessels. Traditionally, this task has depended on human expertise, but with increasing vessel sizes and throughput demands, manual planning has become a bottleneck, driving the need for automation and advanced solutions.
Recent advancements in artificial intelligence, particularly deep reinforcement learning (RL), have shown promise in tackling CSPP. However, a significant gap existed in systematically benchmarking different RL algorithms to understand their performance across varying complexities and problem formulations. A new research paper addresses this by developing a specialized Gym environment and conducting a comprehensive evaluation of five leading RL algorithms.
Understanding the Challenge: Container Stowage Planning
CSPP is an NP-hard problem, meaning it becomes incredibly difficult to solve optimally as the number of containers and slots increases. The goal is to place containers from a storage yard into vessel slots while adhering to numerous physical and logistical constraints, such as bay adjacency rules, vessel stability, and weight distribution limits. A key objective is to minimize “shifters” – additional movements required when a target container is blocked by others in the yard stack – and the total operation duration.
The research extends this problem to include crane scheduling, where the system must not only decide which container to move but also which crane should handle it. This joint optimization is crucial for overall efficiency, as siloed planning can limit improvements.
The Stowage Planning Gym Environment (SPGE)
To facilitate this benchmark study, the researchers developed the Stowage Planning Gym Environment (SPGE), an OpenAI Gym-compatible platform. SPGE abstracts vessels and yards as 3D grids of operational slots, each with attributes like coordinates, occupancy, and container group. This abstraction allows for flexible scaling of problem complexity, ensuring reproducibility and compatibility with standard RL libraries.
The environment was further extended into two distinct formulations for crane scheduling:
Stowage Planning Multiple Cranes (SPGE-MC): This models the problem with a single, centralized agent that controls both container selection and crane assignment. It offers explicit control over which crane handles which container.
Stowage Planning Agent Environment Cycle (SPAEC): This formulates the problem as a multi-agent system, where each crane acts as an independent agent controlled by a shared policy. Crane scheduling is handled implicitly based on availability and a predefined order.
Benchmarking Deep Reinforcement Learning Algorithms
The study benchmarked five prominent RL algorithms: DQN, QR-DQN, A2C, PPO, and TRPO, all implemented using the Stable-Baselines3 framework. To enhance training efficiency in these complex environments, action masking was integrated, preventing agents from attempting invalid actions.
Eight scenarios of progressively increasing complexity were designed, varying in vessel/yard size, container count, container type diversity, and the number of cranes (from 1 to 5). These scenarios allowed for a thorough evaluation of how each algorithm performed under different conditions.
Key Findings and Performance Insights
The results revealed distinct performance gaps as problem complexity increased. In simpler, single-crane scenarios, most algorithms performed comparably. However, with more container types or larger scales, performance diverged significantly:
A2C consistently yielded the poorest results across the board.
Value-based methods (DQN, QR-DQN) struggled with increased combinatorial complexity in larger problems, often converging to suboptimal policies.
PPO and TRPO demonstrated robust performance, with TRPO consistently achieving the best results in the most challenging single-crane scenarios, maintaining significantly lower shifter counts.
In multi-crane scenarios (Scenarios 6-8), the algorithm rankings remained similar, with TRPO continuing its dominance and A2C underperforming. The comparison between the single-agent (SPGE-MC) and multi-agent (SPAEC) formulations showed that their impact became significant only in more complex scenarios and with specific algorithms. Notably, TRPO exhibited the most significant performance differences between the two. The single-agent formulation generally proved more advantageous for optimizing shifters, likely due to its greater flexibility in crane selection, which aids in learning a globally optimal policy.
Also Read:
- Smart Manufacturing: Automating Job Scheduling with AI and Specialized Languages
- AutoMaAS: A Self-Evolving Framework for Multi-Agent AI Systems
Conclusion and Future Directions
This comprehensive benchmark study provides valuable insights into applying deep reinforcement learning to the Container Stowage Planning Problem. It confirms that while basic problems might be solvable by various RL methods, complex real-world scenarios demand more sophisticated algorithms like PPO and TRPO. The research also highlights the importance of problem formulation, with the single-agent approach showing an edge in shifter reduction due to enhanced decision-making flexibility.
The custom-designed SPGE environment, complete with crane scheduling capabilities, offers a robust foundation for future research. Future work could involve designing even richer scenarios, incorporating more realistic vessel hull structures, and developing sophisticated models for operation time to better reflect real-world maritime logistics. For more details, you can read the full research paper here.
