CGSim: A Simulation Framework for Large Scale Distributed Computing Environment

2025-10-01

AI Summary

Existing large-scale distributed computing simulators—such as those tailored for the Worldwide LHC Computing Grid (WLCG)—suffer from poor scalability, inflexible, hard-coded scheduling policies, absence of real-time monitoring, and insufficient support for machine learning–ready data generation. To address these limitations, this work introduces a high-performance, modular simulation framework built upon SimGrid. It features a plugin-based architecture, automatic event-level workflow data generation, real-time visualization dashboards, and a high-level abstraction of heterogeneous execution environments. The framework enables customizable scheduler validation and AI-driven performance modeling. Evaluated at scale—spanning hundreds of sites and thousands of concurrent tasks—the framework achieves near-linear scalability and delivers a 6× speedup in multi-site load simulation. Calibrated against production WLCG data, it significantly improves site-level performance prediction accuracy. Crucially, all large-scale experiments execute efficiently on commodity hardware.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Hardware-aware ML

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Large-scale distributed computing infrastructures such as the Worldwide LHC Computing Grid (WLCG) require comprehensive simulation tools for evaluating performance, testing new algorithms, and optimizing resource allocation strategies. However, existing simulators suffer from limited scalability, hardwired algorithms, lack of real-time monitoring, and inability to generate datasets suitable for modern machine learning approaches. We present CGSim, a simulation framework for large-scale distributed computing environments that addresses these limitations. Built upon the validated SimGrid simulation framework, CGSim provides high-level abstractions for modeling heterogeneous grid environments while maintaining accuracy and scalability. Key features include a modular plugin mechanism for testing custom workflow scheduling and data movement policies, interactive real-time visualization dashboards, and automatic generation of event-level datasets suitable for AI-assisted performance modeling. We demonstrate CGSim's capabilities through a comprehensive evaluation using production ATLAS PanDA workloads, showing significant calibration accuracy improvements across WLCG computing sites. Scalability experiments show near-linear scaling for multi-site simulations, with distributed workloads achieving 6x better performance compared to single-site execution. The framework enables researchers to simulate WLCG-scale infrastructures with hundreds of sites and thousands of concurrent jobs within practical time budget constraints on commodity hardware.

Problem

• Simulates large-scale distributed computing environments for performance evaluation
• Addresses scalability limitations and hardwired algorithms in existing simulators
• Enables AI-assisted modeling through automatic dataset generation

Innovation

• Modular plugin mechanism for custom scheduling policies
• Interactive real-time visualization dashboards for monitoring
• Automatic event-level dataset generation for AI modeling