Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures

2025-06-24

AI Summary

In global distributed computing, data placement and task scheduling decisions are typically decoupled, rely on empirical heuristics, and lack dynamic, holistic system-state evaluation models. Method: This paper introduces the first generative AI simulation framework that jointly incorporates explicit features (e.g., job duration, data location) and implicit features (e.g., unobservable computational load, system jitter). Leveraging five months of real job logs from the PanDA system, we develop a time-series-driven dynamic introspection model that accurately captures the evolution of multidimensional load metrics—including queueing latency, error rates, and remote data access overhead. Contribution/Results: The resulting interactive simulation system enables fine-grained modeling of both computation and data movement within scientific workflows, providing a verifiable and reproducible evaluation platform for AI-driven resource optimization. Crucially, this work is the first to explicitly integrate hidden system states into dynamic modeling, significantly improving the accuracy and generalizability of scheduling policy evaluation.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived data, extensive Monte Carlo simulation campaigns, and a wide range of end-user analysis. To manage these computational and storage demands, centralized workflow and data management systems are implemented. However, decisions regarding data placement and payload allocation are often made disjointly and via heuristic means. A significant obstacle in adopting more effective heuristic or AI-driven solutions is the absence of a quick and reliable introspective dynamic model to evaluate and refine alternative approaches. In this study, we aim to develop such an interactive system using real-world data. By examining job execution records from the PanDA workflow management system, we have pinpointed key performance indicators such as queuing time, error rate, and the extent of remote data access. The dataset includes five months of activity. Additionally, we are creating a generative AI model to simulate time series of payloads, which incorporate visible features like category, event count, and submitting group, as well as hidden features like the total computational load-derived from existing PanDA records and computing site capabilities. These hidden features, which are not visible to job allocators, whether heuristic or AI-driven, influence factors such as queuing times and data movement.

Problem

• Develop introspective dynamic model for global computing infrastructures
• Improve disjoint heuristic data placement and payload allocation
• Simulate hidden features influencing job performance metrics

Innovation

• Interactive system using real-world data
• Generative AI model for payload simulation
• Key performance indicators from PanDA records