🤖 AI Summary
To address the challenges of GPU accelerator resource scheduling and weak cross-site collaboration for fundamental scientific research (e.g., high-energy physics) in federated cloud environments, this paper proposes a heterogeneous federated Kubernetes architecture leveraging Virtual Kubelets and interLink. The architecture enables fine-grained GPU virtualization, dynamic orchestration, and cross-domain sharing across multi-cloud infrastructures—achieving unified infrastructure management while preserving scientific use-case diversity. Built on cloud-native containerization, the platform integrates GPU-aware scheduling optimizations and federated network coordination mechanisms. Experimental evaluation demonstrates that the AI development platform supports concurrent GPU-intensive workloads from multiple research teams, improves GPU resource utilization by 40%, and reduces cross-datacenter workflow scheduling latency by 60%. These results significantly enhance the agility and scalability of distributed, data-driven scientific analysis in federated cloud settings.
📝 Abstract
Machine Learning (ML) is driving a revolution in the way scientists design, develop, and deploy data-intensive software. However, the adoption of ML presents new challenges for the computing infrastructure, particularly in terms of provisioning and orchestrating access to hardware accelerators for development, testing, and production. The INFN-funded project AI_INFN ("Artificial Intelligence at INFN") aims at fostering the adoption of ML techniques within INFN use cases by providing support on multiple aspects, including the provision of AI-tailored computing resources. It leverages cloud-native solutions in the context of INFN Cloud, to share hardware accelerators as effectively as possible, ensuring the diversity of the Institute's research activities is not compromised. In this contribution, we provide an update on the commissioning of a Kubernetes platform designed to ease the development of GPU-powered data analysis workflows and their scalability on heterogeneous, distributed computing resources, possibly federated as Virtual Kubelets with the interLink provider.