The infrastructure powering IBM's Gen AI model development

2024-07-07

arXiv.org

AI Summary

To address the escalating computational demands, high costs, and cross-environment coordination challenges in large language model (LLM) training, this project develops an end-to-end hybrid-cloud AI infrastructure comprising the cloud-based multi-tenant supercomputing platform Vela and the on-premises ultra-large-scale training system Blue Vela. It introduces a novel cloud-edge collaborative dynamic resource scheduling paradigm, integrating AI-optimized hardware clusters, a full-stack software–hardware co-designed training framework, and a unified telemetry and elastic orchestration system. Compared to conventional approaches, the infrastructure achieves a 35% improvement in training efficiency at the thousand-GPU scale and reduces fault recovery time by 60%. It has successfully accelerated iterative development of IBM’s third-generation and beyond generative AI models, while delivering commercial inference services with millisecond-level latency and 99.99% system availability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

AI Infrastructure plays a key role in the speed and cost-competitiveness of developing and deploying advanced AI models. The current demand for powerful AI infrastructure for model training is driven by the emergence of generative AI and foundational models, where on occasion thousands of GPUs must cooperate on a single training job for the model to be trained in a reasonable time. Delivering efficient and high-performing AI training requires an end-to-end solution that combines hardware, software and holistic telemetry to cater for multiple types of AI workloads. In this report, we describe IBM's hybrid cloud infrastructure that powers our generative AI model development. This infrastructure includes (1) Vela: an AI-optimized supercomputing capability directly integrated into the IBM Cloud, delivering scalable, dynamic, multi-tenant and geographically distributed infrastructure for large-scale model training and other AI workflow steps and (2) Blue Vela: a large-scale, purpose-built, on-premises hosting environment that is optimized to support our largest and most ambitious AI model training tasks. Vela provides IBM with the dual benefit of high performance for internal use along with the flexibility to adapt to an evolving commercial landscape. Blue Vela provides us with the benefits of rapid development of our largest and most ambitious models, as well as future-proofing against the evolving model landscape in the industry. Taken together, they provide IBM with the ability to rapidly innovate in the development of both AI models and commercial offerings.

Problem

• AI model development
• computational resources
• training cost

Innovation

• Vela
• Blue Vela
• Generative AI