The Streaming Batch Model for Efficient and Fault-Tolerant Heterogeneous Execution

2025-01-16

AI Summary

To address CPU bottlenecks in data preprocessing, imbalanced resource allocation, and insufficient fault tolerance during ML training and inference on CPU-GPU heterogeneous systems, this paper proposes a Streaming-Batch hybrid execution model. It synergistically integrates the low-latency benefits of streaming execution with the high-throughput advantages of batched processing, enabling shard-wise incremental execution, lineage-based lightweight fault recovery, and dynamic heterogeneous resource scheduling—thereby eliminating reliance on homogeneous hardware. Implemented atop Ray Data, the model achieves 3–8× higher throughput for heterogeneous batch inference. In Stable Diffusion training, it improves end-to-end training throughput by 31% while preserving per-node data loading throughput.

Technology Category

• Machine Learning: Hardware-aware ML
• Humans and AI: Human-in-the-loop Machine Learning
• Computer Vision: Diffusion Models for Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While ML model training and inference are both GPU-intensive, CPU-based data processing is often the bottleneck. Distributed data processing systems based on the batch or stream processing models assume homogeneous resource requirements. They excel at CPU-based computation but either under-utilize heterogeneous resources or impose high overheads on failure and reconfiguration. We introduce the streaming batch model, a hybrid of the two models that enables efficient and fault-tolerant heterogeneous execution. The key idea is to execute one partition at a time to allow lineage-based recovery with dynamic resource allocation. This enables memory-efficient pipelining across heterogeneous resources, similar to stream processing, but also offers the elasticity and fault tolerance properties of batch processing. We present Ray Data, an implementation of the streaming batch model that improves throughput on heterogeneous batch inference pipelines by 3--8$ imes$ compared to traditional batch and stream processing systems. When training Stable Diffusion, Ray Data matches the throughput of single-node ML data loaders while additionally leveraging distributed heterogeneous clusters to further improve training throughput by 31%.

Problem

• Big Data Processing
• Resource Allocation
• Machine Learning Efficiency

Innovation

• Stream-Batch Hybrid Mode
• Heterogeneous Resource Optimization
• Ray Data Performance Enhancement