Efficient Unified Caching for Accelerating Heterogeneous AI Workloads

2025-06-14

AI Summary

To address low cache efficiency caused by heterogeneous workloads—such as preprocessing, training, and inference—in AI clusters accessing remote cloud storage, this paper proposes a unified, non-intrusive caching framework. Methodologically, it introduces (1) AccessStreamTree, a novel hierarchical access abstraction that models multi-granularity data access streams; (2) an online, multi-granularity access pattern recognition mechanism based on statistical hypothesis testing; and (3) dynamic, joint optimization of prefetching, eviction, and cache space allocation. Crucially, the framework requires no modifications to application code. Evaluation demonstrates significant improvements over state-of-the-art caching systems: a 55.6% increase in cache hit rate and a 52.2% reduction in end-to-end job completion time, thereby substantially enhancing shared cache resource utilization in AI clusters.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Modern AI clusters, which host diverse workloads like data pre-processing, training and inference, often store the large-volume data in cloud storage and employ caching frameworks to facilitate remote data access. To avoid code-intrusion complexity and minimize cache space wastage, it is desirable to maintain a unified cache shared by all the workloads. However, existing cache management strategies, designed for specific workloads, struggle to handle the heterogeneous AI workloads in a cluster -- which usually exhibit heterogeneous access patterns and item storage granularities. In this paper, we propose IGTCache, a unified, high-efficacy cache for modern AI clusters. IGTCache leverages a hierarchical access abstraction, AccessStreamTree, to organize the recent data accesses in a tree structure, facilitating access pattern detection at various granularities. Using this abstraction, IGTCache applies hypothesis testing to categorize data access patterns as sequential, random, or skewed. Based on these detected access patterns and granularities, IGTCache tailors optimal cache management strategies including prefetching, eviction, and space allocation accordingly. Experimental results show that IGTCache increases the cache hit ratio by 55.6% over state-of-the-art caching frameworks, reducing the overall job completion time by 52.2%.

Problem

• Unified caching for diverse AI workloads
• Handling heterogeneous access patterns and granularities
• Optimizing cache management strategies for efficiency

Innovation

• Hierarchical access abstraction for pattern detection
• Hypothesis testing for access pattern categorization
• Tailored cache management strategies for efficiency