Elk: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques

2025-07-15

AI Summary

To address the inherent performance trade-offs among computation, inter-core communication, and off-chip I/O in Interconnect-Centric AI (ICCA) chips, this paper proposes a deep learning compiler-driven co-optimization methodology. It introduces an inductive operator scheduling strategy that dynamically adapts to heterogeneous interconnect topologies and a cost-aware on-chip memory allocation algorithm, jointly constructing a configurable global trade-off space to enable architectural design space exploration. The method integrates HBM modeling, ICCA simulation, and roofline-based performance upper-bound analysis. Evaluated on the IPU-POD4 platform, it achieves, on average, 94% of the theoretical roofline performance—significantly improving end-to-end execution efficiency for large-scale models on ICCA chips. The key contribution is the first joint modeling of compiler-level scheduling and on-chip resource allocation, enabling holistic three-dimensional optimization across computation, communication, and I/O.

Technology Category

• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

To meet the increasing demand of deep learning (DL) models, AI chips are employing both off-chip memory (e.g., HBM) and high-bandwidth low-latency interconnect for direct inter-core data exchange. However, it is not easy to explore the efficiency of these inter-core connected AI (ICCA) chips, due to a fundamental tussle among compute (per-core execution), communication (inter-core data exchange), and I/O (off-chip data access). In this paper, we develop Elk, a DL compiler framework to maximize the efficiency of ICCA chips by jointly trading off all the three performance factors discussed above. Elk structures these performance factors into configurable parameters and forms a global trade-off space in the DL compiler. To systematically explore this space and maximize overall efficiency, Elk employs a new inductive operator scheduling policy and a cost-aware on-chip memory allocation algorithm. It generates globally optimized execution plans that best overlap off-chip data loading and on-chip execution. To examine the efficiency of Elk, we build a full-fledged emulator based on a real ICCA chip IPU-POD4, and an ICCA chip simulator for sensitivity analysis with different interconnect network topologies. Elk achieves 94% of the ideal roofline performance of ICCA chips on average, showing the benefits of supporting large DL models on ICCA chips. We also show Elk's capability of enabling architecture design space exploration for new ICCA chip development.

Problem

• Optimizing inter-core connected AI chips efficiency
• Balancing compute, communication, and I/O trade-offs
• Enabling architecture design space exploration

Innovation

• DL compiler framework for inter-core AI chips
• Inductive operator scheduling policy
• Cost-aware on-chip memory allocation algorithm