A System Level Compiler for Massively-Parallel, Spatial, Dataflow Architectures

2025-06-18

AI Summary

To address the lack of efficient and portable compilers for spatial dataflow architectures—such as the Cerebras Wafer-Scale Engine—this paper introduces MACH, a system-level compiler. Methodologically, MACH introduces (i) the first virtual machine abstraction tailored to spatial architectures; (ii) a domain-specific language (DSL) with NumPy semantics, backed by a multi-level intermediate representation (IR); and (iii) an extensible multi-target lowering framework that enables co-compilation across unified memory systems and spatial hardware. By bridging high-level tensor computations—including dense NumPy operations—to low-level Wafer-Scale Engine instructions, MACH achieves end-to-end compilation while preserving semantic expressiveness. Evaluation demonstrates substantial improvements in programming productivity and cross-architecture portability, without sacrificing performance or correctness.

Technology Category

• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Machine learning and data science for the Web

Abstract

We have developed a novel compiler called the Multiple-Architecture Compiler for Advanced Computing Hardware (MACH) designed specifically for massively-parallel, spatial, dataflow architectures like the Wafer Scale Engine. Additionally, MACH can execute code on traditional unified-memory devices. MACH addresses the complexities in compiling for spatial architectures through a conceptual Virtual Machine, a flexible domain-specific language, and a compiler that can lower high-level languages to machine-specific code in compliance with the Virtual Machine concept. While MACH is designed to be operable on several architectures and provide the flexibility for several standard and user-defined data mappings, we introduce the concept with dense tensor examples from NumPy and show lowering to the Wafer Scale Engine by targeting Cerebras'hardware specific languages.

Problem

• Compiling for massively-parallel spatial dataflow architectures
• Bridging high-level languages to machine-specific code efficiently
• Supporting diverse architectures with flexible data mappings

Innovation

• Compiler for massively-parallel spatial dataflow architectures
• Virtual Machine concept for spatial architecture compilation
• Flexible domain-specific language for high-level code lowering