Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel

2026-04-14

AI Summary

This work addresses the inefficiencies in large language model (LLM) inference on modern GPUs, which are hindered by kernel launch overhead and coarse-grained synchronization that impede effective fusion of computations with dynamic shapes and data dependencies. To overcome these limitations, the authors introduce Event Tensor—the first unified compiler abstraction tailored for dynamic megakernels—that explicitly models inter-tile dependencies and natively supports dynamism as a first-class feature. They further develop the Event Tensor Compiler (ETC), which integrates static and dynamic scheduling transformations to automatically generate high-performance persistent kernels. Evaluated on LLM inference tasks, this approach achieves state-of-the-art latency performance while substantially reducing system warm-up overhead.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Modern GPU workloads, especially large language model (LLM) inference, suffer from kernel launch overheads and coarse synchronization that limit inter-kernel parallelism. Recent megakernel techniques fuse multiple operators into a single persistent kernel to eliminate launch gaps and expose inter-kernel parallelism, but struggle to handle dynamic shapes and data-dependent computation in real workloads. We present Event Tensor, a unified compiler abstraction for dynamic megakernels. Event Tensor encodes dependencies between tiled tasks, and enables first-class support for both shape and data-dependent dynamism. Built atop this abstraction, our Event Tensor Compiler (ETC) applies static and dynamic scheduling transformations to generate high-performance persistent kernels. Evaluations show that ETC achieves state-of-the-art LLM serving latency while significantly reducing system warmup overhead.

Problem

• kernel launch overhead
• inter-kernel parallelism
• dynamic shapes
• data-dependent computation
• megakernel

Innovation

• Event Tensor
• dynamic megakernel
• compiler abstraction
• LLM inference
• persistent kernel