cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution

2025-12-18

AI Summary

Addressing challenges in automated CUDA kernel optimization—including tight hardware-software co-design difficulties, inefficient surrogate modeling, and semantic misalignment between evolutionary representations and hardware constraints—this paper proposes a strategy-coordinated multi-agent framework. We introduce “strategy” as a semantic abstraction to model kernel evolution, pioneering strategy-level evolutionary representation and cross-agent coordination mechanisms. Integrating Roofline-guided LLM prompting with strategy-aware population initialization enables hardware-informed end-to-end kernel generation. By unifying large language models, evolutionary algorithms, Roofline performance modeling, and multi-agent scheduling, our approach achieves an average 3.09× speedup over PyTorch across 100 kernel benchmarks; notably, it improves utilization of critical hardware units (e.g., tensor cores, shared memory) in GEMM workloads. All generated kernels are open-sourced.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Adversarial Agents

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Optimizing CUDA kernels is a challenging and labor-intensive task, given the need for hardware-software co-design expertise and the proprietary nature of high-performance kernel libraries. While recent large language models (LLMs) combined with evolutionary algorithms show promise in automatic kernel optimization, existing approaches often fall short in performance due to their suboptimal agent designs and mismatched evolution representations. This work identifies these mismatches and proposes cuPilot, a strategy-coordinated multi-agent framework that introduces strategy as an intermediate semantic representation for kernel evolution. Key contributions include a strategy-coordinated evolution algorithm, roofline-guided prompting, and strategy-level population initialization. Experimental results show that the generated kernels by cuPilot achieve an average speed up of 3.09$ imes$ over PyTorch on a benchmark of 100 kernels. On the GEMM tasks, cuPilot showcases sophisticated optimizations and achieves high utilization of critical hardware units. The generated kernels are open-sourced at https://github.com/champloo2878/cuPilot-Kernels.git.

Problem

• Optimizes CUDA kernels automatically using multi-agent framework
• Addresses performance gaps in existing LLM-based kernel evolution methods
• Enhances hardware utilization through strategy-coordinated evolution algorithm

Innovation

• Strategy-coordinated multi-agent framework for kernel evolution
• Roofline-guided prompting to optimize CUDA kernel performance
• Strategy-level population initialization for evolutionary algorithm