Fearless Concurrency on the GPU

2026-06-14

AI Summary

Developing GPU kernels in Rust presents a significant challenge in simultaneously achieving high performance and memory safety. This work proposes cuTile Rust, the first system to enforce Rust’s ownership semantics within GPU kernels. By introducing a tiled kernel design coupled with a novel ownership separation mechanism, cuTile Rust enables safe, idiomatic low-level control while unifying diverse execution models—including synchronous launches, asynchronous pipelining, and CUDA Graph replay. Evaluated on NVIDIA B200 hardware, the system attains 7 TB/s element-wise memory bandwidth and 2 PFlop/s GEMM performance, reaching 96% of cuBLAS throughput. Furthermore, the Grout inference engine built atop cuTile Rust achieves 171 and 82 tokens/s on Qwen3-4B and Qwen3-32B models, respectively, matching the performance of state-of-the-art frameworks such as vLLM and SGLang.

Abstract

Rust has made safe systems programming practical on the CPU, but writing custom GPU kernels in Rust still forces programmers outside the language's ownership guarantees. We present cuTile Rust, a tile-based system for safe, idiomatic GPU kernel authoring in Rust. cuTile Rust extends Rust's ownership discipline to tile-based GPU kernels: mutable outputs are split into disjoint pieces, kernel launches preserve the host-side ownership contract, and programmers can opt out locally when they need lower-level control. The system also provides a composable host execution model spanning synchronous launches, asynchronous pipelines, and CUDA graph replay. Our evaluation shows that these abstractions can preserve performance on high-end GPUs. On the NVIDIA B200 GPU, cuTile Rust achieves 7 TB/s for element-wise operations and 2 PFlop/s for GEMM (96% of cuBLAS), matching cuTile Python within measurement noise. Grout, a cuTile-Rust-based inference engine, exercises cuTile Rust across an end-to-end Qwen3 inference path. In batch-1 decode, Grout reaches 171 generated tokens/s for Qwen3-4B on the NVIDIA GeForce RTX 5090 and 82 generated tokens/s for Qwen3-32B on the B200, competitive with vLLM and SGLang and consistent with an HBM roofline sanity check.

Problem

• GPU programming
• Rust ownership
• safe concurrency
• kernel safety
• memory safety

Innovation

• safe GPU programming
• Rust ownership
• tile-based kernels
• composable execution model
• high-performance inference