cutlass

This study systematically evaluates the performance and portability of CUDA Tile across Hopper and Blackwell GPU architectures for AI workloads. Focusing on GEMM, fused multi-head attention, and large language model (LLM) inference, the authors benchmark implementations based on cuBLAS, Triton, WMMA, and SIMT under BF16/FP16 precision on the H100 NVL, B200, and RTX PRO 6000 Blackwell Server Edition GPUs. Their analysis reveals that CUDA Tile achieves exceptional performance on the datacenter-grade B200—reaching 1007 TFLOP/s in fused attention (2.5× faster than FlashAttention-2) and attaining 52–79% of cuBLAS’s GEMM throughput—yet demonstrates suboptimal optimization on consumer-class hardware and significantly weaker cross-architecture portability compared to Triton.

This work addresses the inefficiencies of conventional GPU linear algebra libraries, which often fail to fully exploit hardware capabilities due to redundant memory accesses and runtime overhead. To overcome these limitations, the authors propose Bandicoot, a toolkit that leverages C++ template metaprogramming to perform expression fusion at compile time, thereby automatically generating highly optimized GPU kernels that saturate memory bandwidth—without relying on just-in-time compilation or runtime scheduling. Bandicoot provides an API compatible with Armadillo, facilitating straightforward migration of existing CPU codebases. Experimental results demonstrate that Bandicoot significantly outperforms PyTorch, TensorFlow, and JAX across multiple benchmarks, achieving substantial speedups in several scenarios.

This study addresses the GPU computational efficiency bottleneck in deep and machine learning. Methodologically, it proposes a task-aware GPU parallel architecture adaptation framework that systematically integrates CUDA stream-based concurrency, dynamic parallelism, and heterogeneous hardware (FPGA/TPU/ASIC) co-selection—implemented via deep integration into PyTorch, TensorFlow, and XGBoost. Its core contribution lies in establishing a transferable GPU optimization methodology, transcending model- or library-specific tuning. Experimental evaluation demonstrates 3–8× speedup across representative training and inference workloads. Furthermore, the authors open-source a modular, well-documented GPU optimization practice guide, substantially lowering the barrier to entry for AI practitioners seeking parallelization optimizations.

This work addresses the challenge of automatically generating high-performance GPU tiled kernels from high-level tensor algebra expressions, thereby alleviating the burden of manual optimization. The authors propose an end-to-end compilation framework that integrates layer-wise lowering, expression rewriting, automated schedule search, reduction fusion, and tiling optimizations. For the first time, this framework automatically discovers high-efficiency kernels—comparable to FlashAttention-3—from the mathematical specification of attention operators, while introducing a novel scheduler that preserves program structural regularity. Evaluated on GH200 and RTX 5090 GPUs, the generated kernels achieve up to 23% and 42% higher throughput, respectively, and match or surpass hand-optimized cuDNN kernels across multiple long-sequence configurations.

Existing CPU-based linear algebra libraries (e.g., Armadillo) face significant challenges in efficient GPU porting. To address this, we propose Bandicoot—a CUDA-accelerated C++ library that maintains full interface compatibility with Armadillo. Its core innovations include: (1) a compile-time expression template system leveraging CUDA and C++ template metaprogramming to enable delayed evaluation and automatic mathematical expression optimization; and (2) a unified abstraction layer that transparently bridges CPU and GPU execution, minimizing required code modifications. Experimental evaluation on representative linear algebra workloads demonstrates speedups of several-fold to over an order of magnitude over CPU-only Armadillo. Bandicoot is open-sourced under the Apache 2.0 license, facilitating integration into existing Armadillo-based scientific computing pipelines.

This work addresses the challenge of type erasure in GPU binaries, where GPU compilers uniformly store all data types in the register file, thereby discarding type information and impeding disassembly and semantic recovery. To systematically resolve this issue, the paper introduces a constraint-propagation-based type inference method that integrates conflict detection, control-flow reconstruction, and multi-instruction pattern recognition to lift SASS binaries into typed LLVM IR. Evaluated on 24,437 GPU functions across eight benchmark suites, the approach achieves a 99.98% lifting success rate. Ablation studies further demonstrate that disabling type recovery reduces the x86 compatibility rate from 73.8% to 0%, underscoring the critical role of accurate type reconstruction in generating semantically correct intermediate representations.

This work addresses the CPU computational bottleneck encountered by Presto in large-scale analytical queries by introducing the first deep integration of GPU execution into the Presto framework. The authors propose an end-to-end query processing architecture that retains data in GPU memory throughout the execution pipeline. By redesigning the execution engine, they enable GPU-aware scheduling, efficient data transfer from storage to GPU, and intra-operator data exchange directly within GPU memory in distributed environments. The system is built upon NVIDIA’s cuDF C++ library and guided by TPC-H benchmarks for optimization. Experimental results demonstrate that, under standard analytical workloads, the proposed approach achieves up to a 6× improvement in cost-performance ratio compared to native CPU-based Presto. The implementation has been upstreamed into the open-source Presto/Velox engine and deployed in production.

This work addresses the challenge of efficiently solving structured, sparse, symmetric positive-definite systems arising in high-resolution thermal simulation of 3D integrated circuits, where general-purpose GPU sparse solvers struggle to achieve high performance. To this end, the authors propose CUTh-Solver, a GPU-accelerated preconditioned conjugate gradient (PCG) framework tailored for this domain. Its key innovations include a compressed diagonal storage format (cDIA), diagonal-level sparse matrix-vector multiplication (SpMV) to coalesce memory accesses, a highly parallelizable preconditioning strategy, and an adaptive fine-grained mixed-precision mechanism that enhances throughput while preserving numerical stability. Experimental results demonstrate that CUTh-Solver achieves up to 25.8× speedup over GPU-accelerated COMSOL Multiphysics 6.4 and more than 3× acceleration compared to general-purpose libraries such as NVIDIA AmgX, cuSPARSE, and cuDSS, with ablation studies confirming the effectiveness of each optimization component.

Developing GPU kernels in Rust presents a significant challenge in simultaneously achieving high performance and memory safety. This work proposes cuTile Rust, the first system to enforce Rust’s ownership semantics within GPU kernels. By introducing a tiled kernel design coupled with a novel ownership separation mechanism, cuTile Rust enables safe, idiomatic low-level control while unifying diverse execution models—including synchronous launches, asynchronous pipelining, and CUDA Graph replay. Evaluated on NVIDIA B200 hardware, the system attains 7 TB/s element-wise memory bandwidth and 2 PFlop/s GEMM performance, reaching 96% of cuBLAS throughput. Furthermore, the Grout inference engine built atop cuTile Rust achieves 171 and 82 tokens/s on Qwen3-4B and Qwen3-32B models, respectively, matching the performance of state-of-the-art frameworks such as vLLM and SGLang.

This work addresses the challenge of optimizing and analyzing GPU kernels for deep convolution in cloud environments lacking hardware performance counters. The authors propose a counter-free performance analysis framework that integrates execution path decomposition with memory traffic modeling to systematically optimize the forward, input gradient, and weight gradient computation paths. Leveraging CUDA event timing, effective bandwidth estimation, and the roofline model, the approach employs optimization strategies such as shared memory tiling and warp-level partitioning. The resulting warp-tiled kernels achieve a 3.26× speedup over naive implementations, yielding a 1.29× end-to-end training acceleration. This significantly enhances the efficiency of critical computational paths and provides reproducible, architecture-level performance insights for resource-constrained cloud settings.