model inference

To address the low parallel efficiency and suboptimal Tensor Core utilization of irregular sparse computations—such as Mixture-of-Experts (MoE)—on GPUs, this paper proposes a novel execution paradigm that synergistically combines static batching with dynamic task mapping. It statically compiles a dense task graph during compilation, transforming dynamic sparse inference into a single-kernel execution; a lightweight runtime scheduler then enables fine-grained task mapping onto hardware resources. This approach achieves, for the first time, highly efficient, targeted Tensor Core computation for MoE inference, attaining 91% and 95% of peak throughput utilization on NVIDIA H800 and H20 GPUs, respectively—significantly outperforming existing dynamic batching methods. The core contribution is a pioneering “compiler–runtime” co-optimization framework, establishing a new paradigm for high-throughput deployment of sparse models on hardware accelerators.

Triton GPU kernel optimization heavily relies on expert knowledge and is labor-intensive. Method: This paper proposes the first automated optimization framework that integrates dynamic performance profiling with large language model (LLM)-driven semantic reasoning. It employs a modular architecture to close the loop among bottleneck identification, data-driven code refactoring, and automatic correctness and performance validation—enabling architecture-agnostic, end-to-end tuning without human intervention. Innovatively, it unifies Nsight runtime analysis, static kernel feature extraction, and LLM-guided semantic-level rewriting to jointly optimize memory access patterns, arithmetic intensity, and parallelism. Results: Evaluated across diverse Triton kernels and GPU architectures, the framework achieves an average speedup of 1.76×, with peak improvements up to 5×, significantly lowering the barrier to high-performance kernel development.

In edge vision inference, GPU resources—particularly Streaming Multiprocessors (SMs) and Tensor Cores on Jetson devices—exhibit low utilization (15%–30%), while CPU scheduling emerges as the dominant performance bottleneck. Method: We propose a multi-granularity collaborative profiling framework that integrates NVIDIA Nsight, Linux Perf, and custom event-tracing tools to simultaneously capture GPU hardware metrics (SM/TC utilization, memory bandwidth) and CPU system-level events (context switches, thread scheduling latency). Contribution/Results: Our analysis reveals that high aggregate GPU utilization masks underutilization of individual compute units and confirms CPU-side overhead as the primary contributor to end-to-end latency. Based on these insights, we derive hardware-aware optimization principles tailored for edge AI. Experimental validation demonstrates a 2.1× improvement in concurrent inference throughput. This work establishes a reproducible empirical foundation and systematic optimization pathway for heterogeneous resource orchestration at the edge.

This study systematically investigates and quantifies the critical impact of CPU bottlenecks on large language model (LLM) inference performance in multi-GPU settings, where insufficient CPU resources often lead to suboptimal GPU utilization and elevated latency—even when GPUs are not fully saturated. Through comprehensive profiling of real-world LLM serving workloads, the work identifies key CPU-induced inefficiencies, including kernel launch delays, communication stalls, and tokenization overhead. Experimental results demonstrate that moderately increasing CPU core count—without adding more GPUs—can reduce time-to-first-token latency by 1.36× to 5.40× while substantially improving system stability and responsiveness. These findings underscore the necessity of balanced CPU-GPU provisioning for efficient LLM inference deployment.

Current large language models (LLMs) frequently produce Triton GPU kernels with functional errors and suboptimal performance, primarily due to insufficient understanding of Triton programming conventions and GPU hardware characteristics, compounded by the absence of a dedicated evaluation benchmark. To address this, we introduce TritonBench—the first comprehensive benchmark for Triton kernel generation—comprising 184 real-world GitHub kernels and their PyTorch-aligned implementations. We pioneer the integration of empirical GPU performance metrics (kernel latency and throughput) into code generation evaluation, proposing a dual-axis assessment framework: functional correctness plus hardware-aware performance. TritonBench incorporates PyTorch interface specifications, CUDA configuration profiling, an open-source kernel dataset, and an automated verification pipeline. Experiments reveal that state-of-the-art LLMs achieve an average performance compliance rate of less than 12%, exposing a critical bottleneck in LLM–AI compiler co-optimization. TritonBench establishes a reproducible, extensible evaluation infrastructure for high-performance operator synthesis.

This work addresses the performance bottleneck in modern deep learning caused by the overhead of frequent GPU kernel launches due to numerous small-scale tensor operations. To mitigate this, the authors propose a persistent GPU kernel runtime system that continuously schedules computational tasks via a host-side task queue and dynamically loads operators through NVRTC-based just-in-time compilation and function pointer injection, thereby eliminating redundant kernel launches. The design introduces an innovative dual-slot aliasing mechanism for concurrent updates and a generic tensor abstraction, enabling transparent integration with PyTorch via TorchDispatch. Evaluated on representative workloads dominated by small operations, the system achieves up to 15.3× speedup over standard PyTorch, substantially improves GPU utilization, and maintains strong ecosystem compatibility.

This study addresses a critical limitation in existing machine learning latency prediction methods, which typically assume GPU kernel execution times are independent and overlook inter-kernel timing dependencies introduced by dynamic frequency scaling. Through empirical characterization of NVIDIA GPU frequency behavior under ML/AI workloads, we systematically reveal—for the first time—that recent workload history within an approximately 80 ms window significantly influences operating frequency, particularly on lower-end GPUs. Leveraging GPU performance monitoring, kernel-level latency measurements, and frequency trajectory analysis, we demonstrate that conventional latency models based on summing independent kernel durations exhibit substantial inaccuracies. These findings challenge a foundational assumption in mainstream latency modeling and provide crucial insights for designing novel scheduling and optimization mechanisms that jointly account for frequency dynamics, latency, and energy efficiency.

This work addresses the high latency, substantial memory footprint, and numerical instability—such as NaN outputs under FP16 precision—that hinder real-time inference of Transformer models. Building upon NVIDIA TensorRT, the authors develop a modular, containerized GPU-accelerated inference pipeline featuring a structure-aware mixed-precision strategy: FP16 is employed in linear layers while Softmax and LayerNorm operations retain FP32 to prevent overflow and preserve output fidelity, achieving cosine similarity ≥0.9998. A reproducible evaluation framework spanning over 360 configurations reveals that random inputs severely underestimate FP16 instability. Experiments demonstrate up to a 64.4× speedup over CPU baselines (single-sample latency <10 ms), a 63% reduction in memory usage, no degradation in downstream task accuracy, and stable FP16 acceleration ratios between 1.84× and 2.00×.

This work addresses the high latency and kernel launch overhead that hinder large language models (LLMs) in short-sequence interactive inference. The authors propose a hybrid runtime framework that, for the first time, synergistically integrates just-in-time (JIT) compilation with dynamic CUDA Graph execution for LLM inference. During autoregressive decoding, the Transformer computation is partitioned into static components—replayed via CUDA Graphs—and dynamic components—handled by JIT-compiled kernels—while supporting asynchronous graph capture and cross-step reuse. This approach effectively balances low launch overhead with runtime flexibility. Evaluated on LLaMA-2 7B with batch size 1, the method reduces first-token latency by up to 66.0% and achieves better P99 latency than TensorRT-LLM.

This study addresses the lack of systematic quantification of inference performance differences between NVIDIA T4 and L4 GPUs under controlled conditions. Following the GDEV-AI methodology, we evaluate inference throughput and energy efficiency of ResNet-family models across FP32, FP16, and INT8 precisions using PyTorch and TensorRT on identically configured systems. Our analysis reveals, for the first time, that the L4 significantly outperforms the T4 in low-latency scenarios with small batch sizes (e.g., batch=1–8), whereas the T4 remains competitive at larger batch sizes. INT8 inference achieves up to 58× higher throughput than a CPU baseline, and the L4 delivers up to 4.4× greater throughput than the T4, attaining peak energy efficiency at batch sizes of 16–32—effectively balancing latency and throughput.