tensorrt

This work addresses the high latency, substantial memory footprint, and numerical instability—such as NaN outputs under FP16 precision—that hinder real-time inference of Transformer models. Building upon NVIDIA TensorRT, the authors develop a modular, containerized GPU-accelerated inference pipeline featuring a structure-aware mixed-precision strategy: FP16 is employed in linear layers while Softmax and LayerNorm operations retain FP32 to prevent overflow and preserve output fidelity, achieving cosine similarity ≥0.9998. A reproducible evaluation framework spanning over 360 configurations reveals that random inputs severely underestimate FP16 instability. Experiments demonstrate up to a 64.4× speedup over CPU baselines (single-sample latency <10 ms), a 63% reduction in memory usage, no degradation in downstream task accuracy, and stable FP16 acceleration ratios between 1.84× and 2.00×.

This work addresses the communication bottleneck caused by frequent inter-device synchronization in multi-GPU tensor-parallel inference, which limits scalability. The authors propose the Parallel Track Transformer architecture, featuring an innovative parallel-track computation structure and a novel task partitioning and scheduling strategy that substantially reduces cross-device dependencies while preserving model quality. Implemented within the TensorRT-LLM and vLLM frameworks, the approach achieves up to a 16-fold reduction in synchronization operations, decreases first-token latency by 15–30%, reduces per-token generation time by 2–12%, and improves throughput by as much as 31.9%.

To address the low parallel efficiency and suboptimal Tensor Core utilization of irregular sparse computations—such as Mixture-of-Experts (MoE)—on GPUs, this paper proposes a novel execution paradigm that synergistically combines static batching with dynamic task mapping. It statically compiles a dense task graph during compilation, transforming dynamic sparse inference into a single-kernel execution; a lightweight runtime scheduler then enables fine-grained task mapping onto hardware resources. This approach achieves, for the first time, highly efficient, targeted Tensor Core computation for MoE inference, attaining 91% and 95% of peak throughput utilization on NVIDIA H800 and H20 GPUs, respectively—significantly outperforming existing dynamic batching methods. The core contribution is a pioneering “compiler–runtime” co-optimization framework, establishing a new paradigm for high-throughput deployment of sparse models on hardware accelerators.

Triton GPU kernel optimization heavily relies on expert knowledge and is labor-intensive. Method: This paper proposes the first automated optimization framework that integrates dynamic performance profiling with large language model (LLM)-driven semantic reasoning. It employs a modular architecture to close the loop among bottleneck identification, data-driven code refactoring, and automatic correctness and performance validation—enabling architecture-agnostic, end-to-end tuning without human intervention. Innovatively, it unifies Nsight runtime analysis, static kernel feature extraction, and LLM-guided semantic-level rewriting to jointly optimize memory access patterns, arithmetic intensity, and parallelism. Results: Evaluated across diverse Triton kernels and GPU architectures, the framework achieves an average speedup of 1.76×, with peak improvements up to 5×, significantly lowering the barrier to high-performance kernel development.

To address the high memory movement overhead (≈50% of execution time spent on tensor reordering) and low energy efficiency of Kronecker-sparse (KS) matrix multiplication on GPUs, this work proposes the first KS-structure-aware tiling memory access strategy, co-optimizing GPU’s multi-level memory hierarchy to significantly reduce redundant data reads and writes. We further introduce the first unified benchmarking framework jointly evaluating energy consumption and execution time for KS-sparse operators. Our CUDA-based kernel achieves a median 1.4× speedup and 15% energy reduction across mainstream KS problem sizes. It has been successfully integrated into Transformer inference pipelines, demonstrating practical deployment value. The core innovations lie in (i) a KS-structure-guided tiling design that exploits inherent sparsity and Kronecker factorization patterns, and (ii) a hardware–software co-designed energy-efficiency optimization paradigm tailored to KS computation.

This work addresses the limited scalability of tensor parallelism in large model online inference, where non-scalable overheads hinder near-linear cluster performance scaling. The authors propose Albireo, a system that eliminates such bottlenecks without modifying model architecture by overlapping scheduling with computation, employing sequence-parallel sampling, and optimizing KV cache management. Albireo further introduces the concept of “empirically optimal tensor parallelism degree” to guide parallelism strategy selection. Experimental results demonstrate that, compared to vLLM, Albireo achieves up to 1.9× higher throughput, 48% lower latency, 28% improved GPU utilization, and 54% reduced energy consumption, with a twofold throughput gain observed in production environments.

This work addresses the high latency and kernel launch overhead that hinder large language models (LLMs) in short-sequence interactive inference. The authors propose a hybrid runtime framework that, for the first time, synergistically integrates just-in-time (JIT) compilation with dynamic CUDA Graph execution for LLM inference. During autoregressive decoding, the Transformer computation is partitioned into static components—replayed via CUDA Graphs—and dynamic components—handled by JIT-compiled kernels—while supporting asynchronous graph capture and cross-step reuse. This approach effectively balances low launch overhead with runtime flexibility. Evaluated on LLaMA-2 7B with batch size 1, the method reduces first-token latency by up to 66.0% and achieves better P99 latency than TensorRT-LLM.

This work addresses the limited flexibility in distributed programming for large language model scaling and the inefficiency of existing tensor compilers in handling the complex memory hierarchies of heterogeneous clusters. To overcome these challenges, the authors propose a scalable block-level compiler featuring a novel three-tier hierarchical abstraction—Core, Device, and Task—that uniformly supports diverse parallelization strategies, automatically optimizes intra- and inter-node communication, and enables efficient code generation across both NVIDIA and AMD platforms. When integrated into vLLM, the compiler achieves 5%–30% end-to-end inference speedup and over 10% improvement in training model FLOPs utilization (MFU), translating to approximately 500,000 GPU hours saved per month. The system has been deployed in enterprise settings, delivering over 20% inference performance gains.

This work addresses the significant engineering and platform-support challenges faced by non-GPU AI accelerators in deploying Mixture-of-Experts (MoE) and multimodal large language models. For the first time, it systematically identifies eight major categories of limitations inherent to such hardware during large-model inference. Leveraging Huawei’s Ascend 910 accelerator with the CANN software stack and a customized vLLM-Ascend framework, the authors successfully deploy the DeepSeek-V4-Flash family of MoE and vision-language models through W8A8 quantization, source-level plugin patches, and runtime fault-tolerance mechanisms. Experimental results demonstrate stable, high-concurrency multimodal inference for medical applications, achieving reliable performance on MMMU/MMMU-Pro benchmarks and safety alignment evaluations, while also quantifying the associated integration overhead and quality-of-service trade-offs.