vllm

A systematic analysis of large language model (LLM) inference system architectures—and the underlying technical synergies among their components—remains lacking. Method: We propose the first unified analytical framework that uncovers three foundational principles: workload forecasting, adaptive scheduling, and cost-aware compression. We introduce a deployment-paradigm-based taxonomy—categorizing systems into single-replica, multi-replica, decoupled, and serverless configurations—and comprehensively integrate key techniques including CUDA kernel optimization, continuous batching, PagedAttention, KV cache compression and persistence, weight/activation quantization, and memory offloading. Contribution/Results: Our work establishes the first holistic architecture map of LLM inference systems, explicitly characterizing inter-technique synergies and fundamental trade-offs. The resulting framework provides a systematic design guide for deploying LLMs efficiently, elastically, and cost-effectively.

To address the high deployment barrier and challenging performance tuning of large language model (LLM) inference across heterogeneous hardware and multiple deep learning frameworks, this paper proposes a simulation-driven dynamic modeling and optimization approach. We introduce a globally unified inference engine architecture and develop a dynamic performance model that captures hardware-framework-workload coupling, enabling the first systematic quantification of cross-layer bottlenecks. A hybrid offline/online simulation mechanism is designed to predict multi-dimensional metrics—including batch latency, time-to-first-token, and decoding throughput—with prediction errors ranging from 9.9% to 42.3%. Furthermore, we implement heterogeneous resource-aware scheduling, significantly improving memory utilization and multi-GPU throughput stability. Our method enables zero-code, efficient LLM deployment across vendor-diverse hardware and mainstream frameworks, substantially lowering the adoption barrier for non-expert users.

To address the challenge of balancing throughput and latency in large language model (LLM) inference under GPU memory constraints, this paper proposes a memory-aware and SLA-driven dynamic batching method. The approach continuously monitors GPU memory utilization and per-request latency feedback, enabling runtime adaptation of batch sizes via fine-grained resource modeling and elastic scheduling—thereby strictly satisfying service-level agreement (SLA) constraints. Compared to static batching, our method achieves 8–28% higher inference throughput while maintaining low latency, increases service capacity by 22%, and remains fully compatible with mainstream inference frameworks. Its core innovation lies in unifying memory-state awareness, latency-feedback control, and hard SLA constraint modeling within a single dynamic batching framework—yielding an efficient, robust, and production-deployable online optimization solution.

Hardware adaptation bottlenecks hinder efficient LLM deployment at the edge. Method: We systematically survey and uniformly benchmark state-of-the-art inference acceleration techniques—including pruning, quantization, KV cache optimization, and operator fusion—across CPU, GPU, FPGA, ASIC, and processing-in-memory (PIM) platforms, using identical models and methods under batch sizes of 1 and 8. Our evaluation quantifies throughput (tokens/s) and energy efficiency (tokens/J). Contribution/Results: ASIC and PIM architectures achieve substantially higher energy efficiency than conventional platforms. Based on empirical findings, we identify three key evolutionary directions for edge AI: native multimodal support, dynamic runtime computation scheduling, and orders-of-magnitude improvement in inference capability per unit energy. This work provides empirically grounded guidance for hardware selection and hardware-software co-optimization of LLMs in resource-constrained edge environments.

Deploying large language models (LLMs) on edge devices—such as smartphones—is hindered by high memory footprint and slow inference latency. To address these bottlenecks, we propose an on-device efficient inference framework featuring a novel DRAM-Flash hybrid memory architecture and a mobile CPU/GPU-coordinated dynamic weight-input reordering strategy. Our method tightly integrates multiple optimization techniques: post-training quantization, mixed-precision floating-point computation, multi-core load balancing, geometric computation optimization, and instruction-set-aware weight layout customization. These synergistic optimizations significantly improve hardware utilization and computational efficiency. Experimental results demonstrate up to 8.6× speedup over state-of-the-art LLM inference frameworks, alongside substantial memory reduction. The framework enables real-time execution of mainstream open-weight models—including LLaMA and Phi—on both Android and iOS platforms.

Existing service systems deploy LLM inference and parameter-efficient fine-tuning (PEFT)—e.g., LoRA—on isolated GPU clusters, leading to significant resource waste and low utilization. This paper introduces the first end-to-end co-serving framework enabling inference and PEFT to coexist on a single GPU. Our approach combines static compilation optimizations—including token-level compute fusion, dependency parallelization, and computation graph pruning—with a GPU memory-aware runtime, hybrid token scheduling, and dynamic batching. This joint design simultaneously guarantees low-latency inference (meeting a 20 req/s SLO) and high-throughput fine-tuning (1.9–6.8× throughput improvement). Experiments demonstrate up to 80% GPU memory reduction; under peak load on models such as LLaMA-3.1-8B, the system sustains over 76% fine-tuning progress while maintaining inference responsiveness.

This work addresses the severe inefficiency in long-context large language model (LLM) inference, where memory management overhead accounts for 22%–97% of total execution time and exhibits highly heterogeneous computational characteristics. The study presents the first systematic modeling of the LLM memory processing pipeline, unifying optimizations such as sparse attention and retrieval-augmented generation into a unified four-stage pipeline. It further introduces an innovative GPU-FPGA heterogeneous architecture that offloads sparse, irregular, and memory-intensive operations to the FPGA while retaining compute-intensive tasks on the GPU. Evaluated on AMD MI210 and Alveo U55C platforms, the proposed approach achieves 1.04–2.2× end-to-end speedup and reduces energy consumption by 1.11–4.7× compared to GPU-only baselines, thereby overcoming the limitations of conventional homogeneous acceleration.

This work addresses the high latency and kernel launch overhead that hinder large language models (LLMs) in short-sequence interactive inference. The authors propose a hybrid runtime framework that, for the first time, synergistically integrates just-in-time (JIT) compilation with dynamic CUDA Graph execution for LLM inference. During autoregressive decoding, the Transformer computation is partitioned into static components—replayed via CUDA Graphs—and dynamic components—handled by JIT-compiled kernels—while supporting asynchronous graph capture and cross-step reuse. This approach effectively balances low launch overhead with runtime flexibility. Evaluated on LLaMA-2 7B with batch size 1, the method reduces first-token latency by up to 66.0% and achieves better P99 latency than TensorRT-LLM.

Existing work lacks a systematic understanding of large language model (LLM) inference behavior on GPUs. Method: This paper introduces the first four-dimensional analytical framework—spanning two-phase computational heterogeneity, microarchitectural-level performance root causes, system-scale scaling laws, and boundaries of emerging inference paradigms—validated via large-scale empirical measurement, deep GPU microarchitectural analysis, fine-grained performance modeling, and cross-architecture scalability evaluation across mainstream GPUs (A100/H100) and LLMs (7B–70B). Contribution/Results: The study uncovers latent bottlenecks between attention computation and memory access, identifies previously unrecognized critical constraints, and establishes hardware-aware theoretical performance bounds and deployable optimization strategies. It fills a fundamental gap in system-level LLM inference analysis and enables the design of efficient, scalable next-generation inference systems.

This work addresses the inefficiency of sampling in diffusion-based large language models (dLLMs) on conventional GEMM-centric NPUs, where high memory overhead and irregular memory access patterns lead to sampling latency accounting for up to 70% of total inference delay. The study systematically identifies, for the first time, the essential non-GEMM instruction set required for dLLM sampling and proposes a sampling-oriented NPU microarchitecture that departs from the GEMM-centric paradigm. By integrating lightweight vector primitives, in-place memory reuse, and a decoupled mixed-precision memory hierarchy, the design significantly improves sampling efficiency. Evaluated under equivalent process technology, the proposed architecture achieves up to 2.53× speedup over an NVIDIA RTX A6000 GPU. To ensure reproducibility and functional correctness, the authors open-source a cycle-accurate simulator and RTL implementation.

This work addresses the high computational and memory costs of large autoregressive language models, which stem primarily from the dense and parameter-heavy feedforward layers. To mitigate this, the authors propose an efficient sparse computation framework tailored for modern GPU architectures. By applying L1 regularization to induce unstructured sparsity, and coupling it with a custom CUDA kernel and a novel sparse packing format, the method achieves over 99% sparsity in feedforward layers while preserving near-lossless performance on downstream tasks. The approach substantially improves throughput, energy efficiency, and memory utilization, with benefits amplifying as model scale increases.