runtime systems

This work addresses the challenge that existing AI systems struggle to dynamically observe, intervene in, and optimize agent behavior at runtime, making it difficult to simultaneously achieve high task success rates, low latency, token efficiency, reliability, and safety. To overcome this limitation, the paper proposes a novel runtime infrastructure layer situated between the model and the application, which treats the AI execution process itself as an optimizable object—departing from conventional approaches that restrict optimization to static model or log-level adjustments. This layer enables proactive intervention and multi-dimensional performance co-optimization through mechanisms such as runtime monitoring, real-time inference, adaptive memory management, fault recovery, and policy enforcement. Experimental results demonstrate that the proposed approach significantly enhances the holistic performance of long-horizon agent workflows across task success rate, response latency, token efficiency, system reliability, and safety compliance.

This work addresses the challenges posed by dynamic AI models—whose varying tensor shapes and control flows hinder existing compilers from simultaneously achieving fast compilation, low memory overhead, and effective optimization. To overcome these limitations, the authors propose DVM, a just-in-time compiler that introduces a novel bytecode-based execution mechanism directly targeting NPUs. DVM employs a bytecode virtual machine to efficiently compile dynamic operator instances and integrates symbolic shape inference from static graphs with runtime fusion strategies from dynamic graphs, enabling on-the-fly operator compilation and multi-granularity fusion. Experimental results demonstrate that DVM achieves up to an 11.77× speedup in operator and model execution compared to TorchInductor, PyTorch-eager, and MindSpore-graph-O0, while reducing compilation time by up to five orders of magnitude.

This study systematically investigates the impact of deep learning runtime configurations—specifically engine–execution provider pairings (e.g., PyTorch vs. ONNX Runtime, CUDA vs. CPU)—on energy efficiency, latency, and resource utilization during inference for small language models (SLMs) specialized for code. Leveraging a standardized benchmarking framework, we evaluate 12 open-source code SLMs using RAPL-based power measurement, system-level performance counters, and a unified inference API. To our knowledge, this is the first empirical, cross-engine, cross-provider energy-efficiency comparison for code SLMs. Results show that PyTorch with CUDA delivers the best overall trade-off: it reduces energy consumption by 37.99%–89.16% versus all other configurations while achieving lower latency and higher GPU utilization. For CPU-only deployment, ONNX Runtime with CPU execution improves energy efficiency by 8.98%–72.04% over comparable CPU-based alternatives. The study provides reproducible, evidence-based guidelines for energy-aware SLM deployment in production environments.

Key computational kernels in the Python tensor library pyttb—such as accumulation, tensor multiplication, and low-rank decomposition—suffer from suboptimal runtime performance. Method: This work introduces, for the first time, a systematic integration of Rust to develop high-performance Python extension modules via the Python C API, leveraging Rust’s memory safety and zero-cost abstractions to optimize deeply nested loops and large-scale tensor operations. Contribution/Results: Empirical evaluation on synthetic datasets across multiple problem scales demonstrates consistent and substantial speedups over pure Python, Numba JIT-compiled code, and NumPy-based implementations—reaching up to several-fold acceleration. Performance gains are especially pronounced for loop-intensive kernels. The proposed Rust–Python co-design paradigm provides a reusable, safe, and efficient optimization framework for scientific computing libraries.

Integrating custom hardware accelerators—particularly GEMM-based ones—into mainstream ML compilers remains challenging due to tight coupling between accelerator-specific optimizations and compiler internals. Method: This paper proposes a high-level, low-intrusion integration methodology for TVM that abstracts hardware scheduling interfaces to decouple accelerator characteristics from compiler implementation. Leveraging the CoSA design-space exploration framework, it automates hardware-aware scheduling optimizations—including tensor tiling, non-uniform mapping, and double buffering—without modifying TVM’s core infrastructure. Contribution/Results: Evaluated on the Gemmini accelerator, the approach achieves performance on par with hand-optimized toolchains while significantly improving developer productivity and cross-model/cross-architecture portability. The abstraction enables seamless reuse of scheduling policies across diverse accelerator microarchitectures and neural network workloads, reducing integration effort from weeks to hours.

This work addresses the longstanding challenge in GPU programming of simultaneously achieving high productivity, portability, and performance: low-level models offer high performance but suffer from poor developer productivity, while high-level abstractions improve productivity at the cost of performance. To bridge this gap, the paper proposes nomp, a novel framework that integrates a pragma-based programming model with a metadata-driven runtime system to enable domain-specific code transformation and generation. By incorporating domain-specific optimization patterns, nomp significantly enhances programming productivity while preserving high performance and cross-platform portability, effectively reconciling the trade-offs between low-level efficiency and high-level abstraction in GPU programming.

Current large language model inference systems lack the capability to dynamically adjust model parallelism topologies at runtime, necessitating service restarts under varying workloads and causing multi-minute disruptions, loss of KV cache, and substantial recomputation overhead. This work proposes ReMP, the first framework enabling online elastic reconfiguration of combined tensor and pipeline parallelism. By decoupling topology from execution state, designing a two-dimensional KV cache migration mechanism, and orchestrating an end-to-end reconfiguration pipeline, ReMP reduces topology switching latency to 1–7 seconds across 7B–70B models—orders of magnitude faster than restarting. This significantly improves time-to-first-token (TTFT), time per output token (TPOT), and throughput under dynamic workloads.

Existing scientific computing codes are difficult to efficiently port to specialized architectures such as AMD AI Engine, often requiring extensive manual refactoring. This work proposes a tensor abstraction–based compilation approach that automatically elevates generic loops to tensor semantics by parsing lightweight OpenMP annotations, and constructs an end-to-end compilation pipeline to map computations onto the AI Engine execution model. The method significantly reduces programming complexity through minimal OpenMP directives and enables CPU–NPU cooperative scheduling. Experimental results on six scientific and AI kernel benchmarks show that the NPU achieves higher energy efficiency than a multi-core CPU at float32 precision; for two kernels, cooperative execution yields a 40% performance improvement and 15% energy reduction.

This work addresses the significant degradation in inference throughput caused by GPU memory constraints when concurrently deploying multiple large language models on shared heterogeneous hardware, where resource scheduling, model offloading, and preemption become critical bottlenecks. Through empirical methodologies—including cross-platform performance profiling, layer-wise offloading experiments, and fine-grained decomposition of preemption overhead—the study systematically uncovers, for the first time, the nonlinear relationship between offloading and throughput decline. It further identifies model state reloading as the primary source of preemption overhead. The findings reveal that smaller models are more sensitive to reduced GPU residency, and that such overhead is jointly influenced by model architecture and hardware characteristics. These insights motivate a scheduler design that integrates model-specific sensitivity with data migration costs, offering crucial guidance for building efficient multi-model serving systems.