kserve

Traditional threshold-driven reactive autoscaling struggles to handle dynamic workloads, heterogeneous environments, and latency-sensitive applications, often leading to resource imbalance and performance degradation. This work proposes a predictive autoscaling framework that integrates drift awareness, uncertainty quantification, and privacy-preserving mechanisms, enabling proactive and adaptive scheduling in cloud-edge协同 environments through Kubernetes Custom Resource Definitions (CRDs) and a MAPE (Monitor-Analyze-Plan-Execute) control loop. The core contributions include a comprehensive taxonomy encompassing triggering mechanisms, target entities, prediction models, and evaluation metrics; the formulation of an Autoscaling Drift Index (ADI); and the integration of federated learning, container isolation, and feedback correction techniques. Together, these advances establish a theoretical foundation and key technical pathways for autoscaling in cloud-native and cloud-edge federated systems.

To address memory reliability challenges in large-scale AI training on Kubernetes—including OOM kills, over-allocation, memory leaks, and ephemeral storage exhaustion—this paper proposes the first memory governance framework tailored for ML workloads. Methodologically, it introduces a GPU-aware memory quota policy that jointly constrains GPU memory and system memory; designs a dynamic, cgroup v2–based elastic reclaim mechanism for ephemeral storage; and integrates a Prometheus/Grafana observability stack with a custom Eviction Advisor. Experiments on real-world, thousand-GPU distributed training clusters demonstrate a 92% reduction in OOM incidents, a 37% increase in GPU memory utilization, and SLA compliance exceeding 99.5% for training jobs. The core contribution lies in unifying memory QoS enforcement, GPU–system memory coupling modeling, and elastic ephemeral storage management within Kubernetes’ native scheduling architecture—marking the first such holistic approach.

To address the scalability lag and poor burst-load responsiveness in serverless LLM inference—caused by high model loading overhead—this paper proposes λPipe, a distributed inference architecture integrating RDMA-accelerated multicast with an execute-while-load mechanism. Its core contributions are threefold: (1) an adaptive multi-node pipelined scheduler enabling hierarchical model management across GPU and host memory, along with real-time collaborative inference; (2) heterogeneous memory–aware dynamic model loading; and (3) deep optimizations of the serverless runtime. Evaluated on realistic LLM inference traces, λPipe reduces tail latency by up to 5× and lowers service cost by 31.3% compared to baseline approaches, significantly improving both responsiveness under bursty workloads and resource efficiency.

This study addresses the substantial memory pressure and performance trade-offs in large language model (LLM) inference caused by KV cache growth with increasing context length and concurrent requests. The authors systematically evaluate three state-of-the-art KV cache management frameworks—vLLM, InfiniGen, and H2O—across diverse workloads, model scales, and sparsity conditions, measuring their latency, throughput, and memory efficiency. By integrating key techniques such as tensor offloading, token eviction heuristics, and speculative scheduling, the work provides the first comprehensive characterization of the operational boundaries of these strategies under varied deployment scenarios. It identifies optimal configurations under specific memory and performance constraints, offering empirical insights and practical guidance for designing efficient LLM inference systems.

This work addresses the challenges of high manual development costs and poor cross-platform portability in generating high-performance kernels for heterogeneous AI accelerators. The authors propose an iterative optimization framework based on two collaborative large language model (LLM) agents, which alternately perform functional correctness verification and performance tuning to enable reference-free, cross-backend kernel generation. By integrating compiler feedback, performance profiling, operator fusion, and mixed-precision execution, the method achieves a 2.12% improvement in inference throughput on the NVIDIA B200 and delivers an average 5.13× speedup over Triton kernels across 37 GEMM-plus-epilogue tasks on the Intel Arc B580.

This paper addresses the deployment of multi-model inference pipelines on resource-constrained edge devices. We propose an end-to-end adaptive configuration framework that, for the first time, explicitly incorporates device resource constraints into joint optimization decisions. Our method integrates residual feature extraction, LSTM-based workload forecasting, and policy-gradient reinforcement learning to jointly optimize QoS guarantees (e.g., latency and throughput), operational cost, and real-time adaptability. Evaluated on a real Kubernetes-based edge cluster, the framework achieves a 27% reduction in average inference latency, a 31% increase in throughput, a 22% decrease in deployment cost, and over 42% faster configuration decision-making for complex pipelines—outperforming state-of-the-art baselines. The core contributions are: (i) a resource-aware joint optimization model that unifies hardware constraints with pipeline scheduling and scaling decisions; and (ii) a lightweight, learning-driven configuration mechanism enabling efficient, online adaptation under dynamic edge conditions.

This work addresses the efficient support of generative AI inference workloads on Kubernetes, balancing performance and resource efficiency across both batch and online scenarios. We design a multi-stage inference pipeline that integrates automatic speech recognition (Whisper) with large language model summarization, and for the first time, cohesively combines Kueue, Dynamic Accelerator Slicing (DAS), and Gateway API Inference Extension (GAIE) to enable unified scheduling and high-performance execution. Experimental results demonstrate substantial improvements in system efficiency: Kueue reduces total job completion time by up to 15%, DAS decreases average job completion time by 36%, and GAIE accelerates time-to-first-token by 82%.

This work addresses the challenge of deploying mainstream deep learning frameworks in resource-constrained environments, where their large size and lack of lightweight yet fully featured alternatives pose significant limitations. To this end, we propose and implement a lightweight tensor computation library built in Rust, leveraging its performance and memory safety guarantees to construct an efficient computational engine. The system exposes a PyTorch-like Python interface via PyO3, supporting essential features including n-dimensional tensors, dynamic computation graphs, reverse-mode automatic differentiation, neural network layers, and optimizers. The resulting installable package occupies only a few megabytes—orders of magnitude smaller than PyTorch or TensorFlow—while retaining the core capabilities necessary for research and development on CPU-based systems.

This work addresses the challenge of generating accurate domain-specific code—such as Kubernetes YAML—with small language models (SLMs) by proposing a context-guided data distillation approach. Instead of relying on conventional KL divergence–based distillation, the method synthesizes training data through forward generation and reverse instruction construction, then applies a dual filtering criterion based on external validation and contextual consistency to enforce strict format constraints critical for output quality. Training data is generated via the DeepSeek-V4 Flash API, and the Qwen2.5-Coder-1.5B-Instruct model is fine-tuned using LoRA on CPU. Evaluated on the K8s-Distill-Pilot test set, the approach achieves a full-pass@1 accuracy of 91.5%, demonstrating that precise adherence to syntactic and structural constraints yields greater gains than merely scaling up dataset size.

This work addresses the inefficiency in existing agent reasoning systems, which lack end-to-end workflow awareness across multi-turn interactions, leading to suboptimal KV cache utilization and tool resource management. To overcome this limitation, the paper introduces, for the first time, an LLM program abstraction that unifies agent workflows into a heterogeneous resource view. Building upon this abstraction, the authors design a program-aware scheduler and a tool resource manager that jointly orchestrate KV caches, system states, and external tools. The proposed framework enables asynchronous environment preloading and memory load balancing, achieving 1.5–3.6× higher inference throughput on coding, routing, and scientific discovery tasks, accelerating reinforcement learning rollouts by 1.8–3.9×, and reducing disk memory usage by up to 4.2×.

This work addresses customs clearance delays in global trade caused by ambiguous product descriptions and frequent updates to Harmonized System (HS) codes. To tackle this challenge, the authors propose a serverless MLOps framework that leverages event-driven pipelines and managed services to enable end-to-end, model-agnostic machine learning lifecycle management. The architecture supports automatic scaling, reproducible training, auditable deployment, and automated A/B testing, ensuring secure and seamless model transitions. By integrating custom text embeddings with models such as Text-CNN, the system achieves 98% accuracy on real-world HS code prediction tasks, meeting stringent service-level agreement (SLA) requirements. This approach significantly reduces long-term operational costs and establishes an efficient, cost-effective, and reproducible deployment paradigm for industrial-scale machine learning systems.