SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference

2025-04-22

AI Summary

To address high tail latency, resource underutilization, and SLO violations in multi-LLM GPU sharing—caused by neglecting autoregressive generation characteristics and latency sensitivity—this paper proposes a service-aware latency optimization framework. Methodologically, it introduces: (1) a latency-aware dynamic scheduling algorithm that explicitly models per-token generation steps and end-to-end latency constraints; (2) load-predictive dynamic model placement coupled with adaptive KV cache eviction; and (3) a unified, cross-model KV cache architecture enabling fine-grained memory reuse. Evaluated under realistic LLM workloads, the framework achieves up to 13.60× reduction in normalized latency, 18.69× lower P99 latency, and 3.64× higher SLO compliance rate compared to state-of-the-art approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) with different architectures and sizes have been developed. Serving each LLM with dedicated GPUs leads to resource waste and service inefficiency due to the varying demand of LLM requests. A common practice is to share multiple LLMs. However, existing sharing systems either do not consider the autoregressive pattern of LLM services, or only focus on improving the throughput, which impairs the sharing performance, especially the serving latency. We present SeaLLM, which enables service-aware and latency-optimized LLM sharing. SeaLLM improves the overall sharing performance by (1) a latency-optimized scheduling algorithm utilizing the characteristics of LLM services, (2) a placement algorithm to determine the placement plan and an adaptive replacement algorithm to decide the replacement interval, and (3) a unified key-value cache to share GPU memory among LLM services efficiently. Our evaluation under real-world traces and LLM services demonstrates that SeaLLM improves the normalized latency by up to $13.60 imes$, the tail latency by up to $18.69 imes$, and the SLO attainment by up to $3.64 imes$ compared to existing solutions.

Problem

• Optimizes resource sharing for diverse LLM architectures and sizes
• Reduces latency and improves efficiency in LLM service scheduling
• Enhances GPU memory sharing via unified key-value cache

Innovation

• Latency-optimized scheduling for LLM services
• Adaptive placement and replacement algorithms
• Unified key-value cache for GPU memory sharing