Optimizing LLM Inference for Database Systems: Cost-Aware Scheduling for Concurrent Requests

2024-11-12

AI Summary

To address performance bottlenecks—such as high GPU resource overhead, low throughput, and elevated latency—caused by concurrent requests in database-embedded LLM inference, this work pioneers the adaptation of database multi-query optimization principles to LLM inference systems. We propose a fine-grained GPU memory cost model that jointly characterizes VRAM occupancy and memory bandwidth constraints, along with a cooperative scheduling strategy integrating batch-aware scheduling and dynamic priority reordering. Evaluated against baseline approaches, our method achieves up to 47% higher inference throughput, 32% lower average latency, and 32% reduced GPU resource consumption—all without compromising model accuracy. The core contribution lies in cross-paradigm transfer of database optimization techniques to LLM inference, enabling resource modeling, schedulability optimization, and performance predictability—thereby establishing a principled foundation for efficient, scalable, and predictable LLM inference within database systems.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

LLMs are increasingly used inside database systems and in database applications for better complexity management and decision-making, where LLM inferences require significant GPU costs. LLM inference systems, however, are slow compared to database systems, limiting the expansion of the use of LLMs inside database systems. This paper first analyzes the LLM inference performance and focuses on a data management issue in LLM inference. We reveal that the root of the problem is the lack of an adequate resource cost model and optimization strategy when executing multiple concurrent inference requests. We adapt classic database multi-query optimization techniques by introducing cost models for concurrent inference requests and new scheduling strategies to optimize the use of memory resources by concurrent requests, thereby substantially improving performance.

Problem

• Optimizing GPU costs for LLM inference in databases
• Improving slow LLM inference performance in database systems
• Developing cost models and scheduling for concurrent LLM requests

Innovation

• Cost models for concurrent LLM inference requests
• Scheduling strategies for memory optimization
• Multi-query optimization techniques adaptation