🤖 AI Summary
To address the high deployment barrier and challenging performance tuning of large language model (LLM) inference across heterogeneous hardware and multiple deep learning frameworks, this paper proposes a simulation-driven dynamic modeling and optimization approach. We introduce a globally unified inference engine architecture and develop a dynamic performance model that captures hardware-framework-workload coupling, enabling the first systematic quantification of cross-layer bottlenecks. A hybrid offline/online simulation mechanism is designed to predict multi-dimensional metrics—including batch latency, time-to-first-token, and decoding throughput—with prediction errors ranging from 9.9% to 42.3%. Furthermore, we implement heterogeneous resource-aware scheduling, significantly improving memory utilization and multi-GPU throughput stability. Our method enables zero-code, efficient LLM deployment across vendor-diverse hardware and mainstream frameworks, substantially lowering the adoption barrier for non-expert users.
📝 Abstract
Efficiently deploying large language models (LLMs) in real-world scenarios remains a critical challenge, primarily due to hardware heterogeneity, inference framework limitations, and workload complexities.Efficiently deploying large language models (LLMs) in real-world scenarios remains a critical challenge, primarily due to hardware heterogeneity, inference framework limitations, and workload complexities. These challenges often lead to inefficiencies in memory utilization, latency, and throughput, hindering the effective deployment of LLMs, especially for non-experts. Through extensive experiments, we identify key performance bottlenecks, including sudden drops in memory utilization, latency fluctuations with varying batch sizes, and inefficiencies in multi-GPU configurations. These insights reveal a vast optimization space shaped by the intricate interplay of hardware, frameworks, and workload parameters. This underscores the need for a systematic approach to optimize LLM inference, motivating the design of our framework, GUIDE. GUIDE leverages dynamic modeling and simulation-based optimization to address these issues, achieving prediction errors between 9.9% and 42.3% for key metrics such as batch latency, TTFT, and decode throughput. By effectively bridging the gap between theoretical performance and practical deployment, our framework empowers practitioners, particularly non-specialists, to make data-driven decisions and unlock the full potential of LLMs in heterogeneous environments cheaply.