Kavier: Exploring Performance, Sustainability, and Efficiency of LLM Ecosystems under Inference through Cache-Aware Discrete-Event Simulation

2026-05-24

AI Summary

This work addresses the absence of a systematic model and unified reference architecture in the current large language model (LLM) inference ecosystem, which hinders accurate prediction of performance, efficiency, and sustainability. To bridge this gap, the paper introduces the first reference architecture for LLM inference ecosystems and presents Kavier, the first cache-aware discrete-event simulation framework capable of efficiently modeling and evaluating system behavior under diverse key-value and prefix caching strategies. Validated against real-world traces, Kavier delivers highly accurate and scalable predictions across dimensions of latency, throughput, and cost, thereby offering critical support for the design, optimization, and operation of LLM inference systems.

Abstract

Large Language Models (LLMs) are widely used by our increasingly digitalized society, but raise sustainability, performance, and financial concerns, especially as inference workloads grow. To improve the design and operation of LLM ecosystems, we envision simulators and simulation-based digital twins becoming primary decision-making tools. LLM ecosystems leverage many heterogeneous components, making simulation a non-trivial, yet critical operation. The simulation challenge is exacerbated by the absence of a comprehensive reference architecture of LLM ecosystems; the lack of such a conceptual model can be costly and could misguide the designers and engineers. Without a reference architecture, even the most experienced stakeholders could tinker in researching, engineering, or maintaining LLM ecosystems. In this work, we bring a three-fold contribution to the scientific community. Firstly, we synthesize, propose, and validate a reference architecture (RA) of LLM ecosystems under inference. Then, adhering to the reference architecture, we design Kavier, the first simulation instrument able to predict the performance, sustainability, and efficiency of LLM ecosystems under inference, through discrete-event and cache-aware simulation, focusing on Key-Value-(KV-)Caching and prompt prefix caching policies. Through experiments with a Kavier prototype and real-world traces, (i) we measure the accuracy of Kavier and its performance in massive-scale simulations, (ii) we compare the performance of different KV-Caching policies, and (iii) we analyze the performance, sustainability, and efficiency of LLM ecosystems under various prefix caching policies. Overall, we show that Kavier enables operators, researchers, and engineers to predict LLM ecosystems in a time, performance, and cost-efficient way.

Problem

• LLM ecosystems
• reference architecture
• inference
• sustainability
• performance

Innovation

• discrete-event simulation
• cache-aware
• LLM ecosystem
• KV-Caching
• reference architecture