LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs

2026-06-22

AI Summary

This work addresses the inefficiencies in existing omnimodal large model serving systems, which neglect the interplay between audio playback progress and KV cache reuse across multi-turn dialogues, leading to increased response latency, redundant generation, and premature eviction of critical states. To overcome these limitations, we propose the first interaction-aware serving system that integrates user interaction signals—such as playback position, voice activity, and interruption events—into scheduling and KV cache management. Our approach introduces three key innovations: playback-frontier-constrained generation, near-underload-priority scheduling, and a KV preloading and eviction strategy guided by next-turn usage prediction. Implemented atop the vLLM-Omni framework, our system reduces the P90 time-to-first-audio-token by 1.55× on average (up to 2.21×) across diverse workloads, improves completed-request throughput by 1.15× (up to 1.56×), and shifts the majority of KV reloading out of the critical path of subsequent turns.

Abstract

Realtime omni-modal LMs support speech-centric conversations where users stream inputs, hear generated audio, and interrupt freely. Existing Omni-LM serving systems still rely on throughput-oriented LLM scheduling and LRU KV offloading. These policies ignore audio playback and multi-turn reuse: they may generate tokens far beyond what users hear, wasting work after barge-in, and evict KV state needed in the next turn. LiveServe is an interaction-aware serving system for realtime Omni-LM interaction. It exposes playback progress, speech activity, and barge-in events to the serving pipeline. The scheduler prioritizes first-audio and near-underrun sessions while limiting generation beyond the playback frontier. The KV manager uses next-use-aware eviction and preloads likely-needed KV during user speech to hide reload latency. On vLLM-Omni, LiveServe improves realtime serving across two Omni-LMs and mixed workloads. It lowers P90 audio TTFP by $1.55\times$ on average and up to $2.21\times$, while improving completed-request throughput by $1.15\times$ on average and up to $1.56\times$, and moves most KV reload work off the next-turn critical path.

Problem

• real-time omni-modal LLMs
• interaction-aware serving
• KV cache eviction
• audio playback
• barge-in

Innovation

• interaction-aware serving
• real-time omni-modal LLMs
• KV cache management
• barge-in handling
• playback-aware scheduling