Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving

2026-07-02

AI Summary

This work addresses the metastable failure of real-time interactive models under sustained load, caused by unbounded growth of KV cache—a phenomenon poorly captured by conventional latency metrics. We uncover this failure mechanism and propose a cache-bounded approach that enforces a fixed-window KV cache per session, transforming frame-level latency into a monotonic load signal. By integrating an in-engine windowing mechanism, online admission control, a first-order crash-time prediction model, and quality probes, our method enables end-to-end stable scheduling. Evaluated across four interactive models, the system reduces crash incidence to 0 out of 20 trials (versus 14/20 for the baseline), achieves crash-time prediction errors within a few percentage points, and preserves output quality without degradation.

Abstract

Real-time interaction models -- Moshi, MiniCPM-o, Qwen-Omni -- turn serving into a periodic real-time task: on every frame a session ingests streaming audio and must respond by a recurring wall-clock deadline, while its KV cache grows monotonically and stays pinned for the whole conversation. This regime hides a dangerous failure mode. On a real full-duplex stack, sustained load does not degrade serving gracefully: it falls off a cliff, jumping in one step from milliseconds per frame to a stalled engine when accumulated session state exhausts the KV pool. The collapse is metastable -- identical five-minute runs collapse or survive on run-to-run variance -- and silent: latency and deadline-miss metrics read healthy throughout. We show one move restores both stability and observability: bound each session's resident state, and latency starts telling the truth. Metronome's in-engine KV window eliminates the collapse (0/20 vs. 14/20 runs across two batches) and turns per-frame latency into a monotone load signal, on which an online admission controller discovers the schedulable concurrency; without the window, the identical controller over-admits into the wall. A first-order model predicts the collapse time within a few percent on the headline model, and a quality probe validates the bound's design by ablation: the window alone is quality-free in turn-based decoding, and its few pinned attention-sink tokens are what keep free-running generation healthy. Everything is measured end-to-end on real audio, across four interaction models on one GPU.

Problem

• real-time interaction
• KV cache
• service collapse
• latency observability
• serving stability

Innovation

• KV cache bounding
• real-time interaction models
• metastable failure
• online admission control
• attention sink