Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN

2026-08-17

AI Summary

This study addresses service disruptions caused by the decoupling of LLM inference states during AI-RAN handovers. We propose a proactive KV cache migration framework that introduces a novel pre-handover parallel preparation mechanism. By integrating sequence segmentation, local prefilling, and streaming transmission, this approach enables seamless recovery of inference states at the target node. Furthermore, mobility prediction and online scheduling are employed to dynamically optimize prefetching windows. Extensive evaluations on a vLLM prototype demonstrate that our method reduces service interruption time by 2.28–89.68× and decreases inter-token latency by 16%–50%. These results confirm significant improvements in LLM service continuity within mobile environments, effectively mitigating state separation issues inherent to dynamic network switching.

Abstract

AI-RAN brings large language model (LLM) serving close to mobile users, but cellular handover can separate an active request from its inference state: the user attaches to a target base station (gNB) while the large and growing key-value (KV) cache remains at the source. Retaining inference at the source preserves service continuity but persistently increases inter-token latency (ITL), whereas recovering the state at the target restores serving locality but requires KV-cache transfer, recomputation, or a combination of both only after handover, directly prolonging service interruption time (SIT). This work presents Pallas, a \textit{proactive} KV-cache migration framework that prepares the inference state at the predicted target before handover, in parallel with ongoing source-side inference and token delivery. At the preparation trigger, Pallas partitions the token sequence into a stable historical prefix and an evolving suffix. The target reconstructs the prefix through local prefill, while the source streams the KV blocks generated for the suffix. At handover, the target assembles both portions into an up-to-date KV cache and resumes decoding locally, leaving only unfinished preparation to contribute to SIT. An online scheduler selects the \textit{prefetching window}, which determines how early preparation begins before handover, based on mobility predictions and runtime telemetry. Across three LLMs and $100$--$500~\mathrm{Mbps}$ inter-gNB links, our vLLM-based prototype reduces average SIT by factors of $2.28$--$89.68$ over target-side recovery approaches and lowers average ITL by $16.0\%$--$50.0\%$ compared with source-side forwarding.

Problem

• AI-RAN
• LLM Inference
• KV Cache Migration
• Cellular Handover
• Service Interruption Time

Innovation

• Proactive KV Cache Migration
• AI-RAN
• Handover-aware Scheduling
• Prefix-Suffix Partitioning
• LLM Inference Continuity