LUMEN: Coordinated Failure Recovery for Distributed LLM Serving

2026-06-16

AI Summary

This work addresses the challenge of fault tolerance in distributed large language model serving, where worker node failures lead to loss of key-value (KV) cache and request interruption. Existing recovery mechanisms suffer from low efficiency and neglect real-time cluster load conditions. To overcome these limitations, this study formulates fault recovery as a cross-stage, load-aware cooperative optimization problem that jointly optimizes checkpoint placement, reassignment of interrupted requests, and service capacity restoration during model reloading. The authors introduce a distributed KV cache management scheme, a load-aware scheduling algorithm, and a parallelized service capacity recovery mechanism. Experimental results demonstrate that the proposed approach significantly reduces both service latency and fault recovery time compared to state-of-the-art baselines.

Abstract

Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a worker fails, the cluster simultaneously loses the failed worker's GPU-resident key-value (KV) caches and serving capacity, leaving surviving workers to absorb the redirected traffic while re-running interrupted requests from scratch. Existing fault-tolerant systems either restart interrupted requests from scratch or restore KV caches from checkpoints stored on a fixed neighboring worker, but both approaches route recovery work without considering current cluster load and leave the recovering worker idle during model reload. We present LUMEN, a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. We evaluate LUMEN using both prototype experiments and large-scale simulations and demonstrate significant improvements in serving and recovery times.

Problem

• distributed LLM serving
• failure recovery
• KV cache
• load-aware coordination
• fault tolerance

Innovation

• fault tolerance
• distributed LLM serving
• load-aware coordination
• KV cache recovery
• checkpoint placement