Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference

2026-06-22

AI Summary

This work addresses the inefficiency of existing fault-tolerance mechanisms for large language model (LLM) agents, which rely on full restarts or application-level checkpointing and struggle to recover critical GPU state—such as KV caches and scheduler metadata—efficiently. The authors propose a device-resident, persistent kernel implementation that enables fine-grained, low-overhead checkpointing and recovery directly within the GPU’s native execution context, thereby avoiding CPU bottlenecks. By injecting incremental checkpoint logic into the PTX/SASS layer via JIT compilation, and combining GPU module load interception, lock-free ring buffers, and CXL/host-memory logging, the approach achieves framework-agnostic, transparent fault tolerance. It supports efficient dirty-page detection and incremental snapshots of key structures like KV caches and adapter pages, substantially reducing work loss and recovery latency upon failure.

Abstract

Long-running LLM agents keep valuable state resident on GPUs: KV caches, request schedulers, communication state, and sometimes online adapters. Losing this state after a GPU or communicator failure can discard minutes to hours of work, yet existing recovery mechanisms either restart the whole serving stack or require application-specific checkpoint logic inside every attention and runtime component. This paper argues that fault tolerance for such workloads needs a GPU-resident execution context: checkpoint hooks must run at device synchronization points, observe binary kernels that frameworks and libraries actually execute, and recover without putting the host CPU on the critical path. We present Concordia, a runtime that uses a device-resident persistent kernel as the substrate for fault-tolerant LLM inference. Concordia interposes on GPU module loading and supports PTX- and SASS-level instrumentation, allowing checkpoint and pause hooks to be inserted below framework code and library boundaries. For each registered LLM state region, Concordia JIT-compiles a specialized delta-checkpoint handler -- for example, a KV-block scanner, adapter-page scanner, or recovery applier -- and hot-swaps it into the persistent kernel's operator table. The persistent kernel consumes a lock-free ring buffer of compute, checkpoint, append-log, and recovery tasks, so the same always-on executor triggers dirty-page detection, stages deltas, and appends committed records to a CPU-visible log in CXL memory or host DRAM.

Problem

• fault tolerance
• LLM inference
• GPU-resident state
• checkpointing
• persistent kernel

Innovation

• persistent-kernel
• JIT-compilation
• checkpointing
• fault-tolerant LLM inference
• GPU-resident execution