LOCAL: Enabling Learning On-device Contiguously for Agent LLMs

2026-08-15

AI Summary

This study addresses the challenge of concurrently supporting inference and continuous privacy-preserving learning for on-device LLM agents on a single GPU. We propose the first single-GPU runtime system enabling online learning, which employs co-scheduling, version-aware KV caching, and multi-agent state sharing to overcome resource isolation barriers and achieve global optimization of inference and training. Experimental results on a 24GB GPU demonstrate that the system reduces foreground P95 waiting latency by 3.1× and time-to-first-token by 1.55× while sustaining background continuous learning under strict cache budgets. These findings effectively resolve the real-time adaptation bottleneck for edge-based intelligent agents, establishing a viable paradigm for simultaneous serving and learning in resource-constrained environments.

Abstract

On-device LLM agents interact repeatedly with users on local hardware, producing private traces that are valuable for adaptation but should not be sent to a remote trainer. Ideally, such agents would learn contiguously---adapting from every interaction without pausing or suspending user-facing inference---yet existing inference runtimes assume stable weights and existing RL systems assume separated resources, so neither can support this continuity. We present LOCAL, the first single-GPU runtime that enables contiguous on-device learning for LLM agents. The key insight is that GPU scheduling, adapter version management, and KV-cache validity cannot be handled by independent subsystems: adapter updates invalidate cached KV tensors from older versions, and cache retention affects the memory available for training. LOCAL makes adapter version, task priority, and cache state visible to three cooperating components---a cooperative scheduler, a version-aware KV-cache manager, and a multi-agent model runtime---that share this state to keep scheduling, execution, and cache maintenance mutually consistent. On a single 24 GB GPU with 7B-class models, LOCAL lowers foreground queue-wait p95 by 3.1x over FIFO, lowers p95 time-to-first-token (TTFT) by 1.55x versus non-preemptible training, cuts post-publish first-hit prefill p99 by 25.6% and cross-agent TTFT p99 by 21.9%, and keeps background learning progressing under tight KV budgets.

Problem

• On-device Learning
• Contiguous Learning
• LLM Agents
• Inference Runtime

Innovation

• Contiguous On-device Learning
• Version-aware KV-cache Management
• Cooperative Scheduling
• Single-GPU Runtime
• Agent LLMs