AutoUVM: Automated Prefetching Framework for LLMs under UVM Oversubscription

2026-09-05

AI Summary

为解决LLMs在UVM超订阅下执行效率低的问题，提出AutoUVM系统，通过细粒度的张量级预取策略自动优化内存访问，提高运行速度。

Abstract

Large language models (LLMs) increasingly exceed the memory capacity of commodity GPUs, making memory oversubscription common in practical deployments. NVIDIA Unified Virtual Memory (UVM) provides transparent access to host memory, but its page-fault-driven migrations introduce severe performance overhead. While UVM exposes primitives (e.g., prefetching and placement hints) to mitigate these costs, they require low-level CUDA modifications, limiting their applicability for most LLM users. Meanwhile, existing UVM optimizations operate at coarse managed-object granularity and fail to capture deep learning frameworks'internal tensor-level memory behavior, leading to excessive data movement and CPU-GPU interconnect bottlenecks. We propose AutoUVM, an automated, framework-aware UVM prefetching system for efficient LLM execution under memory oversubscription. AutoUVM bridges the semantic gap between deep learning frameworks and UVM by exposing tensor-level access information and enabling policy-driven prefetching at fine granularity. Implemented as a transparent extension, AutoUVM requires no changes to model code and dynamically adapts to runtime memory pressure. We instantiate AutoUVM with a roofline-inspired policy to identify performance-critical data transfers. Across ten LLMs, AutoUVM achieves an average 3.1x speedup over baseline UVM and consistently surpasses the best-performing prior UVM prefetcher by 1.9x, with improvements of up to 4.7x over object-level prefetchers, while significantly reducing page faults.

Problem

• Large language models
• Memory oversubscription
• Unified Virtual Memory
• Performance overhead
• Tensor-level memory behavior

Innovation

• Automated Prefetching
• Framework-aware
• Tensor-level Access
• Memory Oversubscription
• Dynamic Adaptation