Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

2026-07-11

AI Summary

This work addresses the challenge of running large language models on consumer-grade hardware, where limited GPU memory forces existing systems to rely on coarse-grained CPU offloading strategies that fail to account for intra-layer tensor heterogeneity and dynamic hardware load. To overcome these limitations, we propose ATSInfer, the first system to enable tensor-level fine-grained offloading scheduling. ATSInfer integrates static placement with load-aware dynamic data transfer and introduces an asynchronous CPU-GPU coordination mechanism to efficiently orchestrate storage, data movement, and computation resources. Experimental results demonstrate that ATSInfer achieves up to 1.94× and 3.29× higher throughput during the prefill and decoding phases, respectively, compared to state-of-the-art approaches, while significantly improving GPU utilization and PCIe bandwidth efficiency.

Abstract

Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory. Existing offloading systems, however, typically rely on coarse layer-level or expert-level scheduling, which overlooks substantial heterogeneity among tensors within the same layer and adapts poorly to changing hardware load conditions on such devices. This paper presents ATSInfer, a hybrid CPU-GPU inference system for consumer devices that performs offloading at tensor granularity. ATSInfer combines static tensor placement with load-aware dynamic transfer, and introduces asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends. We implement ATSInfer and evaluate it on representative consumer platforms using both dense and MoE models. Compared with existing systems, ATSInfer improves prefill throughput by up to 1.94$\times$ and decode throughput by up to 3.29$\times$, while also increasing GPU utilization and making more effective use of PCIe bandwidth. These results show that ATSInfer can substantially improve the user experience of local LLM deployment on personal consumer devices.

Problem

• tensor scheduling
• hybrid CPU-GPU inference
• LLM offloading
• consumer devices
• memory heterogeneity

Innovation

• tensor-granularity scheduling
• hybrid CPU-GPU inference
• load-aware offloading
• asynchronous coordination
• LLM inference on consumer devices