The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

2026-08-12

AI Summary

This work addresses the performance overhead in low-batch inference caused by redundant weight loading in existing tensor frameworks, even when operating system page caches already hold GPU-readable data. The authors propose a zero-copy sharing mechanism based on MAP_SHARED memory mapping and DLPack capsules, which— for the first time—integrates the OS page cache into the reclaimable accelerator memory hierarchy. This approach circumvents the redundant data copies imposed by conventional framework ownership models and ensures efficient execution through GPU-side activation residency and instruction scheduling. Experimental results demonstrate a weight-loading bandwidth of 516 GB/s, achieving inference throughput comparable to that of resident storage; under multi-process sharing, throughput improves by 5.5× and first-token latency is reduced by 6.4×.

Abstract

Open-weight models can occupy a middle capacity regime: active weights fit in DRAM as cached file pages, but a second framework-owned representation does not fit or must be refilled as layers run, so low-batch decode rereads the weights every token. On integrated and coherent-memory systems the checkpoint's file pages already occupy a GPU-readable domain, yet accelerator-loading paths still copy them into framework-owned allocations. We call this the ingestion tax: the OS holds the bytes as clean, evictable file pages, the architecture makes them GPU-readable, and only the framework's ownership model stands between the two. We present file-backed weight adoption: a framework-independent producer maps each tensor with MAP_SHARED, wraps the pages as a no-copy GPU buffer, and exports a DLPack capsule that PyTorch or MLX imports as ordinary storage. Zero-copy import alone is not enough: a randomized factorial establishes a three-part execution contract - read the mapping, keep activations accelerator-resident, order on the GPU; a bolt-on extension forfeiting the last two runs 2.3x slower than stock. Under the contract, adoption removes the tax at no rate cost: 516 GB/s where default constructors reach 53-82, at parity with the same kernel over resident storage ([-0.66%, +0.48%]; Qwen2.5-72B 7.14 vs 7.23 tok/s). Parity buys ownership: N processes decode one mapped copy where resident loading pays N (at capacity, 5.5 vs 0.08 tok/s); a 65 GB checkpoint reaches its first token 6.4x sooner; the Kimi K3 spine stage falls from 2.62 to 0.35 s per token, 3.8x from storage alone. The same mechanism improves llama.cpp 1.21x at half the footprint on an AMD APU, ties overlapped streaming on a capacity-exceeding GH200, and loses 39x across PCIe: memory topology, not the API, decides the byte path. The deployment rule treats the page cache as a first-class, reclaimable accelerator-storage tier.

Problem

• ingestion tax
• file-backed weights
• tensor frameworks
• GPU-readable memory
• memory efficiency

Innovation

• file-backed weights
• zero-copy ingestion
• ingestion tax
• DLPack
• memory topology