🤖 AI Summary
This work addresses the high GPU memory cost of dense attention in fine-tuning large language models with long contexts. The authors propose an efficient training method that integrates hierarchical global attention (HGA), segmented backpropagation, and hierarchical KV caching: only the current segment resides in GPU memory, while historical KV caches are offloaded to host memory or NVMe storage and selectively loaded via HGA to provide each query block with a bounded window of relevant past tokens. This approach enables, for the first time, 16K-context QLoRA fine-tuning on a 16GB GPU (Quadro RTX 5000) with a peak memory usage of 15.28 GB, achieving comparable training quality to dense attention (2.7405 vs. 2.7383 nats). Moreover, it supports inference scaling up to 131K context length, substantially alleviating the memory bottleneck in long-context model adaptation.
📝 Abstract
Parameter-efficient fine-tuning reduces model and optimizer memory, but dense attention still makes long training sequences expensive. We combine Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage. Only the active segment remains differentiable in VRAM; older KV is detached into RAM or NVMe, and HGA loads a bounded set of exact historical tokens for each query block. On Qwen3-8B with 4-bit QLoRA and PG19, dense training on a 16 GB Quadro RTX 5000 fits 2,048 tokens but fails at 4,096, whereas HGA reaches 16,384 tokens with 15.28 GB peak VRAM. Under evaluation the same adapter runs through 131,072 tokens on this card; VRAM is not constant but grows gently with the resident chunk summaries, so RAM and NVMe capacity set the practical limit beyond these lengths. At the shared 2K training length, HGA-trained and dense-trained adapters obtain 2.7405 and 2.7383 nat under the same dense-attention readout, while the stock model obtains 2.9541. At this boundary HGA training is already marginally faster (217.75 vs. 207.02 tokens/s), and the HGA-to-dense throughput ratio improves from 1K to 2K; because HGA keeps the attended historical set per token approximately constant while dense work per token grows, we expect this lead to widen as context grows. Dense attention is used for the main quality and retrieval comparisons so that they measure the learned weights and remain compatible with standard generation frameworks. HGA can also be used for retrieval and generation; an optimized production-grade serving implementation is under development.