DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inference

2025-07-25

AI Summary

To address the efficiency bottleneck in deploying large language models (LLMs) on edge devices—stemming from the quadratic computational complexity of attention with respect to sequence length—this paper proposes DeltaLLM, a training-free, plug-and-play inference framework. Its core innovation lies in exploiting temporal sparsity in attention patterns, designing an accuracy- and memory-aware incremental matrix construction strategy, and introducing a context-aware hybrid attention mechanism: full attention within local windows and incremental approximation for long-range dependencies. Integrated with dynamic attention pruning and incremental matrix approximation, DeltaLLM is architecture-agnostic, requiring no fine-tuning or structural modifications to support mainstream models such as BitNet and Llama. On BitNet, it achieves 60% attention sparsity during prefill and 57% end-to-end sparsity, yielding a +1.34 F1 gain on SQuAD-v2; on Llama, it attains 60% sparsity with negligible accuracy degradation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Deploying Large Language Models (LLMs) on edge devices remains challenging due to their quadratically increasing computations with the sequence length. Existing studies for dynamic attention pruning are designed for hardware with massively parallel computation capabilities, such as GPUs or TPUs, and aim at long context lengths (e.g., 64K), making them unsuitable for edge scenarios. We present DeltaLLM, a training-free framework that exploits temporal sparsity in attention patterns to enable efficient LLM inference across both the prefilling and decoding stages, on resource-constrained edge devices. DeltaLLM introduces an accuracy- and memory-aware delta matrix construction strategy that introduces temporal sparsity, and a context-aware hybrid attention mechanism that combines full attention in a local context window with delta approximation outside it to increase accuracy. We evaluate our framework on the edge-device-friendly BitNet-b1.58-2B-4T model and Llama3.2-1B-Instruct model across diverse language tasks. The results show that on BitNet, our framework increases the attention sparsity from 0% to 60% during the prefilling stage with slight accuracy improvement on the WG task, and 0% to 57% across both the prefilling and decoding stages, with even higher F1 score from 29.63 to 30.97 on SQuAD-v2 task. On the Llama model, it can also achieve up to 60% sparsity during the prefilling stage and around 57% across both stages with negligible accuracy drop. These results demonstrate that DeltaLLM offers a promising solution for efficient edge deployment, requiring no fine-tuning and seamlessly integrating with existing inference pipelines.

Problem

• Enabling efficient LLM inference on edge devices
• Exploiting temporal sparsity in attention patterns
• Achieving high sparsity without accuracy loss

Innovation

• Training-free framework exploiting temporal sparsity
• Accuracy- and memory-aware delta matrix construction
• Context-aware hybrid attention mechanism