End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference

2026-06-26

AI Summary

This work addresses the vulnerability of large language models to inference failures and computational waste in dynamic cloud environments, where resource fluctuations—such as preemptive instance interruptions and varying QoS—disrupt execution. To tackle this, the authors propose Learning to Allocate (L2A), a framework that formulates inference as a dynamic allocation problem jointly constrained by input content and real-time resource budgets. L2A employs a lightweight gating network to dynamically apply three sparsity strategies: layer skipping, attention head pruning, and token reduction. Notably, it is the first approach to end-to-end co-optimize task accuracy, logical consistency, and resource cost, enabling a single model to adaptively span the compute-accuracy Pareto frontier. Evaluated on Llama-3-8B and Qwen-3-4B, L2A achieves up to 34% layer sparsity with only a 0.6% drop in GSM8K accuracy and matches zero-shot performance of full models, significantly outperforming static baselines that require separate tuning.

Abstract

Large Language Models (LLMs) inference is typically deployed under a static resource assumption, where models execute a fixed computational graph regardless of the runtime environment. However, real-world cloud infrastructure is inherently dynamic, characterized by fluctuating availability (e.g., spot instance preemption) and tiered Quality-of-Service requirements. In such volatile settings, static models are inflexible: they either crash under resource constraints or waste compute on redundant operations. To bridge this gap, we propose Learning to Allocate (L2A), an end-to-end framework for resource-adaptive inference. Unlike prior methods that condition only on input difficulty, we formulate inference as a constrained allocation problem conditioned on both the input and the runtime resource budget itself. We introduce lightweight, budget-conditioned and input-aware gating networks integrated into the LLM. These gates are trained via a unified objective that jointly optimizes task performance, logical consistency, and resource costs along three axes matching how real-world dynamics manifest: layer skipping for memory and depth pressure, head pruning for throughput contention, and reasoning-token reduction for latency tightening. This lets the model learn a budget-aware policy beyond input difficulty alone: it adaptively configures its computational footprint with respect to real-time resource dynamics, maximizing reasoning depth when resources permit while enforcing strict frugality when budgets tighten. A single L2A model traces the entire compute-accuracy Pareto frontier on Llama-3-8B and Qwen-3-4B: at up to 34% realized layer sparsity, it stays within 0.6% of the dense baseline on GSM8K, with the same gap holding zero-shot on out-of-distribution tasks, while every static or heuristic baseline requires a separately tuned model and still drops by 5-10% at comparable inference time.

Problem

• Dynamic Sparsity
• Resource-Adaptive Inference
• Large Language Models
• Runtime Resource Constraints
• Computational Efficiency

Innovation

• resource-adaptive inference
• dynamic sparsity
• budget-conditioned gating
• end-to-end optimization
• compute-accuracy Pareto frontier