LASER: Load-Aware Serving with Early-Exit for Reasoning LLMs at the Edge

2026-06-30

AI Summary

This work addresses the tension between lengthy chain-of-thought (CoT) inference and constrained resources when deploying large models on edge devices, noting that existing early-exit mechanisms neglect concurrent requests and dynamic load fluctuations. The authors propose a novel load-aware early-exit mechanism that integrates real-time system load into exit decisions, dynamically adjusting confidence thresholds and pre-allocating computation budgets based on both request difficulty and current load conditions. This approach jointly optimizes inference accuracy and service latency under strict service-level objective (SLO) constraints. Experimental results across diverse models and workload scenarios demonstrate that the method reduces average latency by 17–38% and improves SLO compliance by 3–6%, with only approximately 1% degradation in accuracy.

Abstract

Large reasoning models (LRMs) such as DeepSeek-R1 have achieved strong performance through extended chain-of-thought (CoT) generation. However, deploying them on edge devices raises a conflict between long CoT sequences and constrained resources. Recent confidence-based early exit methods reduce CoT length for individual requests, yet they apply fixed thresholds from a single-request perspective, ignoring multi-request concurrency and load fluctuation in edge serving. To bridge this gap, we propose \underline{L}oad-\underline{A}ware \underline{S}erving with \underline{E}arly-exit for \underline{R}easoning (LASER). LASER couples two complementary designs: (1) a load-aware adaptive exit threshold that adjusts the confidence bar based on real-time system load within an empirically validated robust range, and (2) a difficulty- and load-aware reasoning budget pre-allocation that assigns compute resources by request difficulty and system capacity. We formulate the problem as a joint optimization of reasoning quality and service latency. Experiments on two reasoning models, four benchmarks, and diverse load conditions show that LASER reduces average latency by 17--38\% and improves service-level objective (SLO) satisfaction by 3--6\% over fixed-threshold baselines, at an average accuracy cost of only 1\%.

Problem

• edge serving
• reasoning LLMs
• early-exit
• load fluctuation
• chain-of-thought

Innovation

• early-exit
• load-aware serving
• reasoning LLMs
• adaptive threshold
• edge inference