EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems

2026-07-09

AI Summary

This work addresses the challenge of accurately predicting inference latency of deep learning models on resource-constrained edge devices, a critical bottleneck for latency-aware model compression. To overcome this limitation, the authors propose EvoLP, a novel framework that introduces a self-evolving mechanism into latency prediction. During compression, EvoLP dynamically enhances prediction accuracy through lightweight modeling, online feedback, and evolutionary optimization strategies. This adaptive approach effectively guides the compression process to meet strict latency constraints while preserving high model accuracy. Extensive experiments demonstrate that EvoLP consistently outperforms state-of-the-art methods across three distinct edge devices and four model variants, achieving significant improvements in both compression efficacy and latency prediction fidelity.

Abstract

Edge devices are increasingly utilized for deploying deep learning applications on embedded systems. The real-time nature of many applications and the limited resources of edge devices necessitate latency-targeted neural network compression. However, measuring latency on real devices is challenging and expensive. Therefore, this letter presents a novel and efficient framework, named EvoLP, to accurately predict the inference latency of models on edge devices. This predictor can evolve to achieve higher latency prediction precision during the network compression process. Experimental results demonstrate that EvoLP outperforms previous state-of-the-art approaches by being evaluated on three edge devices and four model variants. Moreover, when incorporated into a model compression framework, it effectively guides the compression process for higher model accuracy while satisfying strict latency constraints. We open source EvoLP at https://github.com/ntuliuteam/EvoLP.

Problem

• latency prediction
• model compression
• edge devices
• real-time systems
• neural networks

Innovation

• self-evolving
• latency prediction
• model compression
• edge computing
• real-time inference