Covariance-Guided Resource Adaptive Learning for Efficient Edge Inference

2026-03-15

AI Summary

This work addresses the challenge of configuring hardware resources for deep learning inference on edge devices, where power consumption varies significantly across configurations achieving the same throughput. Existing approaches rely on static presets or costly offline profiling, making it difficult to efficiently satisfy both power and throughput constraints simultaneously. To overcome this, we propose CORAL, which formulates resource configuration as a throughput–power co-optimization problem under dual constraints. CORAL introduces distance covariance to capture online the nonlinear relationships between hardware settings—such as DVFS levels and concurrency—and system performance, enabling adaptive exploration of near-optimal configurations without offline overhead. Experiments on two Jetson platforms with three detection models show that CORAL achieves 96%–100% of the optimal performance found by exhaustive search in single-objective scenarios and substantially outperforms baseline methods that either violate constraints or exceed power budgets in strict dual-constraint settings.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Algorithm Configuration
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

For deep learning inference on edge devices, hardware configurations achieving the same throughput can differ by 2$\times$ in power consumption, yet operators often struggle to find the efficient ones without exhaustive profiling. Existing approaches often rely on inefficient static presets or require expensive offline profiling that must be repeated for each new model or device. To address this problem, we present CORAL, an online optimization method that discovers near-optimal configurations without offline profiling. CORAL leverages distance covariance to statistically capture the non-linear dependencies between hardware settings, e.g., DVFS and concurrency levels, and performance metrics. Unlike prior work, we explicitly formulate the challenge as a throughput-power co-optimization problem to satisfy power budgets and throughput targets simultaneously. We evaluate CORAL on two NVIDIA Jetson devices across three object detection models ranging from lightweight to heavyweight. In single-target scenarios, CORAL achieves 96% $\unicode{x2013}$ 100% of the optimal performance found by exhaustive search. In strict dual-constraint scenarios where baselines fail or exceed power budgets, CORAL consistently finds proper configurations online with minimal exploration.

Problem

• edge inference
• power consumption
• throughput
• hardware configuration
• offline profiling

Innovation

• distance covariance
• resource adaptation
• edge inference
• throughput-power co-optimization
• online optimization