Scheduling Techniques of AI Models on Modern Heterogeneous Edge GPU - A Critical Review

2025-06-02

AI Summary

To address the lack of efficient DNN scheduling frameworks for resource-constrained heterogeneous edge GPU platforms (e.g., NVIDIA Jetson), this paper systematically surveys state-of-the-art multi-accelerator co-scheduling across CPU, GPU, DLA, PVA, and VIC. Leveraging empirical benchmarks—TensorRT and DeepStream—we establish, for the first time, a comprehensive DNN scheduling evaluation framework tailored to edge environments, incorporating latency, throughput, energy efficiency, and memory footprint. We rigorously analyze over 12 mainstream scheduling strategies (rule-based, heuristic, ML-driven, and hybrid), identifying two fundamental bottlenecks: coarse-grained hardware abstraction and poor adaptability to dynamic workloads. Based on these insights, we propose principled design guidelines and an evolutionary roadmap for lightweight, adaptive schedulers. This work provides both theoretical foundations and practical guidance for next-generation edge AI scheduling systems.

Technology Category

• Planning, Routing, and Scheduling: Scheduling
• Machine Learning: Hardware-aware ML
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

In recent years, the development of specialized edge computing devices has significantly increased, driven by the growing demand for AI models. These devices, such as the NVIDIA Jetson series, must efficiently handle increased data processing and storage requirements. However, despite these advancements, there remains a lack of frameworks that automate the optimal execution of optimal execution of deep neural network (DNN). Therefore, efforts have been made to create schedulers that can manage complex data processing needs while ensuring the efficient utilization of all available accelerators within these devices, including the CPU, GPU, deep learning accelerator (DLA), programmable vision accelerator (PVA), and video image compositor (VIC). Such schedulers would maximize the performance of edge computing systems, crucial in resource-constrained environments. This paper aims to comprehensively review the various DNN schedulers implemented on NVIDIA Jetson devices. It examines their methodologies, performance, and effectiveness in addressing the demands of modern AI workloads. By analyzing these schedulers, this review highlights the current state of the research in the field. It identifies future research and development areas, further enhancing edge computing devices' capabilities.

Problem

• Automating optimal DNN execution on edge GPUs
• Efficiently utilizing heterogeneous accelerators in edge devices
• Reviewing DNN schedulers for NVIDIA Jetson performance

Innovation

• Automated DNN schedulers for edge GPUs
• Optimized utilization of heterogeneous accelerators
• Performance review of NVIDIA Jetson schedulers