Edge Physical AI Deployment of Vision Transformers on Heterogeneous Edge GPU Targeting Autonomous Vehicles

2026-07-12

AI Summary

This work addresses the challenges of deploying Vision Transformers for autonomous driving on edge GPUs, where operator incompatibility, execution fragmentation, and poor energy efficiency hinder real-time performance. The authors propose H-FraDS, the first approach to jointly tackle operator adaptation, resource scheduling, and real-time constraints across heterogeneous hardware (GPU–DLA–OFA). By leveraging a hardware-aware frame scheduling strategy, tasks are proportionally allocated between the GPU and dual DLAs. DLA compatibility is achieved through tensor reshaping, replacing the ERF activation with a tanh approximation, and substituting LayerNorm with bounded tanh operations. The adapted model retains 92% F1 score, achieves 125.93 FPS inference throughput—2.36× faster than DLA-only execution—and attains an energy efficiency of 4.0 FPS/W, satisfying the 30 FPS requirement for real-time perception.

Abstract

Physical AI systems, such as autonomous vehicles and intelligent machines, require transformer-based perception models that satisfy stringent edge latency and energy constraints. However, heterogeneous edge-GPU deployment remains limited by underutilized hardware engines and accelerator-incompatible operators, causing fragmented execution and lower throughput per watt. This paper presents Heterogeneous Frame Dispatch Scheduling (H-FraDS), a hardware-aware frame scheduling methodology for transformer inference on a recent NVIDIA edge GPU. H-FraDS routes frames across the GPU and dual deep learning accelerator (DLA) cores using fixed dispatch ratios to improve utilization under latency and power constraints. To enable scheduling, incompatible transformer components are adapted for DLA execution by reshaping tensors, approximating error function (ERF) with tanh, and replacing layer normalization with bounded tanh. The adapted model maintains a 92% F1 score, with only a 2% reduction from the original. Optical flow accelerator (OFA) is further used for inference-side optical-flow estimation. To the best of the authors' knowledge, prior work has not addressed these combined issues. Using Swin Transformer for autonomous-driving perception, H-FraDS Balanced Dispatch (1:2) achieves 125.93 FPS, a 2.36x speedup over standalone adapted-DLA execution, 4.0 FPS/W, and approximately 24 ms DLA latency, satisfying 30 FPS real-time operation; the GPU-DLA-OFA case achieves a 2.02x DLA throughput speedup.

Problem

• Edge AI
• Vision Transformers
• Heterogeneous GPU
• Autonomous Vehicles
• Hardware Acceleration

Innovation

• Heterogeneous Edge GPU
• Vision Transformers
• Hardware-aware Scheduling
• DLA Compatibility
• Autonomous Vehicles