Hyperion: Low-Latency Ultra-HD Video Analytics via Collaborative Vision Transformer Inference

2025-12-25

AI Summary

To address the dual bottlenecks of limited edge-device compute capacity and high cloud transmission overhead in real-time ultra-high-definition (UHD) video analytics, this paper proposes a cloud-edge collaborative vision transformer (ViT) inference framework. Methodologically, it introduces: (1) a patch-level importance scoring mechanism grounded in ViT’s architectural characteristics, enabling semantic-aware lightweight collaborative perception; (2) a dynamic network-adaptive patch-level quality scheduling strategy that allocates bitrate and computational resources on demand; and (3) an edge-cloud weighted fusion inference architecture with a coordinated pipelined execution flow. Experimental results demonstrate that the framework achieves up to a 1.61× improvement in frame processing throughput and up to a 20.2% gain in inference accuracy across heterogeneous network conditions—outperforming state-of-the-art approaches. It thus effectively enables low-latency, high-accuracy cloud-edge cooperative visual understanding.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Learning on the Edge & Model Compression
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Cloud, edge and content delivery systems for the Web
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Recent advancements in array-camera videography enable real-time capturing of ultra-high-definition (Ultra-HD) videos, providing rich visual information in a large field of view. However, promptly processing such data using state-of-the-art transformer-based vision foundation models faces significant computational overhead in on-device computing or transmission overhead in cloud computing. In this paper, we present Hyperion, the first cloud-device collaborative framework that enables low-latency inference on Ultra-HD vision data using off-the-shelf vision transformers over dynamic networks. Hyperion addresses the computational and transmission bottleneck of Ultra-HD vision transformers by exploiting the intrinsic property in vision Transformer models. Specifically, Hyperion integrates a collaboration-aware importance scorer that identifies critical regions at the patch level, a dynamic scheduler that adaptively adjusts patch transmission quality to balance latency and accuracy under dynamic network conditions, and a weighted ensembler that fuses edge and cloud results to improve accuracy. Experimental results demonstrate that Hyperion enhances frame processing rate by up to 1.61 times and improves the accuracy by up to 20.2% when compared with state-of-the-art baselines under various network environments.

Problem

• Enables low-latency Ultra-HD video analytics using vision transformers
• Reduces computational and transmission bottlenecks in collaborative inference
• Balances latency and accuracy under dynamic network conditions

Innovation

• Collaborative framework for low-latency Ultra-HD video analytics
• Dynamic patch transmission balancing latency and accuracy
• Weighted ensembler fusing edge and cloud results