MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes

2026-05-14

AI Summary

This work addresses the challenges of deploying video object detection on embedded vision nodes, where high computational costs and multi-frame buffering hinder practicality. The authors propose MR2-ByteTrack, a method that alternates between high- and low-resolution inference, leverages ByteTrack for cross-frame association, and introduces a lightweight multi-resolution rescore mechanism to fuse confidence scores across frames, thereby suppressing false positives. Notably, this approach achieves real-time Transformer-based video detection for the first time on an MCU-class platform (GAP9) and is compatible with both CNN and Transformer backbones. Experimental results on ImageNetVID show mAP scores of 49.0 for CNN and 48.7 for Transformer models, with computational costs reduced by 53% and 32%, respectively, and energy consumption savings of up to 55%.

Abstract

Modern smart vision sensors need on-device intelligence to process video streams, as cloud computing is often impractical due to bandwidth, latency, and privacy constraints. However, these sensory systems typically rely on ultra-low-power microcontrollers (MCUs) with limited memory and compute, making conventional video object detection methods, which require feature storage or multi-frame buffering, unfeasible. To address this challenge, we introduce Multi-Resolution Rescored ByteTrack (MR2-ByteTrack), a Video Object Detection (VOD) method tailored for MCU-based embedded vision nodes. MR2-ByteTrack reduces computational cost by alternating between full- and low-resolution inference, while linking detections across frames via ByteTrack and correcting misclassifications through the Rescore algorithm, which applies probability union rules to aggregate detection confidence scores across frames. We apply our approach to both a CNN-based detector and a Transformer-based model, demonstrating its generality across architectures with fundamentally different spatial processing. Experiments on ImageNetVID demonstrate that MR2-ByteTrack maintains accuracy, achieving mAP scores of up to 49.0 for the CNN-based models and 48.7 for the Transformer, while reducing multiply-accumulate operations by as much as 53\% for the CNNs and 32\% for the Transformer. When deployed on GAP9, an ultra-low-power RISC-V multicore MCU, our method yields up to 55\% energy savings compared to processing only full-resolution images, enabling the first real-time Transformer-based VOD on an MCU-class embedded vision node. Code available at https://github.com/Bomps4/Multi_Resolution_Rescored_ByteTrack/tree/IEEE_Access

Problem

• Video Object Detection
• Embedded Vision
• Ultra-low-power MCU
• On-device Intelligence
• Resource-constrained Systems

Innovation

• Video Object Detection
• Multi-Resolution Inference
• Embedded Vision
• Transformer on MCU
• Energy-Efficient AI