Efficient Video Neural Network Processing Based on Motion Estimation

2025-01-25

AI Summary

To address the high computational cost and power consumption of video neural networks (VNNs), this paper proposes the first end-to-end Bayer-domain video inference framework—bypassing conventional ISP-based RGB conversion and directly processing raw Bayer sensor data for vision tasks. It deeply integrates motion estimation into the VNN computation flow, enabling dynamic elimination of inter-frame redundancy via optical-flow-guided sparse feature extraction and hardware-efficient motion-compensated skipping. The method supports mainstream tasks including object detection and action recognition, achieving over 67% reduction in computational load across multiple standard benchmarks without accuracy degradation. Key contributions are: (1) establishing the first end-to-end VNN inference paradigm operating natively in the Bayer domain; and (2) introducing a motion-aware dynamic sparse computation scheduling architecture that jointly optimizes feature sparsity and temporal redundancy exploitation.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Video neural network (VNN) processing using the conventional pipeline first converts Bayer video information into human understandable RGB videos using image signal processing (ISP) on a pixel by pixel basis. Then, VNN processing is performed on a frame by frame basis. Both ISP and VNN are computationally expensive with high power consumption and latency. In this paper, we propose an efficient VNN processing framework. Instead of using ISP, computer vision tasks are directly accomplished using Bayer pattern information. To accelerate VNN processing, motion estimation is introduced to find temporal redundancies in input video data so as to avoid repeated and unnecessary computations. Experiments show greater than 67% computation reduction, while maintaining computer vision task accuracy for typical computer vision tasks and data sets.

Problem

• Video Neural Networks
• Efficiency Improvement
• Energy Reduction

Innovation

• Motion Estimation
• Video Neural Network Efficiency
• Repetitive Part Identification