🤖 AI Summary
For automatic video object segmentation (AVOS) without first-frame annotations, existing methods rely on computationally expensive optical flow to extract motion cues, hindering real-time deployment on edge devices. This paper proposes replacing optical flow with lightweight frame differencing as a motion prior and designs an extended U-Net architecture for end-to-end segmentation. We systematically demonstrate— for the first time—that frame differencing provides motion cues comparable in effectiveness to optical flow under static-camera conditions, while drastically reducing computational overhead. Experiments on standard AVOS benchmarks show our method achieves a 0.8% improvement in mean J&F over optical-flow-based baselines, with 3.2× faster inference speed and substantially reduced memory consumption and latency—enabling real-time edge deployment. Our core contribution is establishing frame differencing as a viable, efficient motion prior and introducing a new lightweight AVOS paradigm that jointly optimizes accuracy and efficiency.
📝 Abstract
Automatic Video Object Segmentation (AVOS) refers to the task of autonomously segmenting target objects in video sequences without relying on human-provided annotations in the first frames. In AVOS, the use of motion information is crucial, with optical flow being a commonly employed method for capturing motion cues. However, the computation of optical flow is resource-intensive, making it unsuitable for real-time applications, especially on edge devices with limited computational resources. In this study, we propose using frame differences as an alternative to optical flow for motion cue extraction. We developed an extended U-Net-like AVOS model that takes a frame on which segmentation is performed and a frame difference as inputs, and outputs an estimated segmentation map. Our experimental results demonstrate that the proposed model achieves performance comparable to the model with optical flow as an input, particularly when applied to videos captured by stationary cameras. Our results suggest the usefulness of employing frame differences as motion cues in cases with limited computational resources.