🤖 AI Summary
This work addresses the challenges of deploying optical flow–based video frame interpolation on mobile devices, where incompatibility of sampling operators, iterative refinement failure under 8-bit quantization, and memory constraints hinder practicality. The authors propose a novel architecture that eliminates learned optical flow estimation by repurposing motion vectors from an H.264 decoder to pre-align input frames. By discarding spatial sampling and iterative accumulation, the method relies solely on a convolutional network to refine residual details, yielding a fully compute-intensive inference graph. This approach pioneers the use of codec-derived motion vectors as priors for real-time mobile interpolation, enabling a low-latency design fully compatible with neural processing units (NPUs) and uncovering the causal mechanism behind iterative optical flow failure under quantization. On a Snapdragon 8 Gen 3 platform, it achieves 12.8 ms latency for 1080p interpolation, with an end-to-end median latency of 28.4 ms in an open-source Android player, validated over 54,623 samples across 30 minutes of continuous playback.
📝 Abstract
Mobile displays refresh at 90-120 Hz, yet most video is encoded at 24-30 frames per second; real-time frame-rate doubling requires each synthesized frame within 33.3 ms on mobile neural processing units. We show that mainstream flow-based video frame interpolation faces three structural deployment barriers on mobile accelerators: spatial sampling operators exceed the frame budget or lack hardware support, iterative flow refinement collapses under 8-bit post-training quantization, and memory-bound operators dominate the inference graph. ANVIL addresses these barriers by reusing motion vectors already computed by the H.264 decoder to prealign input frames, removing learned optical flow, spatial sampling, and iterative accumulation from the accelerator graph. The remaining residual is refined by a convolution-dominated network whose inference graph is composed almost entirely of compute-bound operators. On a Snapdragon 8 Gen 3 device, ANVIL achieves 12.8 ms 1080p network inference in 8-bit integer precision; an open-source Android player sustains 28.4 ms median end-to-end latency per interpolated frame pair over 54,623 consecutively logged samples during 30-minute continuous playback. Per-operator causal analysis identifies quantized accumulation on recurrent flow states as a key mechanism behind integer quantization failure in iterative methods. The current design targets H.264 playback scenarios with decoder-exposed motion vectors.