Boosting Neural Video Codec via Scale-Driven Online Flow Refinement

2026-06-22

AI Summary

Existing neural video codecs exhibit limited generalization under unseen complex motion, leading to significant motion estimation errors. To address this issue, this work proposes a training-free online optical flow refinement method (SOFR) that corrects motion fields in real time during encoding. SOFR integrates multi-scale optical flow, employs scale-aware and rate-distortion-aware dynamic weight assignment, and validates flow reliability based on warping error. The method is plug-and-play with minimal computational overhead, achieving an average bitrate saving of 2.84% in PSNR and 4.05% in MS-SSIM across multiple neural codec frameworks, while incurring negligible encoding time increase.

Abstract

Although state-of-the-art neural video codecs (NVCs) have achieved remarkable performance, they suffer from limited generalization when encountering complex motion patterns unseen during training. To bridge this domain gap without the expensive cost of online fine-tuning, we propose a Training-Free Scale-Driven Online Flow Refinement (SOFR) method. Serving as a plug-and-play module, SOFR integrates motion information from coarse and fine scales and dynamically fuses them according to warping accuracy, effectively rectifying motion estimation errors with negligible computational overhead. Furthermore, we design a rate-aware strategy that selects different dynamic fusion strategies according to bitrate modes, and employs a reliability check based on warping error to ensure robustness. Extensive experiments on the USTC-TD dataset verify the effectiveness and generalization of SOFR across various NVC frameworks, including DCVC-SDD, DCVC-FM, and EHVC. Notably, it brings an average of 2.84% and 4.05% bitrate savings in terms of PSNR and MS-SSIM, respectively, to DCVC-FM with negligible coding time increase. Our code is available at https://github.com/SunnyMass/SOFR.

Problem

• neural video codec
• motion estimation
• generalization
• complex motion patterns
• domain gap

Innovation

• neural video codec
• online flow refinement
• scale-driven fusion
• rate-aware strategy
• training-free