URVC: A Unified Real-Time Neural Video Coding Model with Temporal, Spatial, and Perceptual Adaptivity

2026-07-16

AI Summary

Existing neural video codecs struggle to simultaneously achieve real-time performance and adaptability to varying content dynamics, user preferences, and quality requirements in dynamic environments. This work proposes the first unified real-time neural video coding framework that integrates temporal, spatial, and perceptual adaptation mechanisms within a single model. Temporal adaptability to motion complexity is enabled through rate-aware multi-candidate temporal prediction, while spatial bitrate allocation is finely controlled at test time via feature decomposition. Additionally, a lightweight module library allows flexible switching between fidelity- and perception-oriented coding modes. The proposed approach maintains real-time encoding speed while significantly enhancing adaptability across diverse motion patterns, regions of visual interest, and application-specific quality preferences.

Abstract

Neural video coding has advanced rapidly, achieving competitive compression performance while also enabling real-time coding speed. Yet, existing codecs exhibit severe rigidity when deployed in dynamic environments, failing to adapt to different video content, user requirements, and quality preferences. First, to meet the real-time constraint, they discard explicit motion estimation and motion compression, thereby losing the ability to adapt temporal prediction to motion complexity and bitrate constraints. Second, their spatial bit allocation strategy is coarse and, once trained, is fixed. It cannot adapt to dynamic user requirements at test time, preventing users from freely controlling the spatial distribution of bits. Third, they cannot adapt their quality preference to varying application requirements without deploying separate models. We address all three limitations within a single real-time neural video codec--URVC, transforming a rigid system into a unified framework with temporal, spatial, and perceptual adaptivity. First, we propose a rate-aware adaptive temporal prediction method that generates diverse prediction candidates through a multi-candidate architecture and couples candidate selection directly to rate-distortion optimization. Second, we propose a decomposition-based spatial rate control method that achieves finer-grained spatial bit allocation through feature decomposition and separate quantization, and allows users to perform direct spatial rate control at test time without retraining. Third, we propose a perceptual switching method that only requires learning a secondary module bank alongside a frame generator, enabling a codec to switch between signal fidelity and perceptual quality modes.

Problem

• neural video coding
• temporal adaptivity
• spatial adaptivity
• perceptual adaptivity
• real-time compression

Innovation

• adaptive temporal prediction
• spatial rate control
• perceptual switching
• neural video coding
• rate-distortion optimization