Neural Stereo Video Compression with Hybrid Disparity Compensation

2025-04-29

AI Summary

To address insufficient cross-view redundancy modeling in stereo video compression, this paper proposes a Hybrid Disparity Compensation (HDC) strategy—the first to jointly integrate explicit pixel-wise displacement priors with implicit cross-view attention mechanisms, thereby enhancing binocular feature alignment accuracy and redundancy representation capability. Methodologically, we design an end-to-end differentiable encoder–decoder framework, incorporating similarity-graph-normalized pixel-level explicit attention and deeply embedding HDC throughout the entire cross-view entropy modeling pipeline. Evaluated on KITTI 2012/2015 and Nagoya datasets, HDC consistently outperforms conventional and state-of-the-art neural compression methods, achieving an average BD-rate reduction of 18.7%, demonstrating significant improvements in rate-distortion performance.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Learning on the Edge & Model Compression
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Disparity compensation represents the primary strategy in stereo video compression (SVC) for exploiting cross-view redundancy. These mechanisms can be broadly categorized into two types: one that employs explicit horizontal shifting, and another that utilizes an implicit cross-attention mechanism to reduce cross-view disparity redundancy. In this work, we propose a hybrid disparity compensation (HDC) strategy that leverages explicit pixel displacement as a robust prior feature to simplify optimization and perform implicit cross-attention mechanisms for subsequent warping operations, thereby capturing a broader range of disparity information. Specifically, HDC first computes a similarity map by fusing the horizontally shifted cross-view features to capture pixel displacement information. This similarity map is then normalized into an"explicit pixel-wise attention score"to perform the cross-attention mechanism, implicitly aligning features from one view to another. Building upon HDC, we introduce a novel end-to-end optimized neural stereo video compression framework, which integrates HDC-based modules into key coding operations, including cross-view feature extraction and reconstruction (HDC-FER) and cross-view entropy modeling (HDC-EM). Extensive experiments on SVC benchmarks, including KITTI 2012, KITTI 2015, and Nagoya, which cover both autonomous driving and general scenes, demonstrate that our framework outperforms both neural and traditional SVC methodologies.

Problem

• Hybrid disparity compensation for stereo video compression
• Combining explicit and implicit cross-view redundancy reduction
• End-to-end neural framework outperforming traditional SVC methods

Innovation

• Hybrid disparity compensation combines explicit and implicit methods
• Explicit pixel displacement simplifies optimization for warping
• End-to-end framework integrates HDC in key coding operations