Symmetric Entropy-Constrained Video Coding for Machines

2025-10-17

AI Summary

Video coding for machine vision systems (MVS) must jointly optimize multi-task generalization and semantic fidelity, yet existing video coding for machines (VCM) methods suffer from limited generalizability due to downstream-model coupling or supervised fine-tuning. To address this, we propose SEC-VCM, a Symmetric Entropy-Constrained VCM framework. It introduces the first bidirectional entropy constraint between the codec and a vision backbone (VB), enabling simultaneous semantic preservation and redundancy suppression. A semantic-pixel dual-path fusion module explicitly models high-level semantics while suppressing machine-harmful artifacts. Furthermore, VB-driven unsupervised semantic-guided encoding eliminates reliance on labeled data or task-specific downstream models. Evaluated on instance segmentation and object detection, SEC-VCM achieves state-of-the-art performance, reducing bitrates by 37.41%–46.22% compared to VTM.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

As video transmission increasingly serves machine vision systems (MVS) instead of human vision systems (HVS), video coding for machines (VCM) has become a critical research topic. Existing VCM methods often bind codecs to specific downstream models, requiring retraining or supervised data and thus limiting generalization in multi-task scenarios. Recently, unified VCM frameworks have employed visual backbones (VB) and visual foundation models (VFM) to support multiple video understanding tasks with a single codec. They mainly utilize VB/VFM to maintain semantic consistency or suppress non-semantic information, but seldom explore how to directly link video coding with understanding under VB/VFM guidance. Hence, we propose a Symmetric Entropy-Constrained Video Coding framework for Machines (SEC-VCM). It establishes a symmetric alignment between the video codec and VB, allowing the codec to leverage VB's representation capabilities to preserve semantics and discard MVS-irrelevant information. Specifically, a bi-directional entropy-constraint (BiEC) mechanism ensures symmetry between the process of video decoding and VB encoding by suppressing conditional entropy. This helps the codec to explicitly handle semantic information beneficial for MVS while squeezing useless information. Furthermore, a semantic-pixel dual-path fusion (SPDF) module injects pixel-level priors into the final reconstruction. Through semantic-pixel fusion, it suppresses artifacts harmful to MVS and improves machine-oriented reconstruction quality. Experimental results show our framework achieves state-of-the-art (SOTA) in rate-task performance, with significant bitrate savings over VTM on video instance segmentation (37.41%), video object segmentation (29.83%), object detection (46.22%), and multiple object tracking (44.94%). We will release our code.

Problem

• Develops symmetric video coding for machine vision systems
• Aligns video codec with visual backbone to preserve semantics
• Reduces bitrate while maintaining machine task performance

Innovation

• Symmetric alignment between video codec and visual backbone
• Bi-directional entropy constraint suppresses conditional entropy
• Semantic-pixel dual-path fusion improves machine reconstruction quality