ResiComp: Loss-Resilient Image Compression via Dual-Functional Masked Visual Token Modeling

2025-02-15

IEEE transactions on circuits and systems for video technology (Print)

AI Summary

To address the severe degradation in reconstruction quality of neural image coders (NICs) under packet loss in real-time communication, this paper proposes the first latent-space-oriented Masked Visual Token Modeling (MVTM) framework, unifying entropy coding and packet-loss recovery. Methodologically, it introduces a dual-function Transformer for context-aware latent feature modeling; jointly predicts probability mass functions and reconstructs missing tokens; and enables dynamic rate-distortion–robustness trade-off adjustment based on network conditions. Experiments demonstrate state-of-the-art packet-loss robustness on benchmarks including CIC2023, achieving up to 2.1 dB PSNR gain under 10% packet loss. Moreover, its rate-distortion performance approaches that of leading lossless compressors. This work marks the first successful co-optimization of high compression efficiency and strong error resilience in NICs.

Technology Category

• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advancements in neural image codecs (NICs) are of significant compression performance, but limited attention has been paid to their error resilience. These resulting NICs tend to be sensitive to packet losses, which are prevalent in real-time communications. In this paper, we investigate how to elevate the resilience ability of NICs to combat packet losses. We propose ResiComp, a pioneering neural image compression framework with feature-domain packet loss concealment (PLC). Motivated by the inherent consistency between generation and compression, we advocate merging the tasks of entropy modeling and PLC into a unified framework focused on latent space context modeling. To this end, we take inspiration from the impressive generative capabilities of large language models (LLMs), particularly the recent advances of masked visual token modeling (MVTM). During training, we integrate MVTM to mirror the effects of packet loss, enabling a dual-functional Transformer to restore the masked latents by predicting their missing values and conditional probability mass functions. Our ResiComp jointly optimizes compression efficiency and loss resilience. Moreover, ResiComp provides flexible coding modes, allowing for explicitly adjusting the efficiency-resilience trade-off in response to varying Internet or wireless network conditions. Extensive experiments demonstrate that ResiComp can significantly enhance the NIC's resilience against packet losses, while exhibits a worthy trade-off between compression efficiency and packet loss resilience.

Problem

• Enhance neural image codecs' resilience to packet losses
• Unify entropy modeling and packet loss concealment in one framework
• Optimize compression efficiency and loss resilience trade-off

Innovation

• Masked Visual Token Modeling
• Dual-functional Transformer framework
• Feature-domain packet loss concealment