SCENE: Semantic-aware Codec Enhancement with Neural Embeddings

2026-01-29

AI Summary

This work addresses the degradation of visual perceptual quality caused by compression artifacts introduced by standard video codecs. To mitigate this issue, the authors propose a lightweight, codec-agnostic, semantic-aware preprocessing framework that integrates semantic embeddings from vision-language models with a differentiable codec proxy. The framework employs an efficient convolutional network to enable end-to-end training, selectively enhancing detail fidelity in perceptually salient regions without altering the existing encoding pipeline. Experimental results demonstrate that the proposed method significantly outperforms baseline approaches on high-resolution benchmarks, achieving consistent improvements in both MS-SSIM and VMAF metrics while effectively preserving texture details in visually critical areas.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Compression artifacts from standard video codecs often degrade perceptual quality. We propose a lightweight, semantic-aware pre-processing framework that enhances perceptual fidelity by selectively addressing these distortions. Our method integrates semantic embeddings from a vision-language model into an efficient convolutional architecture, prioritizing the preservation of perceptually significant structures. The model is trained end-to-end with a differentiable codec proxy, enabling it to mitigate artifacts from various standard codecs without modifying the existing video pipeline. During inference, the codec proxy is discarded, and SCENE operates as a standalone pre-processor, enabling real-time performance. Experiments on high-resolution benchmarks show improved performance over baselines in both objective (MS-SSIM) and perceptual (VMAF) metrics, with notable gains in preserving detailed textures within salient regions. Our results show that semantic-guided, codec-aware pre-processing is an effective approach for enhancing compressed video streams.

Problem

• compression artifacts
• perceptual quality
• video codecs
• semantic awareness
• texture preservation

Innovation

• semantic-aware enhancement
• neural embeddings
• differentiable codec proxy
• perceptual video quality
• lightweight preprocessing