Log-Likelihood Loss for Semantic Compression

2026-01-23

AI Summary

This study addresses lossy source coding in semantic compression, where the reconstruction goal is not to approximate the original signal pointwise but to generate its semantic representation. To this end, the work proposes a distortion measure based on the negative log-likelihood induced by the conditional distribution \(P_{X|U}\) and formally frames the problem within the rate-distortion theory framework. The analysis reveals intrinsic connections between the proposed distortion and log-loss compression, classical rate-distortion theory, and perfect perceptual rate-distortion. Furthermore, it characterizes fundamental properties of the associated rate-distortion function and establishes a theoretical bridge linking semantic compression with several established compression paradigms.

Technology Category

• Machine Learning: Information Theory
• Data Mining & Knowledge Management: Data Compression
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We study lossy source coding under a distortion measure defined by the negative log-likelihood induced by a prescribed conditional distribution $P_{X|U}$. This \emph{log-likelihood distortion} models compression settings in which the reconstruction is a semantic representation from which the source can be probabilistically generated, rather than a pointwise approximation. We formulate the corresponding rate-distortion problem and characterize fundamental properties of the resulting rate-distortion function, including its connections to lossy compression under log-loss, classical rate-distortion problems with arbitrary distortion measures, and rate-distortion with perfect perception.

Problem

• semantic compression
• log-likelihood distortion
• lossy source coding
• rate-distortion
• perception

Innovation

• log-likelihood distortion
• semantic compression
• rate-distortion theory
• probabilistic reconstruction
• lossy source coding