Estimating Rate-Distortion Functions Using the Energy-Based Model

2025-07-21

AI Summary

Computing high-dimensional rate-distortion (RD) functions remains challenging due to the high computational complexity of the Blahut–Arimoto algorithm and limitations of existing neural approaches—namely, inaccurate reconstruction of the optimal conditional distribution or reliance on restrictive prior assumptions. Method: We propose a novel energy-based framework that theoretically links the RD dual formulation to statistical physics’ free energy. Our approach trains only a single energy function network and employs Markov Chain Monte Carlo (MCMC) sampling to circumvent the intractable partition function, thereby avoiding explicit prior modeling. Contribution/Results: The method requires no structural assumptions about source or reconstruction distributions and enables end-to-end learning of the optimal conditional distribution. Experiments demonstrate significant improvements over state-of-the-art neural methods in both high-dimensional RD curve estimation and faithful reconstruction of the optimal conditional distribution.

Technology Category

• Machine Learning: Information Theory
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

The rate-distortion (RD) theory is one of the key concepts in information theory, providing theoretical limits for compression performance and guiding the source coding design, with both theoretical and practical significance. The Blahut-Arimoto (BA) algorithm, as a classical algorithm to compute RD functions, encounters computational challenges when applied to high-dimensional scenarios. In recent years, many neural methods have attempted to compute high-dimensional RD problems from the perspective of implicit generative models. Nevertheless, these approaches often neglect the reconstruction of the optimal conditional distribution or rely on unreasonable prior assumptions. In face of these issues, we propose an innovative energy-based modeling framework that leverages the connection between the RD dual form and the free energy in statistical physics, achieving effective reconstruction of the optimal conditional distribution.The proposed algorithm requires training only a single neural network and circumvents the challenge of computing the normalization factor in energy-based models using the Markov chain Monte Carlo (MCMC) sampling. Experimental results demonstrate the significant effectiveness of the proposed algorithm in estimating high-dimensional RD functions and reconstructing the optimal conditional distribution.

Problem

• Estimating high-dimensional rate-distortion functions efficiently
• Overcoming computational limits of Blahut-Arimoto algorithm
• Reconstructing optimal conditional distribution without prior assumptions

Innovation

• Energy-based model for RD function estimation
• Single neural network avoids normalization factor
• MCMC sampling for optimal distribution reconstruction