Protect Before Generate: Error Correcting Codes within Discrete Deep Generative Models

2024-10-10

arXiv.org

AI Summary

Discrete variational autoencoders (DVAEs) suffer from inaccurate posterior inference and large variational lower-bound gaps. To address these challenges, this work systematically introduces error-correcting codes—such as repetition and polar codes—from communication theory into discrete variational inference. By embedding structured redundancy into the latent space, our approach constructs robust hierarchical latent codes that enable differential protection of latent variables and calibrated uncertainty estimation. Methodologically, we jointly optimize the variational objective and the redundancy-encoding loss, while remaining compatible with importance-weighted autoencoder (IWAE) estimators. Experiments on MNIST, Fashion-MNIST, CIFAR-10, and Tiny ImageNet demonstrate substantial improvements in reconstruction quality, generation fidelity, and uncertainty calibration. Our method consistently outperforms both unencoded baselines and standard IWAE models across all metrics.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Despite significant advancements in deep probabilistic models, learning low-dimensional discrete latent representations remains a challenging task. In this paper, we introduce a novel method that enhances variational inference in discrete latent variable models by leveraging Error Correcting Codes (ECCs) to introduce redundancy in the latent representations. This redundancy is then exploited by the variational posterior to yield more accurate estimates, thereby narrowing the variational gap. Inspired by ECCs commonly used in digital communications and data storage, we demonstrate proof-of-concept using a Discrete Variational Autoencoder (DVAE) with binary latent variables and block repetition codes. We further extend this idea to a hierarchical structure based on polar codes, where certain latent bits are more robustly protected. Our method improves generation quality, data reconstruction, and uncertainty calibration compared to the uncoded DVAE, even when trained with tighter bounds such as the Importance Weighted Autoencoder (IWAE) objective. In particular, we demonstrate superior performance on MNIST, FMNIST, CIFAR10, and Tiny ImageNet datasets. The general approach of integrating ECCs into variational inference is compatible with existing techniques to boost variational inference, such as importance sampling or Hamiltonian Monte Carlo. We also outline the key properties ECCs must have to effectively enhance discrete variational inference.

Problem

• Improving inference in discrete VAEs with error-correcting codes
• Reducing variational gap through redundant latent representations
• Enhancing generation quality and uncertainty calibration in discrete models

Innovation

• Using Error-Correcting Codes for redundancy
• Conceptualizing VAE as communication system
• Hierarchical structure for feature disentanglement