Deep Discrete Encoders: Identifiable Deep Generative Models for Rich Data with Discrete Latent Layers

2025-01-02

AI Summary

Deep generative models (DGMs) suffer from statistical non-identifiability and semantic entanglement of latent variables, severely limiting interpretability—especially when modeling heterogeneous data (e.g., text, images, response times). Method: We propose a class of identifiable and interpretable multilayer binary latent variable models, unifying heterogeneous data representation. We establish the first rigorous identifiability theory for discrete deep generative models, proving that latent layer sizes must decrease strictly across layers. We introduce inter-layer nonlinear spectral initialization and a sparse-penalized stochastic approximation EM algorithm, integrating directed graphical modeling, nonlinear spectral analysis, and hierarchical binary encoding. Contribution/Results: Our approach yields semantically transparent, interpretable outcomes in topic modeling, image representation learning, and educational response-time analysis. Simulation studies confirm parameter estimation consistency and demonstrate efficient estimation of exponentially many latent components.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

In the era of generative AI, deep generative models (DGMs) with latent representations have gained tremendous popularity. Despite their impressive empirical performance, the statistical properties of these models remain underexplored. DGMs are often overparametrized, non-identifiable, and uninterpretable black boxes, raising serious concerns when deploying them in high-stakes applications. Motivated by this, we propose an interpretable deep generative modeling framework for rich data types with discrete latent layers, called Deep Discrete Encoders (DDEs). A DDE is a directed graphical model with multiple binary latent layers. Theoretically, we propose transparent identifiability conditions for DDEs, which imply progressively smaller sizes of the latent layers as they go deeper. Identifiability ensures consistent parameter estimation and inspires an interpretable design of the deep architecture. Computationally, we propose a scalable estimation pipeline of a layerwise nonlinear spectral initialization followed by a penalized stochastic approximation EM algorithm. This procedure can efficiently estimate models with exponentially many latent components. Extensive simulation studies validate our theoretical results and demonstrate the proposed algorithms' excellent performance. We apply DDEs to three diverse real datasets for hierarchical topic modeling, image representation learning, response time modeling in educational testing, and obtain interpretable findings.

Problem

• Complex Data Processing
• Statistical Ambiguity
• Model Interpretability

Innovation

• DDEs
• Discrete Latent Variables
• Efficient Computation