🤖 AI Summary
This study addresses the challenge of quantifying and comparing information content in probabilistic representation spaces, moving beyond conventional point-estimate assumptions. We propose two differentiable information-theoretic measures: (1) a generalization of hard-clustering-based information comparison to arbitrary probabilistic spaces; and (2) a lightweight, fingerprint-based estimation method enabling efficient information assessment under ultra-low-bit communication constraints. Our framework integrates extensions of mutual information and conditional entropy, VAE/InfoGAN-based representation modeling, fingerprint sampling, and end-to-end differentiable optimization. Experiments demonstrate that the method accurately identifies redundant information in disentangled representations, uncovers cross-dataset and cross-model latent-space information consistency, and effectively guides the fusion of weak learners to produce unified representations with enhanced information completeness.
📝 Abstract
Probabilistic representation spaces convey information about a dataset and are shaped by factors such as the training data, network architecture, and loss function. Comparing the information content of such spaces is crucial for understanding the learning process, yet most existing methods assume point-based representations, neglecting the distributional nature of probabilistic spaces. To address this gap, we propose two information-theoretic measures to compare general probabilistic representation spaces by extending classic methods to compare the information content of hard clustering assignments. Additionally, we introduce a lightweight method of estimation that is based on fingerprinting a representation space with a sample of the dataset, designed for scenarios where the communicated information is limited to a few bits. We demonstrate the utility of these measures in three case studies. First, in the context of unsupervised disentanglement, we identify recurring information fragments within individual latent dimensions of VAE and InfoGAN ensembles. Second, we compare the full latent spaces of models and reveal consistent information content across datasets and methods, despite variability during training. Finally, we leverage the differentiability of our measures to perform model fusion, synthesizing the information content of weak learners into a single, coherent representation. Across these applications, the direct comparison of information content offers a natural basis for characterizing the processing of information.