The geometry of efficient codes: how rate-distortion trade-offs distort the latent representations of generative models

2024-06-11

arXiv.org

AI Summary

This work investigates how model capacity, data distribution, and task objectives jointly induce systematic geometric distortions in latent representations of generative models—exemplified by β-VAEs—under rate-distortion constraints. Leveraging rate-distortion theory, differential geometric analysis of latent spaces, and information bottleneck experiments, we systematically identify and formalize three distinct distortion patterns: *prototypification* (cluster-like organization), *specialization* (task-aligned alignment), and *orthogonalization* (dimension-wise decorrelation). We reveal their coexistence mechanisms and synergistic effects on representation structure, demonstrating how compression-induced distortion shapes interpretable geometric landscapes in latent space. Our findings establish a unified geometric principles framework for understanding efficient representation learning under resource constraints—applicable to both artificial and biological intelligence. (149 words)

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Living organisms rely on internal models of the world to act adaptively. These models cannot encode every detail and hence need to compress information. From a cognitive standpoint, information compression can manifest as a distortion of latent representations, resulting in the emergence of representations that may not accurately reflect the external world or its geometry. Rate-distortion theory formalizes the optimal way to compress information, by considering factors such as capacity limitations, the frequency and the utility of stimuli. However, while this theory explains why the above factors distort latent representations, it does not specify which specific distortions they produce. To address this question, here we systematically explore the geometry of the latent representations that emerge in generative models that operate under the principles of rate-distortion theory ($eta$-VAEs). Our results highlight that three main classes of distortions of internal representations -- prototypization, specialization, orthogonalization -- emerge as signatures of information compression, under constraints on capacity, data distributions and tasks. These distortions can coexist, giving rise to a rich landscape of latent spaces, whose geometry could differ significantly across generative models subject to different constraints. Our findings contribute to explain how the normative constraints of rate-distortion theory distort the geometry of latent representations of generative models of artificial systems and living organisms.

Problem

• Explores distortions in generative model representations.
• Investigates rate-distortion trade-offs in image encoding.
• Identifies three primary types of latent distortions.

Innovation

• Rate-distortion theory applied
• Beta Variational Autoencoders used
• Identifies three latent distortions