An Informational Parsimony Perspective on Symmetry-Based Structure Extraction

2024-12-12

arXiv.org

AI Summary

This work addresses the automatic identification and interpretable utilization of group symmetries in intelligent models, focusing on modeling and extracting probabilistic symmetries from an information-minimality perspective. Method: We propose a unified “group–information” framework that deeply integrates group representation theory with the information bottleneck principle. It defines tunable-precision “soft symmetries” and formally characterizes distributional/channel-level invariance and equivariance via a compression–divergence trade-off. A geometric complexity measure quantifies hierarchical symmetry structure. The method combines group-theoretic modeling, probabilistic symmetry compression, divergence-constrained optimization, and discrete-channel invariance analysis. Contribution/Results: On synthetic benchmarks, our framework successfully recovers nested equivariant structures in a layered manner and, for the first time, observes symmetry bifurcation driven by compression parameters—demonstrating its effectiveness in extracting and interpreting generalized probabilistic symmetries.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Extraction of structure, in particular of group symmetries, is increasingly crucial to understanding and building intelligent models. In particular, some information-theoretic models of parsimonious learning have been argued to induce invariance extraction. Here, we formalise these arguments from a group-theoretic perspective. We then extend them to the study of more general probabilistic symmetries, through compressions preserving well-studied geometric measures of complexity. More precisely, we formalise a trade-off between compression and preservation of the divergence from a given hierarchical model, yielding a novel generalisation of the Information Bottleneck framework. Through appropriate choices of hierarchical models, we fully characterise (in the discrete and full support case) channel invariance, channel equivariance and distribution invariance under permutation. Allowing imperfect divergence preservation then leads to principled definitions of"soft symmetries", where the"coarseness"corresponds to the degree of compression of the system. In simple synthetic experiments, we demonstrate that our method successively recovers, at increasingly compressed"resolutions", nested but increasingly perturbed equivariances, where new equivariances emerge at bifurcation points of the trade-off parameter. Our framework suggests a new path for the extraction of generalised probabilistic symmetries.

Problem

• Extracting group symmetries for intelligent models
• Generalizing probabilistic symmetries via compression
• Characterizing channel and distribution invariance

Innovation

• Group-theoretic formalization of symmetry extraction
• Compression preserving geometric complexity measures
• Generalized Information Bottleneck for soft symmetries