Compositional Sparsity as an Inductive Bias for Neural Architecture Design

2026-05-14

AI Summary

This work addresses the curse of dimensionality in deep neural networks by introducing combinatorial sparsity as an inductive bias to construct interpretable sparse architectures. The proposed method integrates Information Filtering Networks (IFN) with Homological Neural Networks (HNN), leveraging constrained information maximization to extract sparse dependency structures among variables and mapping them into fixed, sparse neural graphs that embody hierarchical compositionality. Empirical evaluations demonstrate that the resulting models achieve performance on par with or superior to dense counterparts on both synthetic and real-world datasets, while using significantly fewer parameters. Moreover, the approach exhibits lower variance, enhanced stability under increasing input dimensionality, and robustness to hyperparameter choices.

Abstract

Identifying the structural priors that enable Deep Neural Networks (DNNs) to overcome the curse of dimensionality is a fundamental challenge in machine learning theory. Existing literature suggests that effective high-dimensional learning is driven by compositional sparsity, where target functions decompose into constituents supported on low-dimensional variable subsets. To investigate this hypothesis, we combine Information Filtering Networks (IFNs), which extract sparse dependency structures via constrained information maximisation, with Homological Neural Networks (HNNs), which map the inferred topology into fixed-wiring sparse neural graphs. We formalise the design principles underlying this construction and present an interpretable pipeline in which abstraction emerges through hierarchical composition. HNNs are orders of magnitude sparser than standard DNNs and require only minimal hyperparameter tuning. On synthetic tasks with known sparse hierarchies, HNNs recover the underlying compositional structure and remain stable in regimes where dense alternatives degrade as dimensionality increases. Across a broad suite of real-world datasets, HNNs consistently match or outperform dense baselines while using far fewer parameters, exhibiting lower variance and showing reduced sensitivity to hyperparameters.

Problem

• compositional sparsity
• curse of dimensionality
• inductive bias
• neural architecture design
• high-dimensional learning

Innovation

• compositional sparsity
• Information Filtering Networks
• Homological Neural Networks
• inductive bias
• sparse neural architecture