Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds

2026-06-18

AI Summary

Deep neural networks trained under standard optimization struggle to spontaneously develop compositional structures, limiting their generalization capabilities. This work systematically investigates how network depth and connection sparsity influence the emergence of compositionality, revealing that strong compositionality arises only within a narrow regime of depth and connectivity. To steer models toward such solutions, the authors propose similarity-based pruning combined with a depth-informed heuristic, grounded in theories of combinatorial sparsity, volume ratio analysis, and feature interference bounds. Experiments demonstrate that models achieve significantly enhanced compositionality when operating at task-dependent optimal depths with specifically configured sparse connectivity; outside this region, optimization tends to converge to fragmented, non-compositional solutions.

Abstract

Compositionality is believed to be the foundation for generalization, enabling models to reuse meaningful primitives in novel combinations. Yet, models trained with standard gradient-based optimization rarely, and often only weakly, exhibit compositional internal structure, and it remains unclear how or why such compositionality forms. In this work, we show that compositionality emerges in a narrow connectivity-depth sweet spot. Along the connectivity axis, compositionality only appears in some specifically sparse networks, heavily depends on which connections remain rather than on weights' sparsity alone. Along the depth axis, compositionality emerges within a narrow, target-dependent regime, peaking at specific depths, while both shallower and deeper networks fail. When either the depth or connectivity condition is violated, gradient descent silently converges to fractured solutions rather than compositional ones. To discover and exploit this emergence, we introduce (i) similarity-based pruning (SP) to recover compositional connectivity and (ii) a heuristic depth predictor to estimate where compositionality is most likely to appear. Finally, we support these empirical findings with a theoretical framework based on compositional sparsity, volume-ratio arguments, and feature-interference bounds, explaining why compositional solutions are reachable only in a narrow depth-connectivity regime.

Problem

• compositionality
• generalization
• neural architecture
• gradient descent
• sparsity

Innovation

• compositionality
• connectivity sparsity
• network depth
• similarity-based pruning
• feature interference