Emergent Generalization by Representation Learning in Artificial Neural Networks

2026-07-11

AI Summary

This study investigates whether low-dimensional neural representations possess causal biological significance and enhance generalization. By imposing an explicit information bottleneck in recurrent neural networks to enforce low-dimensional learning, the authors integrate causal emergence and information-theoretic measures to analyze rotational invariance and out-of-distribution generalization in temporal prediction tasks. They find that representational complexity exhibits a task-dependent, non-monotonic trajectory during the transition from memorization to generalization, with its peak predictive of generalization performance. This dynamic is corroborated in neural recordings from mouse hippocampal CA1 and aligns with behavioral outcomes. The results demonstrate a strong correlation between the strength of emergent low-dimensional representations and generalization capacity, supporting their functional role in cognition.

Abstract

Dimensionality reduction has proven powerful for identifying neural manifolds, which are low-dimensional structures underlying high-dimensional neural activity. These low-dimensional representations have improved the interpretability of population-level coding. Yet whether such low-dimensional representations are biologically relevant and confer functional advantages in learning systems, or merely reflect neuron-level activity, remains contested in neuroscience. We show that an explicit information bottleneck forcing a recurrent neural network to learn a low-dimensional representation is necessary for rotational and out-of-distribution generalisation in a time-series prediction task. Using information-theoretic measures of causal emergence, we characterise the dynamics of this representation across the memorisation-to-generalisation transition, finding a non-monotonic trajectory which shows an initial decrease, a minimum, and a subsequent rise to a maximum, even as prediction loss falls monotonically. This trajectory scales with task complexity, and the magnitude of emergent structure reliably predicts generalisation performance. Analysis of CA1 hippocampal activity in mice learning an alternating maze task reveals analogous non-monotonic emergence dynamics that track behavioural performance. Together, these findings indicate that the ability of neural networks to learn compact, distributed and emergent representations confers a functional advantage for generalisation, supporting a causal role for learned representations in cognition.

Problem

• representation learning
• generalization
• neural manifolds
• causal emergence
• dimensionality reduction

Innovation

• information bottleneck
• causal emergence
• low-dimensional representation
• out-of-distribution generalization
• neural manifold