Range, not Independence, Drives Modularity in Biological Inspired Representation

2024-10-08

AI Summary

This study investigates the mechanistic origins of modular versus entangled neural representations—where single neurons selectively encode individual semantic variables—in biologically inspired networks. We propose the “range-driven modularity” theory, positing that sufficient dispersion of source variable supports—not statistical independence—is the fundamental condition for modularity. We derive necessary and sufficient theoretical conditions for modularity in nonnegative, energy-constrained linear autoencoders, and extend the framework to nonlinear feedforward and recurrent architectures. Methodologically, we integrate nonnegative matrix factorization, energy-based optimization, modeling of entorhinal cortical neural data, and multi-task training validation. Our approach successfully reproduces and reconciles contradictory experimental observations in both artificial networks and empirical neural recordings. It provides a novel interpretation of neural mixed selectivity, introduces the first operationally testable criterion for modularity, and substantially enhances representational controllability and interpretability in both neural systems and AI models.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Why do biological and artificial neurons sometimes modularise, each encoding a single meaningful variable, and sometimes entangle their representation of many variables? In this work, we develop a theory of when biologically inspired networks -- those that are nonnegative and energy efficient -- modularise their representation of source variables (sources). We derive necessary and sufficient conditions on a sample of sources that determine whether the neurons in an optimal biologically-inspired linear autoencoder modularise. Our theory applies to any dataset, extending far beyond the case of statistical independence studied in previous work. Rather we show that sources modularise if their support is ``sufficiently spread''. From this theory, we extract and validate predictions in a variety of empirical studies on how data distribution affects modularisation in nonlinear feedforward and recurrent neural networks trained on supervised and unsupervised tasks. Furthermore, we apply these ideas to neuroscience data, showing that range independence can be used to understand the mixing or modularising of spatial and reward information in entorhinal recordings in seemingly conflicting experiments. Further, we use these results to suggest alternate origins of mixed-selectivity, beyond the predominant theory of flexible nonlinear classification. In sum, our theory prescribes precise conditions on when neural activities modularise, providing tools for inducing and elucidating modular representations in brains and machines.

Problem

• Neural Modularity
• Bio-inspired Networks
• Variable Representation

Innovation

• Modularisation Theory
• Sufficiently Spread Source Support
• Neural Networks Representation