Dimensionality Controls When Modularity Helps in Continual Learning

2026-06-16

AI Summary

This study addresses the central challenge in continual learning of balancing plasticity and stability while mitigating interference between tasks, particularly when they share network structures. Using a sequential A-B-A paradigm, the authors systematically compare modular recurrent networks against monolithic baselines to investigate how modularity, task similarity, and representational dimensionality interact. They find that the effectiveness of modularity hinges critically on the representational dimensionality induced by weight initialization scale: in low-dimensional “rich” regimes, modular networks adaptively form task-specific subspaces that overlap, partially align, or separate according to task similarity, significantly outperforming monolithic architectures; in high-dimensional “lazy” regimes, both approaches perform comparably. The work frames robustness and safety as problems of adaptive subspace allocation and validates this mechanism through controlled weight scaling and internal geometric analysis.

Abstract

Compositional learning systems must balance plasticity, the ability to acquire new knowledge, with stability, the preservation of previously learned components, especially when tasks share structure and risk interference. We study how modular architecture, task similarity, and representational dimensionality jointly shape compositional continual learning in a sequential A-B-A paradigm, comparing a task-partitioned recurrent network to a single-network baseline while inducing high- and low-dimensional regimes via weight-scale manipulations. In a high-dimensional "lazy" regime, both architectures achieve similar performance and internal geometry, suggesting that explicit modular structure has little impact when representations are weakly constrained. In a lower-dimensional "rich" regime, modularity becomes decisive: the modular network develops graded task-specific subspaces that overlap for similar tasks, partially align for moderately dissimilar tasks, and separate for dissimilar tasks, yielding a more compositional and interpretable organization than the single network. These findings identify the representational regime induced by initialization scale, which co-varies with representational dimensionality, as a key factor governing when compositional, modular structure is functionally beneficial in continual learning, and support viewing safety and robustness as problems of adaptive allocation of representational subspaces rather than fixed separation versus sharing.

Problem

• continual learning
• modularity
• dimensionality
• compositional learning
• representational geometry

Innovation

• modularity
• representational dimensionality
• continual learning
• compositional learning
• task similarity